[
https://issues.apache.org/jira/browse/SPARK-14193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takeshi Yamamuro updated SPARK-14193:
-------------------------------------
Description:
This ticket describes an opportunity to skip unnecessary sorts if input data
have been already ordered in InMemoryTable.
Let's say we have a cached table with column 'a' sorted;
{code}
val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b")
val df2 = df1.sort("a").cache
df2.show // just cache data
{code}
If you say `df2.sort("a")`, the current spark generates a plan like;
{code}
== Physical Plan ==
Sort [a#13 ASC], true, 0
+- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true,
10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None
{code}
Since the current implementation cannot tell a difference between global sorted
columns and partition-locally sorted ones from `SparkPan#outputOrdering`.
was:
This ticket is to skip unnecessary sorts if input data have been already
ordered in InMemoryTable.
Let's say we have a cached table with column 'a' sorted;
{code}
val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b")
val df2 = df1.sort("a").cache
df2.show // just cache data
{code}
If you say `df2.sort("a")`, the current spark generates a plan like;
{code}
== Physical Plan ==
Sort [a#13 ASC], true, 0
+- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true,
10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0, None
{code}
This ticket targets at removing this unncessary sort.
> Skip unnecessary sorts if input data have been already ordered in
> InMemoryRelation
> ----------------------------------------------------------------------------------
>
> Key: SPARK-14193
> URL: https://issues.apache.org/jira/browse/SPARK-14193
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.1
> Reporter: Takeshi Yamamuro
>
> This ticket describes an opportunity to skip unnecessary sorts if input data
> have been already ordered in InMemoryTable.
> Let's say we have a cached table with column 'a' sorted;
> {code}
> val df1 = Seq((1, 0), (3, 0), (2, 0), (1, 0)).toDF("a", "b")
> val df2 = df1.sort("a").cache
> df2.show // just cache data
> {code}
> If you say `df2.sort("a")`, the current spark generates a plan like;
> {code}
> == Physical Plan ==
> Sort [a#13 ASC], true, 0
> +- InMemoryColumnarTableScan [a#13,b#14], InMemoryRelation [a#13,b#14], true,
> 10000, StorageLevel(true, true, false, true, 1), Sort [a#13 ASC], true, 0,
> None
> {code}
> Since the current implementation cannot tell a difference between global
> sorted columns and partition-locally sorted ones from
> `SparkPan#outputOrdering`.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
