Terry Kim created SPARK-31078:
---------------------------------
Summary: outputOrdering should handle aliases correctly
Key: SPARK-31078
URL: https://issues.apache.org/jira/browse/SPARK-31078
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Terry Kim
Currently, `outputOrdering` doesn't respect aliases. Thus, the following would
produce an unnecessary sort node:
{code:java}
withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "0") {
val df = (0 until 20).toDF("i").as("df")
df.repartition(8, df("i")).write.format("parquet")
.bucketBy(8, "i").sortBy("i").saveAsTable("t")
val t1 = spark.table("t")
val t2 = t1.selectExpr("i as ii")
t1.join(t2, t1("i") === t2("ii")).explain
}
== Physical Plan ==
*(3) SortMergeJoin [i#8], [ii#10], Inner
:- *(1) Project [i#8]
: +- *(1) Filter isnotnull(i#8)
: +- *(1) ColumnarToRow
: +- FileScan parquet default.t[i#8] Batched: true, DataFilters:
[isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/...,
PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int>,
SelectedBucketsCount: 8 out of 8
+- *(2) Sort [ii#10 ASC NULLS FIRST], false, 0
+- *(2) Project [i#8 AS ii#10]
+- *(2) Filter isnotnull(i#8)
+- *(2) ColumnarToRow
+- FileScan parquet default.t[i#8] Batched: true, DataFilters:
[isnotnull(i#8)], Format: Parquet, Location: InMemoryFileIndex[file:/...,
PartitionFilters: [], PushedFilters: [IsNotNull(i)], ReadSchema: struct<i:int>,
SelectedBucketsCount: 8 out of 8
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
