Yuming Wang created SPARK-32351:
-----------------------------------
Summary: Partially pushed partition filters are not explained
Key: SPARK-32351
URL: https://issues.apache.org/jira/browse/SPARK-32351
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Yuming Wang
How to reproduce this issue:
{code:scala}
spark.sql(
s"""
|CREATE TABLE t(i INT, p STRING)
|USING parquet
|PARTITIONED BY (p)""".stripMargin)
spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp")
for (part <- Seq(1, 2, 3, 4)) {
sql(s"""
|INSERT OVERWRITE TABLE t PARTITION (p='$part')
|SELECT col FROM temp""".stripMargin)
}
spark.sql("SELECT * FROM t WHERE WHERE (p = '1' AND i = 1) OR (p = '2' and i =
2)").explain
{code}
We have pushed down {{p = '1' or p = '2'}} since SPARK-28169, but this pushed
down filter not in explain
{noformat}
== Physical Plan ==
*(1) Filter (((p#21 = 1) AND (i#20 = 1)) OR ((p#21 = 2) AND (i#20 = 2)))
+- *(1) ColumnarToRow
+- FileScan parquet default.t[i#20,p#21] Batched: true, DataFilters: [],
Format: Parquet, Location:
InMemoryFileIndex[file:/Users/yumwang/spark/SPARK-32289/sql/core/spark-warehouse/org.apache.spark...,
PartitionFilters: [], PushedFilters: [], ReadSchema: struct<i:int>
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
