Yuming Wang created SPARK-32352:
-----------------------------------
Summary: Partially push down support data filter
Key: SPARK-32352
URL: https://issues.apache.org/jira/browse/SPARK-32352
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Yuming Wang
We support partially push partition filters since SPARK-28169. We can also
support partially push down data filters if it mixed in partition filters and
data filters. For example:
{code:scala}
spark.sql(
s"""
|CREATE TABLE t(i INT, p STRING)
|USING parquet
|PARTITIONED BY (p)""".stripMargin)
spark.range(0, 1000).selectExpr("id as col").createOrReplaceTempView("temp")
for (part <- Seq(1, 2, 3, 4)) {
sql(s"""
|INSERT OVERWRITE TABLE t PARTITION (p='$part')
|SELECT col FROM temp""".stripMargin)
}
spark.sql("SELECT * FROM t WHERE WHERE (p = '1' AND i = 1) OR (p = '2' and i =
2)").explain()
{code}
We can also push \{{i = 1 or i = 2 }}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
