Takeshi Yamamuro created SPARK-35479:
----------------------------------------
Summary: Format PartitionFilters IN strings in scan nodes
Key: SPARK-35479
URL: https://issues.apache.org/jira/browse/SPARK-35479
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.2.0
Reporter: Takeshi Yamamuro
This ticket proposes to format strings correctly for `PushedFilters`. For
example, `explain()` for a query below prints `v in (array('a'))` as
`PushedFilters: [In(v, [WrappedArray(a)])]`;
{code}
scala> sql("create table t (v array<string>) using parquet")
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN
([a],null)], Format: Parquet, Location:
InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t],
PartitionFilters: [], PushedFilters: [In(v, [WrappedArray(a),null])],
ReadSchema: struct<v:array<string>>
{code}
This PR makes `explain()` print it as `PushedFilters: [In(v, [[a]])]`;
{code}
scala> sql("select * from t where v in (array('a'), null)").explain()
== Physical Plan ==
*(1) Filter v#4 IN ([a],null)
+- FileScan parquet default.t[v#4] Batched: false, DataFilters: [v#4 IN
([a],null)], Format: Parquet, Location:
InMemoryFileIndex[file:/Users/maropu/Repositories/spark/spark-3.1.1-bin-hadoop2.7/spark-warehouse/t],
PartitionFilters: [], PushedFilters: [In(v, [[a],null])], ReadSchema:
struct<v:array<string>>
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
