Yaoxuan Wu created SPARK-60598:
----------------------------------
Summary: Parquet aggregate pushdown returns MIN/MAX/COUNT of the
wrong column when a nested column precedes it
Key: SPARK-60598
URL: https://issues.apache.org/jira/browse/SPARK-60598
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 3.5.8, 4.0.0
Reporter: Yaoxuan Wu
{code:python}
spark.conf.set("spark.sql.sources.useV1SourceList", "") # Parquet through
the DSv2 scan
spark.createDataFrame([((100, 200), 1, 2), ((300, 400), None, 6), ((500, 600),
5, 4)],
"s struct<x:int,y:int>, a int, b int") \
.coalesce(1).write.mode("overwrite").parquet("/tmp/agg_nested")
spark.read.parquet("/tmp/agg_nested").createOrReplaceTempView("t")
spark.conf.set("spark.sql.parquet.aggregatePushdown", "false")
spark.sql("SELECT max(a), min(b), count(a) FROM t").collect() #
[Row(max(a)=5, min(b)=2, count(a)=2)]
spark.conf.set("spark.sql.parquet.aggregatePushdown", "true")
spark.sql("SELECT max(a), min(b), count(a) FROM t").collect() #
[Row(max(a)=600, min(b)=1, count(a)=3)]
{code}
With pushdown, {{max(a)}} is the max of {{s.y}}, {{min(b)}} is the min of
{{a}}, and {{count(a)}} uses the null count of {{s.y}}. When the shifted column
has a different physical type the query fails instead, e.g. {{max}} of a
BOOLEAN column: {{ClassCastException: class java.lang.Integer cannot be cast to
class java.lang.Boolean}}.
Cause: in {{ParquetUtils.getPushedDownAggResult}}, {{index =
dataSchema.fieldNames.toList.indexOf(colName)}} is the column's position among
the top-level fields, but {{getCurrentBlockMaxOrMin}} / {{getNumNulls}} use it
as {{columnChunkMetaData.get(index)}}, i.e. as the index of a leaf column chunk
of the row group. A struct, array or map column has one chunk per leaf, so
every later column is shifted. Master uses
{{dataSchema.getFieldIndex(colName)}} (SPARK-58711), the same top-level index.
Looking the chunk up by its path ({{ColumnChunkMetaData.getPath}}) would fix it.
Reproduces on 3.5.8, 4.0.0, 4.1.1 and 4.2.0 with
{{spark.sql.parquet.aggregatePushdown=true}} (default false).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
