Raviston Thanasekar created SPARK-61499:
-------------------------------------------
Summary: Fix ORC aggregate push-down reading statistics of the
wrong column
Key: SPARK-61499
URL: https://issues.apache.org/jira/browse/SPARK-61499
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Raviston Thanasekar
ORC aggregate push-down (spark.sql.orc.aggregatePushdown=true, DSv2 scan)
answers MIN/MAX/COUNT from the column statistics in the file footer.
OrcUtils.createAggInternalRowFromFooter looks up a column's statistics with
dataSchema.getFieldIndex(columnName), the column's position in the data schema,
and uses it as the index into the statistics that
OrcFooterReader.readStatistics builds from the ORC file's own schema. The ORC
reader instead matches columns by name (OrcUtils.requestedColumnIds), unless
the file has Hive-style _colN names or orc.force.positional.evolution is set.
So push-down returns another column's statistics when the data schema orders or
names its columns differently from the file:
{code:python}
spark.conf.set("spark.sql.sources.useV1SourceList", "")
spark.sql("SELECT * FROM VALUES (1, 10, 100), (2, 20, 200) AS t(x, a,
b)").coalesce(1).write.orc("/tmp/t")
spark.conf.set("spark.sql.orc.aggregatePushdown", "true")
df = spark.read.schema("b INT, a INT, z INT").orc("/tmp/t")
df.selectExpr("min(b)").collect() # 1, expected 100 (from x)
df.selectExpr("count(z)").collect() # 2, expected 0 (from b; z is not in the
file)
df.selectExpr("max(z)").collect() # 200, expected NULL
{code}
With spark.sql.orc.aggregatePushdown=false, the queries return 100, 0 and NULL.
A data-schema position past the file's last column can also fail with
IndexOutOfBoundsException.
The same positional lookup was fixed for Parquet in SPARK-60598. For ORC, the
footer lookup should resolve the column with the reader's rules: by position
for _colN files and with forced positional evolution, otherwise by name
(exactly with spark.sql.caseSensitive=true, through toLowerCase(Locale.ROOT)
otherwise, failing on an ambiguous match), and treat a column the file does not
have as all null.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]