[
https://issues.apache.org/jira/browse/SPARK-57739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092197#comment-18092197
]
Max Gekk commented on SPARK-57739:
----------------------------------
Note: the error thrown on an all-null row group, {{_LEGACY_ERROR_TEMP_3172}},
has been given a proper name in SPARK-57746
(https://github.com/apache/spark/pull/56859):
{{PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX}} (the sibling
{{_LEGACY_ERROR_TEMP_3171}} -> {{.NO_NUM_NULLS}} in the same PR).
So when implementing this behavioral change (return NULL instead of throwing),
the condition to update/remove is
{{PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX}} in
error-conditions.json, thrown by {{ParquetUtils.getCurrentBlockMaxOrMin}},
rather than the old {{_LEGACY_ERROR_TEMP_3172}}.
> Parquet MIN/MAX aggregate push-down fails on an all-null row group instead of
> returning NULL
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-57739
> URL: https://issues.apache.org/jira/browse/SPARK-57739
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> During Parquet MIN/MAX aggregate push-down,
> {{ParquetUtils.getCurrentBlockMaxOrMin}} throws {{_LEGACY_ERROR_TEMP_3172}}
> ("No min/max found for Parquet file ... Set SQLConf ... to false and execute
> again.") when a column chunk has {{!statistics.hasNonNullValue}} -- i.e. a
> row group where the column is all-null. The throw is inside the per-block
> loop, so it fires if *any* row group has an all-null chunk for the column
> (not only when the whole column is null). There is no fallback in
> {{ParquetPartitionReaderFactory}} (row and columnar paths), so the exception
> propagates and *fails the query*.
> h2. Why it's a bug
> MIN/MAX over an all-null input is NULL by SQL semantics, and the ORC
> aggregate push-down path already returns NULL for this case
> ({{OrcUtils.getMinMaxFromColumnStatistics}} guards {{getNumberOfValues == 0
> -> null}}). So the same query succeeds on ORC but fails on Parquet:
> {code:sql}
> -- column c has a row group that is entirely NULL
> SELECT min(c) FROM parquet_t; -- Parquet: throws _LEGACY_ERROR_TEMP_3172;
> ORC: returns NULL
> {code}
> This is type-agnostic (affects every push-down-eligible type), with
> {{spark.sql.parquet.aggregatePushdown=true}}.
> h2. Proposed fix
> In {{getCurrentBlockMaxOrMin}}, treat an all-null block as "no contribution"
> (skip it / return NULL) rather than throwing, so an all-null column yields
> NULL and a partially-null column combines the non-null row groups -- matching
> ORC and SQL semantics. The {{_LEGACY_ERROR_TEMP_3172}} condition can then be
> removed (or, if kept for a genuinely-unreadable-stats case, given a proper
> named error class as part of the legacy-error migration).
> h2. User-facing change
> Yes -- a query that currently errors will return the correct NULL. Needs a
> migration/release note.
> h2. Tests
> Add an all-null (non-empty) column case to
> {{FileSourceAggregatePushDownSuite}} asserting MIN/MAX = NULL uniformly
> across Parquet V1/V2 and ORC V1/V2 (the case intentionally left unasserted in
> SPARK-57568 because of this divergence).
> h2. Notes
> Found during review of SPARK-57568 (TIME aggregate push-down), but this is
> type-agnostic and unrelated to TIME, so filed as a standalone bug rather than
> a sub-task.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
