[
https://issues.apache.org/jira/browse/SPARK-57746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-57746.
------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 56859
[https://github.com/apache/spark/pull/56859]
> Assign a name to the error conditions _LEGACY_ERROR_TEMP_3171, 3172
> -------------------------------------------------------------------
>
> Key: SPARK-57746
> URL: https://issues.apache.org/jira/browse/SPARK-57746
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> h2. What
> _LEGACY_ERROR_TEMP_3171 and _LEGACY_ERROR_TEMP_3172 are placeholder error
> conditions
> with no proper name or SQLSTATE. Assign them proper named error conditions as
> part of
> the error-class migration (SPARK-37935).
> Both are thrown by Parquet aggregate push-down in ParquetUtils when the
> required
> column-chunk statistics are missing. They carry the same parameters
> (filePath, config)
> and the same remedy, and differ only in which statistic is absent, so they
> are merged
> into a single parent condition with two sub-conditions.
> Current definitions (error-conditions.json):
> {code:json}
> "_LEGACY_ERROR_TEMP_3171": {
> "message": [
> "Number of nulls not set for Parquet file <filePath>. Set SQLConf
> <config> to false and execute again."
> ]
> },
> "_LEGACY_ERROR_TEMP_3172": {
> "message": [
> "No min/max found for Parquet file <filePath>. Set SQLConf <config> to
> false and execute again."
> ]
> }
> {code}
> New definition:
> {code:json}
> "PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED": {
> "message": [
> "Aggregate push-down is not supported for Parquet file <filePath>. Set
> SQLConf <config> to false and execute again."
> ],
> "subClass": {
> "NO_MIN_MAX": {
> "message": ["No min/max value found in the column statistics."]
> },
> "NO_NUM_NULLS": {
> "message": ["The number of nulls is not set in the column statistics."]
> }
> },
> "sqlState": "0A000"
> }
> {code}
> h2. Where
> * _LEGACY_ERROR_TEMP_3172 ->
> PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX:
> thrown by ParquetUtils.getCurrentBlockMaxOrMin during MIN/MAX push-down
> when a
> column chunk has no non-null values (!statistics.hasNonNullValue).
> * _LEGACY_ERROR_TEMP_3171 ->
> PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_NUM_NULLS:
> thrown by ParquetUtils.getNumNulls during COUNT push-down when the null
> count is
> not set (!statistics.isNumNullsSet).
> h2. Notes
> The scope of this ticket is extended to also cover the sibling condition
> _LEGACY_ERROR_TEMP_3171 (originally only 3172), since both live in the same
> code path
> and are naturally expressed as sub-conditions of one parent.
> There is also a separate, behavioral follow-up - SPARK-57739 - proposing that
> Parquet
> return NULL for an all-null block (matching ORC) rather than throwing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
