[
https://issues.apache.org/jira/browse/SPARK-57746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-57746:
-----------------------------
Description:
h2. What
_LEGACY_ERROR_TEMP_3171 and _LEGACY_ERROR_TEMP_3172 are placeholder error
conditions
with no proper name or SQLSTATE. Assign them proper named error conditions as
part of
the error-class migration (SPARK-37935).
Both are thrown by Parquet aggregate push-down in ParquetUtils when the required
column-chunk statistics are missing. They carry the same parameters (filePath,
config)
and the same remedy, and differ only in which statistic is absent, so they are
merged
into a single parent condition with two sub-conditions.
Current definitions (error-conditions.json):
{code:json}
"_LEGACY_ERROR_TEMP_3171": {
"message": [
"Number of nulls not set for Parquet file <filePath>. Set SQLConf <config>
to false and execute again."
]
},
"_LEGACY_ERROR_TEMP_3172": {
"message": [
"No min/max found for Parquet file <filePath>. Set SQLConf <config> to
false and execute again."
]
}
{code}
New definition:
{code:json}
"PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED": {
"message": [
"Aggregate push-down is not supported for Parquet file <filePath>. Set
SQLConf <config> to false and execute again."
],
"subClass": {
"NO_MIN_MAX": {
"message": ["No min/max value found in the column statistics."]
},
"NO_NUM_NULLS": {
"message": ["The number of nulls is not set in the column statistics."]
}
},
"sqlState": "0A000"
}
{code}
h2. Where
* _LEGACY_ERROR_TEMP_3172 -> PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX:
thrown by ParquetUtils.getCurrentBlockMaxOrMin during MIN/MAX push-down when a
column chunk has no non-null values (!statistics.hasNonNullValue).
* _LEGACY_ERROR_TEMP_3171 ->
PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_NUM_NULLS:
thrown by ParquetUtils.getNumNulls during COUNT push-down when the null count
is
not set (!statistics.isNumNullsSet).
h2. Notes
The scope of this ticket is extended to also cover the sibling condition
_LEGACY_ERROR_TEMP_3171 (originally only 3172), since both live in the same
code path
and are naturally expressed as sub-conditions of one parent.
There is also a separate, behavioral follow-up - SPARK-57739 - proposing that
Parquet
return NULL for an all-null block (matching ORC) rather than throwing.
was:
h2. What
{{_LEGACY_ERROR_TEMP_3172}} is a placeholder error condition with no proper
name or SQLSTATE. Assign it a proper named error class as part of the
error-class migration (SPARK-37935).
Current definition (error-conditions.json):
{code}
"_LEGACY_ERROR_TEMP_3172": {
"message": [
"No min/max found for Parquet file <filePath>. Set SQLConf <config> to
false and execute again."
]
}
{code}
h2. Where
Thrown by {{ParquetUtils.getCurrentBlockMaxOrMin}} (sql/core) during Parquet
MIN/MAX aggregate push-down when a column chunk has no non-null values
({{!statistics.hasNonNullValue}}).
h2. Notes
Found during review of SPARK-57568 (TIME aggregate push-down). There is also a
separate, *behavioral* follow-up -- SPARK-57739 -- proposing that Parquet
return {{NULL}} for an all-null block (matching ORC) rather than throwing. If
SPARK-57739 lands first, this error may be removed from the all-null path
entirely rather than renamed; otherwise assign it a proper name here.
> Assign a name to the error conditions _LEGACY_ERROR_TEMP_3171, 3172
> -------------------------------------------------------------------
>
> Key: SPARK-57746
> URL: https://issues.apache.org/jira/browse/SPARK-57746
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Max Gekk
> Priority: Major
>
> h2. What
> _LEGACY_ERROR_TEMP_3171 and _LEGACY_ERROR_TEMP_3172 are placeholder error
> conditions
> with no proper name or SQLSTATE. Assign them proper named error conditions as
> part of
> the error-class migration (SPARK-37935).
> Both are thrown by Parquet aggregate push-down in ParquetUtils when the
> required
> column-chunk statistics are missing. They carry the same parameters
> (filePath, config)
> and the same remedy, and differ only in which statistic is absent, so they
> are merged
> into a single parent condition with two sub-conditions.
> Current definitions (error-conditions.json):
> {code:json}
> "_LEGACY_ERROR_TEMP_3171": {
> "message": [
> "Number of nulls not set for Parquet file <filePath>. Set SQLConf
> <config> to false and execute again."
> ]
> },
> "_LEGACY_ERROR_TEMP_3172": {
> "message": [
> "No min/max found for Parquet file <filePath>. Set SQLConf <config> to
> false and execute again."
> ]
> }
> {code}
> New definition:
> {code:json}
> "PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED": {
> "message": [
> "Aggregate push-down is not supported for Parquet file <filePath>. Set
> SQLConf <config> to false and execute again."
> ],
> "subClass": {
> "NO_MIN_MAX": {
> "message": ["No min/max value found in the column statistics."]
> },
> "NO_NUM_NULLS": {
> "message": ["The number of nulls is not set in the column statistics."]
> }
> },
> "sqlState": "0A000"
> }
> {code}
> h2. Where
> * _LEGACY_ERROR_TEMP_3172 ->
> PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_MIN_MAX:
> thrown by ParquetUtils.getCurrentBlockMaxOrMin during MIN/MAX push-down
> when a
> column chunk has no non-null values (!statistics.hasNonNullValue).
> * _LEGACY_ERROR_TEMP_3171 ->
> PARQUET_AGGREGATE_PUSH_DOWN_UNSUPPORTED.NO_NUM_NULLS:
> thrown by ParquetUtils.getNumNulls during COUNT push-down when the null
> count is
> not set (!statistics.isNumNullsSet).
> h2. Notes
> The scope of this ticket is extended to also cover the sibling condition
> _LEGACY_ERROR_TEMP_3171 (originally only 3172), since both live in the same
> code path
> and are naturally expressed as sub-conditions of one parent.
> There is also a separate, behavioral follow-up - SPARK-57739 - proposing that
> Parquet
> return NULL for an all-null block (matching ORC) rather than throwing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
