[
https://issues.apache.org/jira/browse/SPARK-58945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58945:
--------------------------------------------
Description:
Several Spark error paths are broken by mismatches between
{{messageParameters}} keys and the placeholders declared in
{{error-conditions.json}}. In addition, one legacy error class has a message
template duplicated from its neighbour. These either surface {{INTERNAL_ERROR}}
instead of the intended error condition, or report the wrong message text.
h3. Mechanism
Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException =
true}}
({{common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29}}),
so an unresolved placeholder throws {{IllegalArgumentException}}, which
{{ErrorClassesJSONReader}} converts into {{SparkException.internalError}}.
Message rendering is eager: the relevant exception constructors call
{{SparkThrowableHelper.getMessage}} during construction (for example
{{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}).
When substitution fails, the intended exception instance is never created, and
the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter for
error class: ...") in place of the actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
h3. Parameter key mismatches
Verified against {{error-conditions.json}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath; classifyException receives only pre-rendered
strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
path exists at that point
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
h3. Duplicated message template
{{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have byte-identical
message templates. 3069 ({{FileSourceStrategy.scala:271}}) is the
reserved-column-name collision and its text is correct. 3070
({{FileSourceStrategy.scala:291}}) is the unrecognized-metadata-field fallback
and inherited 3069's text during the error-class migration in SPARK-46351.
Here the call site is not at fault -- its {{Map("field" -> field.toString)}}
was already correct. The template is the bug, and 3070 currently reports a
reserved-column-name collision that never occurred. Before SPARK-46351 this
branch threw {{s"Unrecognized file metadata field: $field"}}.
h3. Why these survived
None of these error paths are exercised by tests. Because message rendering is
eager, any test asserting the intended error condition on an affected path
would fail during exception construction.
Note the site list is a lower bound. The sweep covered
{{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}}, and
{{mllib/}}, and skips non-literal parameter maps, non-literal error class
arguments, and {{.java}} sources.
was:
Several error constructors pass {{messageParameters}} keys that do not match
the placeholders declared in {{error-conditions.json}}, and one error class has
a message template that was duplicated from its neighbour. Both produce
{{INTERNAL_ERROR}} or a wrong message in place of the intended diagnosis.
h3. Mechanism
Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException =
true}}
({{common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29}}),
so an unresolved placeholder throws {{IllegalArgumentException}}, which
{{ErrorClassesJSONReader}} converts into {{SparkException.internalError}}.
Message rendering is eager - {{SparkThrowableHelper.getMessage}} is called from
the exception constructor ({{SparkException.scala:288}}) - so the intended
exception is never thrown. The user receives {{INTERNAL_ERROR}} ("Undefined
error message parameter for error class: ...") in place of the actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
h3. Parameter key mismatches
Verified against {{error-conditions.json}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath; classifyException receives only pre-rendered
strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
path exists at that point
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
h3. Duplicated message template
{{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have byte-identical
message templates. 3069 ({{FileSourceStrategy.scala:271}}) is the
reserved-column-name collision and its text is correct. 3070
({{FileSourceStrategy.scala:291}}) is the unrecognized-metadata-field fallback
and inherited 3069's text during the error-class migration.
Here the call site is not at fault - its {{Map("field" -> field.toString)}} was
always correct. The template is the bug, and 3070 currently reports a
reserved-column-name collision that never occurred. Spark 3.5 threw
{{s"Unrecognized file metadata field: $field"}} at this branch before the
migration.
h3. Why these survived
None of these error paths are exercised by tests; any test constructing these
exceptions would fail at construction.
Note the site list is a lower bound. The sweep covered
{{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}}, and
{{mllib/}}, and skips non-literal parameter maps, non-literal error class
arguments, and {{.java}} sources.
> Mismatched `messageParameters` keys cause `INTERNAL_ERROR` instead of the
> intended error
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-58945
> URL: https://issues.apache.org/jira/browse/SPARK-58945
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
>
> Several Spark error paths are broken by mismatches between
> {{messageParameters}} keys and the placeholders declared in
> {{error-conditions.json}}. In addition, one legacy error class has a message
> template duplicated from its neighbour. These either surface
> {{INTERNAL_ERROR}} instead of the intended error condition, or report the
> wrong message text.
> h3. Mechanism
> Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException
> = true}}
> ({{common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29}}),
> so an unresolved placeholder throws {{IllegalArgumentException}}, which
> {{ErrorClassesJSONReader}} converts into {{SparkException.internalError}}.
> Message rendering is eager: the relevant exception constructors call
> {{SparkThrowableHelper.getMessage}} during construction (for example
> {{common/utils/src/main/scala/org/apache/spark/SparkException.scala:288}}).
> When substitution fails, the intended exception instance is never created,
> and the user receives {{INTERNAL_ERROR}} ("Undefined error message parameter
> for error class: ...") in place of the actual diagnosis.
> h3. Repro
> {code:scala}
> spark.range(1).write.option("extension", "12").csv(path)
> {code}
> Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
> Actual: {{INTERNAL_ERROR}}
> h3. Parameter key mismatches
> Verified against {{error-conditions.json}}, ordered by reachability:
> {noformat}
> QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
> expects invalidValue; passes fileExtension + acceptable
> reached from CSVOptions.scala:128 via the "extension" write option
> QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
> expects clazz; passes invalidClass
> reached from HiveSessionStateBuilder.scala:253
> QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
> expects detail; passes details
> reached from PythonBatchWriterFactory.scala:58,62
> H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
> missing searchPath; classifyException receives only pre-rendered
> strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
> path exists at that point
> StateStoreErrors.scala:399
> STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
> expects colFamilyName/oldSchema/newSchema
> passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
> NOTE: the stateStoreColumnFamilyMismatch factory currently has no
> callers, so this one is latent rather than user-facing today.
> {noformat}
> h3. Duplicated message template
> {{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have
> byte-identical message templates. 3069 ({{FileSourceStrategy.scala:271}}) is
> the reserved-column-name collision and its text is correct. 3070
> ({{FileSourceStrategy.scala:291}}) is the unrecognized-metadata-field
> fallback and inherited 3069's text during the error-class migration in
> SPARK-46351.
> Here the call site is not at fault -- its {{Map("field" -> field.toString)}}
> was already correct. The template is the bug, and 3070 currently reports a
> reserved-column-name collision that never occurred. Before SPARK-46351 this
> branch threw {{s"Unrecognized file metadata field: $field"}}.
> h3. Why these survived
> None of these error paths are exercised by tests. Because message rendering
> is eager, any test asserting the intended error condition on an affected path
> would fail during exception construction.
> Note the site list is a lower bound. The sweep covered
> {{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}},
> and {{mllib/}}, and skips non-literal parameter maps, non-literal error class
> arguments, and {{.java}} sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
