[
https://issues.apache.org/jira/browse/SPARK-58945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subhramit Basu Bhowmick updated SPARK-58945:
--------------------------------------------
Description:
Several error constructors pass {{messageParameters}} keys that do not match
the placeholders declared in {{error-conditions.json}}, and one error class has
a message template that was duplicated from its neighbour. Both produce
{{INTERNAL_ERROR}} or a wrong message in place of the intended diagnosis.
h3. Mechanism
Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException =
true}}
({{common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29}}),
so an unresolved placeholder throws {{IllegalArgumentException}}, which
{{ErrorClassesJSONReader}} converts into {{SparkException.internalError}}.
Message rendering is eager - {{SparkThrowableHelper.getMessage}} is called from
the exception constructor ({{SparkException.scala:288}}) - so the intended
exception is never thrown. The user receives {{INTERNAL_ERROR}} ("Undefined
error message parameter for error class: ...") in place of the actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
h3. Parameter key mismatches
Verified against {{error-conditions.json}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath; classifyException receives only pre-rendered
strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
path exists at that point
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
h3. Duplicated message template
{{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have byte-identical
message templates. 3069 ({{FileSourceStrategy.scala:271}}) is the
reserved-column-name collision and its text is correct. 3070
({{FileSourceStrategy.scala:291}}) is the unrecognized-metadata-field fallback
and inherited 3069's text during the error-class migration.
Here the call site is not at fault - its {{Map("field" -> field.toString)}} was
always correct. The template is the bug, and 3070 currently reports a
reserved-column-name collision that never occurred. Spark 3.5 threw
{{s"Unrecognized file metadata field: $field"}} at this branch before the
migration.
h3. Why these survived
None of these error paths are exercised by tests; any test constructing these
exceptions would fail at construction.
Note the site list is a lower bound. The sweep covered
{{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}}, and
{{mllib/}}, and skips non-literal parameter maps, non-literal error class
arguments, and {{.java}} sources.
was:
Several error constructors pass {{messageParameters}} keys that do not match
the placeholders declared in {{{}error-conditions.json{}}}.
Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException =
true}}
({{{}common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29{}}}),
so an unresolved placeholder throws {{{}IllegalArgumentException{}}}, which
{{ErrorClassesJSONReader}} converts into {{{}SparkException.internalError{}}}.
The message rendering is eager - {{SparkThrowableHelper.getMessage}} is called
from the exception constructor ({{{}SparkException.scala:288{}}}) - so the
intended exception is never thrown. The user receives {{INTERNAL_ERROR}}
("Undefined error message parameter for error class: ...") in place of the
actual diagnosis.
h3. Repro
{code:scala}
spark.range(1).write.option("extension", "12").csv(path)
{code}
Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
Actual: {{INTERNAL_ERROR}}
h3. Affected sites
Verified against {{{}error-conditions.json{}}}, ordered by reachability:
{noformat}
QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
expects invalidValue; passes fileExtension + acceptable
reached from CSVOptions.scala:128 via the "extension" write option
QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
expects clazz; passes invalidClass
reached from HiveSessionStateBuilder.scala:253
QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
expects detail; passes details
reached from PythonBatchWriterFactory.scala:58,62
H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
missing searchPath
FileSourceStrategy.scala:291 _LEGACY_ERROR_TEMP_3070
expects colName/internalName; passes field
StateStoreErrors.scala:399 STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
expects colFamilyName/oldSchema/newSchema
passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
NOTE: the stateStoreColumnFamilyMismatch factory currently has no
callers, so this one is latent rather than user-facing today.
{noformat}
These survived because none of these error paths are exercised by tests; any
test constructing these exceptions would fail at construction.
> Mismatched `messageParameters` keys cause `INTERNAL_ERROR` instead of the
> intended error
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-58945
> URL: https://issues.apache.org/jira/browse/SPARK-58945
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Subhramit Basu Bhowmick
> Priority: Minor
>
> Several error constructors pass {{messageParameters}} keys that do not match
> the placeholders declared in {{error-conditions.json}}, and one error class
> has a message template that was duplicated from its neighbour. Both produce
> {{INTERNAL_ERROR}} or a wrong message in place of the intended diagnosis.
> h3. Mechanism
> Spark's {{StringSubstitutor}} defaults to {{enableUndefinedVariableException
> = true}}
> ({{common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29}}),
> so an unresolved placeholder throws {{IllegalArgumentException}}, which
> {{ErrorClassesJSONReader}} converts into {{SparkException.internalError}}.
> Message rendering is eager - {{SparkThrowableHelper.getMessage}} is called
> from the exception constructor ({{SparkException.scala:288}}) - so the
> intended exception is never thrown. The user receives {{INTERNAL_ERROR}}
> ("Undefined error message parameter for error class: ...") in place of the
> actual diagnosis.
> h3. Repro
> {code:scala}
> spark.range(1).write.option("extension", "12").csv(path)
> {code}
> Expected: {{INVALID_PARAMETER_VALUE.EXTENSION}}
> Actual: {{INTERNAL_ERROR}}
> h3. Parameter key mismatches
> Verified against {{error-conditions.json}}, ordered by reachability:
> {noformat}
> QueryExecutionErrors.scala:3214 INVALID_PARAMETER_VALUE.EXTENSION
> expects invalidValue; passes fileExtension + acceptable
> reached from CSVOptions.scala:128 via the "extension" write option
> QueryCompilationErrors.scala:4677 _LEGACY_ERROR_TEMP_2450
> expects clazz; passes invalidClass
> reached from HiveSessionStateBuilder.scala:253
> QueryExecutionErrors.scala:3242 INVALID_WRITER_COMMIT_MESSAGE
> expects detail; passes details
> reached from PythonBatchWriterFactory.scala:58,62
> H2Dialect.scala:230 TABLE_OR_VIEW_NOT_FOUND
> missing searchPath; classifyException receives only pre-rendered
> strings from JDBCTableCatalog.scala:104-106, so no Spark-side search
> path exists at that point
> StateStoreErrors.scala:399
> STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE
> expects colFamilyName/oldSchema/newSchema
> passes columnFamilyName/oldColumnFamilySchema/newColumnFamilySchema
> NOTE: the stateStoreColumnFamilyMismatch factory currently has no
> callers, so this one is latent rather than user-facing today.
> {noformat}
> h3. Duplicated message template
> {{_LEGACY_ERROR_TEMP_3069}} and {{_LEGACY_ERROR_TEMP_3070}} have
> byte-identical message templates. 3069 ({{FileSourceStrategy.scala:271}}) is
> the reserved-column-name collision and its text is correct. 3070
> ({{FileSourceStrategy.scala:291}}) is the unrecognized-metadata-field
> fallback and inherited 3069's text during the error-class migration.
> Here the call site is not at fault - its {{Map("field" -> field.toString)}}
> was always correct. The template is the bug, and 3070 currently reports a
> reserved-column-name collision that never occurred. Spark 3.5 threw
> {{s"Unrecognized file metadata field: $field"}} at this branch before the
> migration.
> h3. Why these survived
> None of these error paths are exercised by tests; any test constructing these
> exceptions would fail at construction.
> Note the site list is a lower bound. The sweep covered
> {{errorClass}}/{{messageParameters}} literal pairs in {{sql/}}, {{core/}},
> and {{mllib/}}, and skips non-literal parameter maps, non-literal error class
> arguments, and {{.java}} sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
