[
https://issues.apache.org/jira/browse/SPARK-58707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie resolved SPARK-58707.
------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57924
[https://github.com/apache/spark/pull/57924]
> JSON schema pruning returns a null corrupt record column
> --------------------------------------------------------
>
> Key: SPARK-58707
> URL: https://issues.apache.org/jira/browse/SPARK-58707
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Assignee: Yang Jie
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> {{OptimizeCsvJsonExprs}} prunes the schema of a {{JsonToStructs}} down to the
> single field being
> selected. When that field is the corrupt record column, the pruned schema
> leaves the parser nothing
> to convert, so a malformed value in a dropped-away field is never detected
> and the corrupt record
> column comes back NULL instead of the record text.
> This is reachable with default configs: the rule requires
> {{options.isEmpty}}, which is the
> PERMISSIVE default, and
> {{spark.sql.optimizer.enableJsonExpressionOptimization}} defaults to true.
> Reproduction, schema {{a int, b int, _corrupt_record string}}, input {{ {"a":
> 1, "b": "bad"} }}:
> {code:scala}
> val df = Seq("""{"a": 1, "b": "bad"}""").toDS()
> .selectExpr("from_json(value, 'a int, b int, _corrupt_record string') as p")
> .selectExpr("p._corrupt_record")
> {code}
> ||spark.sql.optimizer.enableJsonExpressionOptimization||result||
> |true|null|
> |false|{{ {"a": 1, "b": "bad"} }}|
> The optimized plan shows the schema pruned to the corrupt column alone:
> {noformat}
> Project [from_json(StructField(_corrupt_record,StringType,true), value#1,
> ...)._corrupt_record ...]
> {noformat}
> Two branches are affected, {{GetStructField}} and {{CreateNamedStruct}}. The
> CSV branch added by
> SPARK-32968 already guards with {{schema(ordinal).name !=
> nameOfCorruptRecord}}; the JSON branches
> do not.
> {{GetArrayStructFields}} is not affected: an {{ArrayType}} schema takes
> {{JsonToStructsEvaluator}}'s
> {{case other =>}}, so {{corruptFieldIndex}} is {{None}} and corrupt-record
> semantics do not apply
> on that path.
> The existing SPARK-33907 test misses this because its bad record is
> structurally malformed
> ({{ {"a" 1, ...} }}), which fails at tokenization regardless of the requested
> schema. Exposing the
> defect needs a type mismatch on a field that pruning drops.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
