mzhang opened a new pull request, #56681:
URL: https://github.com/apache/spark/pull/56681
### What changes were proposed in this pull request?
`from_xml(..., 'variant', map('mode', ...))` ignored the parse `mode`.
`XmlToStructsEvaluator.evaluate` called `StaxXmlParser.parseVariant` directly
for variant output, bypassing the `FailureSafeParser` the struct path uses, so
a malformed record aborted the whole job regardless of `mode`.
This routes variant output through `FailureSafeParser` like the struct path:
a new `StaxXmlParser.parseVariantColumn` wraps parse failures in
`BadRecordException`, and the evaluator parses into a single `value: variant`
row and unwraps it. PERMISSIVE rescues the row to `null`, FAILFAST rethrows.
It also fixes `from_xml` to variant under whole-stage codegen:
`XmlToStructs.doGenCode` hardcast the result to `InternalRow` (a compile error
for variant output) and did not null-check the result (NPE on a
PERMISSIVE-rescued `null`). It now casts to the actual output type and
null-checks.
### Why are the changes needed?
`from_xml` to variant is unusable on input that contains any malformed
record: the documented `mode` option has no effect for variant output, so
PERMISSIVE cannot rescue bad records and a single bad row fails the whole query
(including streaming jobs). Separately, `from_xml` to variant fails to compile
under whole-stage codegen.
### Does this PR introduce _any_ user-facing change?
Yes. With variant output, the `mode` option is now honored, matching struct
output and `from_json`:
- before: `from_xml(value, 'variant', map('mode', 'PERMISSIVE'))` on a
malformed record fails the whole query.
- after: it returns `null` for that record; well-formed records parse
normally. FAILFAST still throws.
This is a change within the unreleased master branch (variant support for
`from_xml` has not shipped in a release).
### How was this patch tested?
New unit test in `XmlFunctionsSuite` ("from_xml variant output honors the
parse mode"): a malformed record (an illegal XML control char) plus a
well-formed record, asserting PERMISSIVE rescues the bad record to `null` and
parses the good one, and FAILFAST throws. Run under both interpreted execution
and whole-stage codegen.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
