cloud-fan opened a new pull request, #57229:
URL: https://github.com/apache/spark/pull/57229
### What changes were proposed in this pull request?
Gate XML schema inference on the `nullValue` option. In
`XmlInferSchema.inferFrom(datum, typeSoFar)`, a value equal to
`options.nullValue` now preserves `typeSoFar` (is skipped for type inference)
instead of being inferred by its string content:
```scala
if (value == null || value.isEmpty || value == options.nullValue) {
return typeSoFar
}
```
This matches `CSVInferSchema.inferField`, which already gates on `nullValue`:
```scala
if (field == null || field.isEmpty || field == options.nullValue) {
typeSoFar
} else { ... }
```
### Why are the changes needed?
The XML **parser** already treats a value equal to `nullValue` as null when
reading (see the `options.nullValue` checks in `StaxXmlParser`), but schema
**inference** did not skip it — so a value equal to `nullValue` was inferred
from its string content. This produces an inference-vs-parse inconsistency: the
inferred type does not reflect how the data is actually read.
Concretely, with `nullValue=1` and input `<ROW>1</ROW>`, inference produced
`LongType` for the value tag while the parser read the value as `null`. It also
diverges from the CSV datasource, whose `inferField` gates date/numeric/etc.
inference on `nullValue`. XML inference is meant to mirror CSV (the two text
datasources share the same `tryParse*` cascade design), so this aligns the two.
### Does this PR introduce _any_ user-facing change?
Yes. With a non-default `nullValue`, XML schema inference no longer infers a
type from values equal to `nullValue`. For example, with `nullValue=1`,
`<ROW>1</ROW>` previously inferred the value tag as `LongType` (while the data
read as null); it now infers `StringType` (the canonicalized type of an
all-null field) and still reads null. With the default `nullValue` (unset),
behavior is unchanged.
### How was this patch tested?
- Added a unit test in `XmlInferSchemaTypeCastingSuite` ("A value matching
the nullValue option keeps the type inferred so far") asserting that a
`nullValue`-matching value preserves `typeSoFar` from
`NullType`/`LongType`/`TimestampType`, that a non-null value is still inferred
normally, and that a numeric-looking `nullValue` token (`nullValue=0`, value
`0`) is treated as null rather than inferred as `LongType`.
- Updated the existing `XmlInferSchemaSuite` end-to-end test "value tag -
equals to null value" to the corrected schema (`StringType` instead of
`LongType`), reflecting that the all-null value tag now canonicalizes to
`StringType`.
- Existing XML inference suites pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
