anew opened a new pull request, #57733:
URL: https://github.com/apache/spark/pull/57733
### What changes were proposed in this pull request?
`StructType.merge(left, right, caseSensitive)` honors the `caseSensitive`
flag only for top-level struct fields. When it recurses into a matched field's
type, it calls `merge(leftType, rightType)` without passing `caseSensitive`, so
the parameter falls back to its `= true` default:
```scala
leftField.copy(
dataType = merge(leftType, rightType), // caseSensitive defaults back
to true
nullable = leftNullable || rightNullable)
```
As a result, a field whose name differs only in case, nested inside a
matched `struct` / `array<struct>` / `map<_, struct>`, is treated
as a distinct field even under case-insensitive merging -- producing a
merged schema that carries both spellings (e.g. `value` and
`Value`).
This PR forwards `caseSensitive` through the recursive call
(`merge(leftType, rightType, caseSensitive)`) so nested fields fold
case-only differences onto the existing (left) field, consistent with the
top-level behavior.
The `caseSensitive` parameter was originally added by SPARK-45346
("Parquet schema inference should respect case sensitive flag when
merging schema"), which fixed the top-level case but left the recursive
call unchanged; the recursive merge itself predates any
case-sensitivity concept (SPARK-5182, 2015). This change completes
SPARK-45346's intent for nested types.
### Why are the changes needed?
It is a correctness bug affecting every caller of the case-insensitive
`StructType.merge` overload, not just one subsystem. The most direct impact is
parquet/orc schema merging: `SchemaMergeUtils` passes the session's
`caseSensitive` to `StructType.merge`, but nested case-only-differing fields
were still merged case-sensitively, so a later case-insensitive
duplicate-column check could reject files
that should merge. This is the same class of bug SPARK-45346 fixed at the
top level, still latent for nested types.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix. Under case-insensitive resolution, merging two schemas
whose nested (struct/array/map) fields differ only in case now yields a single
field (keeping the left/existing spelling) instead of two conflicting fields.
This makes nested schema merging consistent with top-level merging and with
the engine's case-insensitive resolver. Case-sensitive merging (the default) is
unchanged.
### How was this patch tested?
Added a `StructTypeSuite` test covering top-level and nested
case-insensitive merges -- struct, `array<struct>`, and `map<_, struct>` --
asserting the case-only field folds onto the existing one, plus a
case-sensitive control that keeps both spellings. Ran `StructTypeSuite` and
`DataTypeSuite` (catalyst), and `ParquetSchemaSuite` / `OrcSourceSuite` (the
parquet/orc callers of the case-insensitive overload) to confirm no
regressions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
