szehon-ho opened a new pull request, #58222:
URL: https://github.com/apache/spark/pull/58222
### What changes were proposed in this pull request?
Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of
`docs/sql-migration-guide.md`, documenting the behavior change made by
SPARK-58517 (#57722):
- Declarative Pipelines now honors `spark.sql.caseSensitive` when inferring
and
evolving pipeline table schemas. Under case-insensitive resolution (the
default), column names differing only in case identify the same column, so
flows contribute a single column rather than one per spelling, and the note
states which spelling survives.
- All flows writing to one pipeline table must agree on the effective
`spark.sql.caseSensitive`, or the update fails with the new
`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` error condition.
Documentation only; no code change.
### Why are the changes needed?
SPARK-58517 first ships in 4.3.0 (merged to `master`, `branch-4.x`, and
`branch-4.3`), and the behavior it changed has been in place since 4.1.0:
`SchemaMergingUtils.mergeSchemas` merged positionally, leaving
`StructType.merge`'s `caseSensitive = true` default;
`DataflowGraph.inferredSchema`
merged with `reduce(SchemaMergingUtils.mergeSchemas)`; and
`SchemaInferenceUtils.diffSchemas` keyed columns on exact field names. Users
upgrading from 4.1.x or 4.2.x will see two observable differences that a
migration note should cover:
- A target schema that previously evolved to carry both spellings, then
failed
later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`,
now folds into a single column.
- A pipeline whose flows disagree on `spark.sql.caseSensitive` now fails up
front with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change. It documents a user-facing change
that
was already made by SPARK-58517.
### How was this patch tested?
No tests; documentation only. The error condition names in the notes were
checked against the tree (`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` in
`error-conditions.json`, `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA` in
`GraphErrors.scala`), as was the "lowest flow identifier wins" rule
(`SchemaInferenceUtils.inferSchemaFromFlows` sorts by
`_.identifier.unquotedString`). The link form
`declarative-pipelines-programming-guide.html` matches the existing link in
`docs/index.md`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 (Cursor)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
