This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 030b1e16a879 [SPARK-58940][SDP][DOC] Add migration notes for
Declarative Pipelines case sensitivity
030b1e16a879 is described below
commit 030b1e16a8790e12b229b1ad158665616abf4003
Author: Szehon Ho <[email protected]>
AuthorDate: Mon Aug 24 14:08:28 2026 -0700
[SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case
sensitivity
### What changes were proposed in this pull request?
Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of
`docs/sql-migration-guide.md`, documenting the behavior change made by
SPARK-58517 (#57722):
- Declarative Pipelines now honors `spark.sql.caseSensitive` when inferring
and
evolving pipeline table schemas. Under case-insensitive resolution (the
default), column names differing only in case identify the same column, so
flows contribute a single column rather than one per spelling, and the
note
states which spelling survives.
- All flows writing to one pipeline table must agree on the effective
`spark.sql.caseSensitive`, or the update fails with the new
`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` error condition.
Documentation only; no code change.
### Why are the changes needed?
SPARK-58517 first ships in 4.3.0 (merged to `master`, `branch-4.x`, and
`branch-4.3`), and the behavior it changed has been in place since 4.1.0:
`SchemaMergingUtils.mergeSchemas` merged positionally, leaving
`StructType.merge`'s `caseSensitive = true` default;
`DataflowGraph.inferredSchema`
merged with `reduce(SchemaMergingUtils.mergeSchemas)`; and
`SchemaInferenceUtils.diffSchemas` keyed columns on exact field names. Users
upgrading from 4.1.x or 4.2.x will see two observable differences that a
migration note should cover:
- A target schema that previously evolved to carry both spellings, then
failed
later with errors such as `COLUMN_ALREADY_EXISTS` or
`AMBIGUOUS_REFERENCE`,
now folds into a single column.
- A pipeline whose flows disagree on `spark.sql.caseSensitive` now fails up
front with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change. It documents a user-facing change
that
was already made by SPARK-58517.
### How was this patch tested?
No tests; documentation only. The error condition names in the notes were
checked against the tree (`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` in
`error-conditions.json`, `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA` in
`GraphErrors.scala`), as was the "lowest flow identifier wins" rule
(`SchemaInferenceUtils.inferSchemaFromFlows` sorts by
`_.identifier.unquotedString`). The link form
`declarative-pipelines-programming-guide.html` matches the existing link in
`docs/index.md`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 (Cursor)
Closes #58222 from szehon-ho/docs-sdp-case-sensitivity-migration-note.
Authored-by: Szehon Ho <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit 5a26f48078e4db37ea399aa68070c87af62528d5)
Signed-off-by: Szehon Ho <[email protected]>
---
docs/sql-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 0b2b403dc881..9f639309e69e 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -48,6 +48,8 @@ license: |
- Since Spark 4.3, the Spark-recognized time-travel options (`versionAsOf` and
`timestampAsOf`, or the keys configured by `spark.sql.timeTravelVersionKey` and
`spark.sql.timeTravelTimestampKey`) are rejected with
`UNSUPPORTED_FEATURE.TIME_TRAVEL` on catalog-backed Data Source V2 writes,
including table creation and replacement through `DataFrameWriterV2`, because
writes must target the current table state rather than a historical version.
Previously, Spark passed them to the connector as [...]
- Since Spark 4.3, `HAVING` is evaluated before window functions when the
`SELECT` list also contains generator functions such as `explode`. Previously,
window functions could include groups removed by `HAVING` and produce incorrect
results.
- Since Spark 4.3, the Spark Connect session errors
`INVALID_HANDLE.SESSION_CHANGED`/`SESSION_CLOSED`/`SESSION_NOT_FOUND` carry
SQLSTATE `08003` instead of `HY000`; the condition names are unchanged. Code
matching these errors on SQLSTATE should match `08003` or class `08`.
+- Since Spark 4.3, [Declarative
Pipelines](declarative-pipelines-programming-guide.html) honors
`spark.sql.caseSensitive` when inferring and evolving pipeline table schemas.
Under case-insensitive resolution (the default), column names that differ only
in case now identify the same column: flows writing to one table contribute a
single column rather than one per spelling, and a column that differs only in
case from one already persisted in the target is written to that column instead
of [...]
+- Since Spark 4.3, all flows writing to the same pipeline table must agree on
the effective `spark.sql.caseSensitive`, which each flow takes from its own SQL
configuration (a `SET` in pipeline source, which never reaches the session) and
otherwise from the session. A disagreement fails the update with
`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`, because that value decides
whether names differing only in case identify the same column and would
otherwise make the table's schema depend on [...]
## Upgrading from Spark SQL 4.1 to 4.2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
