anew opened a new pull request, #58223: URL: https://github.com/apache/spark/pull/58223
### What changes were proposed in this pull request? `SchemaInferenceUtils.inferSchemaFromFlows` merges the flows that write to a table in a deterministic order so that, when two flows emit a column whose names differ only in case, the surviving spelling is well-defined and every caller agrees on it. It established that order by sorting on `flow.identifier.unquotedString`. `TableIdentifier.unquotedString` joins the identifier's name parts with an **unescaped** `.`. That mapping is not one-to-one: two structurally distinct identifiers whose parts contain dots (a dot is legal inside a back-tick-quoted schema or flow name) can render to the same string -- e.g. `` `c`.`a.b`.`x` `` and `` `c`.`a`.`b.x` `` both render to `c.a.b.x`. Because `sortBy` is stable, colliding keys fall back to the incoming `flows` order, which is the nondeterministic flow-resolution completion order the sort was meant to remove. The surviving column casing could then differ from one run to the next. This PR sorts on `identifier.quotedString` instead. `quotedString` back-tick-quotes each part (escaping embedded back-ticks by doubling them), so its ordering stays one-to-one with the identifier and distinct identifiers can no longer collide. The comment on `inferSchemaFromFlows` is updated to record why `quotedString` is used. This is a follow-up to SPARK-58517, which added the sort. ### Why are the changes needed? The sort exists to guarantee a deterministic surviving column spelling. `unquotedString` is a lossy key, so under identifiers that contain dots the guarantee silently breaks: the merge order reverts to the nondeterministic completion order of concurrent flow resolution, and a case-only column spelling can flip between runs of an unchanged pipeline. On the non-merging evolution paths `diffSchemas` keys column identity on the exact name, so a run-to-run flip surfaces as a `deleteColumn` + `addColumn` for a column that only changed case. `quotedString` is injective and removes the collision. ### Does this PR introduce _any_ user-facing change? No. The behavior only ever differed for identifiers containing dotted (back-tick-quoted) name parts, and only within the unreleased `master` / `branch-4.x`; the ordering for ordinary identifiers is unchanged. ### How was this patch tested? Added `InferSchemaFromFlowsSuite`, a unit test that builds two resolved flows whose identifiers (`` `c`.`a.b`.`x` `` and `` `c`.`a`.`b.x` ``) collide under `unquotedString` but stay distinct under `quotedString`, each carrying a case-only-differing column, and asserts the inferred schema is identical regardless of the order the flows are passed in. The test fails with the previous `unquotedString` key and passes with `quotedString`. Ran the new suite via `build/sbt 'pipelines/testOnly *InferSchemaFromFlowsSuite'` (1 test, passed). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
