szehon-ho commented on code in PR #57722:
URL: https://github.com/apache/spark/pull/57722#discussion_r3725943837
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/util/SchemaInferenceUtils.scala:
##########
@@ -39,55 +114,71 @@ object SchemaInferenceUtils {
* The user-specified schema will take precedence over the inferred
schema.
* Returns an error if encountered during schema inference or merging the
inferred schema with
* the user-specified one.
+ *
+ * All merges honor the effective `spark.sql.caseSensitive` of the flows
writing to
+ * `tableIdentifier`, falling back to `sessionCaseSensitive` for flows that
do not set it. Under
+ * case-insensitive analysis, flows emitting column names that differ only
in case contribute a
+ * single column, and a declared column matches a flow column differing only
in case -- consistent
+ * with how the rest of the engine resolves those names. If two flows differ
only in column
+ * casing, the first flow's spelling wins because flows are merged in the
order provided.
Review Comment:
Suggestion: sort here -- `flows.sortBy(_.identifier.unquotedString)` before
the fold -- rather than at the single call site in
`DataflowGraph.inferSchemas`, so every caller merges in the same order.
As written the ordering is the caller's responsibility, and only one of the
three callers sorts. `GraphValidations.scala:266` is harmless (it compares
against a declared schema, whose casing wins that merge anyway), but
`elements.scala:193` produces the schema downstream flows resolve against:
```sql
CREATE STREAMING TABLE events;
-- declared first, identifier sorts second
CREATE FLOW from_us AS INSERT INTO events BY NAME SELECT id, payload AS
Value FROM STREAM us_src;
-- declared second, identifier sorts first
CREATE FLOW from_eu AS INSERT INTO events BY NAME SELECT id, payload AS
value FROM STREAM eu_src;
CREATE MATERIALIZED VIEW events_summary AS SELECT id, value FROM events;
```
`events` materializes via `inferSchemas`, which sorts, so it gets `value`.
`events_summary` resolves against `VirtualTableInput`, whose `availableFlows`
is in declaration order, so it sees `Value` and persists `Value` -- Spark takes
the resolved attribute's own name. One clean run therefore leaves the view's
column spelled differently from both the source column it selects and what the
user wrote. Swapping the two `CREATE FLOW` statements then flips the virtual
schema to `value` while the catalog still holds `Value`, and since
`diffSchemas` keys on exact names the next refresh drops and re-adds the column.
Worth noting `availableFlows` is declaration order, not the resolved graph's
completion order, so this is reproducible rather than a race.
If the sort moves here, this docstring can own the rule and
`DataflowGraph.inferSchemas` (line 179) can defer to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
