anew commented on PR #57722: URL: https://github.com/apache/spark/pull/57722#issuecomment-5182750899
Multi-flow inference still merges case-sensitively: - `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraph.scala:180` still calls `SchemaMergingUtils.mergeSchemas(_, _)`, which defaults to `caseSensitive = true`. - That inferred schema is then used as the table output schema in `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala:308`. - So under `spark.sql.caseSensitive=false`, two valid streaming append flows to the same table emitting value and Value can still infer a target schema containing both columns before the fixed evolveTable path helps. The same default also remains in `SchemaInferenceUtils.inferSchemaFromFlows` for declared-schema validation, so a declared `value` plus flow `Value` can still be rejected case-sensitively. This leaves an externally reachable part of SDP schema evolution inconsistent with the PR’s contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
