anew commented on code in PR #57722:
URL: https://github.com/apache/spark/pull/57722#discussion_r3723536028
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraph.scala:
##########
@@ -170,15 +171,27 @@ case class DataflowGraph(
/**
* A map of the inferred schema of each table, computed by merging the
analyzed schemas
* of all flows writing to that table.
+ *
+ * The merge honors the session's `spark.sql.caseSensitive`: under
case-insensitive analysis two
+ * flows emitting column names that differ only in case contribute a single
column rather than
+ * both, which would otherwise produce a target schema the engine's own
resolver cannot
+ * disambiguate. Which of the two spellings survives follows the order the
flows are merged in,
+ * which this map does not define, so callers should not depend on a
particular casing.
Review Comment:
I am ok with sorting by flow name to make to make it deterministic.
Regarding using the rename: it's a good suggestion, but it feels as if this
goes beyond the scope of this PR. Also, since Hive does not support rename,
this might be a significant restriction. Can we do this as a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
