Matthew Bayer created SPARK-57472:
-------------------------------------
Summary: Make FileTable.mergedOptions merge table and relation
options case-insensitively
Key: SPARK-57472
URL: https://issues.apache.org/jira/browse/SPARK-57472
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3, 4.1.2, 4.2.1
Reporter: Matthew Bayer
FileTable.mergedOptions merges a table's options with the relation's options,
with the relation's taking precedence. The merge is case-sensitive, so when the
table and relation set the same key with different casing (e.g. lineSep vs
linesep), both entries survive and CaseInsensitiveStringMap collapses them by
arbitrary HashMap order, silently dropping one. The documented precedence is
not honored across casing, and the winner is non-deterministic.
Fix: drop any table option the relation already sets (case-insensitively, via
CaseInsensitiveStringMap.containsKey) before merging, so the relation value
deterministically wins. No change for correctly-cased options. Affects
unreleased master only.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
