Anton Okolnychyi created SPARK-57544:
----------------------------------------
Summary: Rework column ID validation for nested fields in DSv2
Key: SPARK-57544
URL: https://issues.apache.org/jira/browse/SPARK-57544
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Anton Okolnychyi
The original implementation detected dropped-and-re-added columns by comparing
top-level Column.id() strings in a dedicated validateColumnIds pass, but this
approach had no visibility into nested struct fields, array elements, or map
keys/values. To work around this limitation, connectors had to encode nested
field IDs into the top-level ID string (as demonstrated by
ComposedColumnIdTableCatalog), placing an unreasonable burden on connector
authors and making the feature fragile by design.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
