Yan Yan created SPARK-59014:
-------------------------------
Summary: DataSourceV2 refresh validation misses a data column that
hides a captured metadata column
Key: SPARK-59014
URL: https://issues.apache.org/jira/browse/SPARK-59014
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Yan Yan
When a DataSourceV2 relation is refreshed / re-resolved,
{{V2TableUtil.validateCapturedMetadataColumns}} validates the captured metadata
columns against the current table. It compares the captured metadata columns
only against the metadata columns the connector still reports, so a name
conflict that arrives on the *data* side is invisible to it.
If a data column takes the name of a captured metadata column and the connector
does not rename conflicts
({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
{{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
suppresses the metadata column. A captured reference to it can then no longer
be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
the read schema onto the relation output by name and collapses the two
same-named fields onto a single attribute, so a query that asked for the
metadata column silently returns the *data* column's values.
*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query
silently returns the data column's values where the metadata column was
expected (a wrong-results bug); with full pruning it happens to return the
correct value.
Connectors that rename conflicting metadata columns are unaffected. The
{{SupportsMetadataColumns}} contract already recommends that a non-renaming
source reject such a data-column name, but that is only a recommendation and is
not enforced, so Spark should validate it.
*Fix:* detect the conflict in the shared {{validateCapturedMetadataColumns}}
and raise
{{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
the suppressed case is rejected; the renaming case keeps working.
PR: https://github.com/apache/spark/pull/58295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
