[
https://issues.apache.org/jira/browse/SPARK-59014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-59014:
----------------------------
Issue Type: Improvement (was: Bug)
> DataSourceV2 refresh validation misses a data column that hides a captured
> metadata column
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59014
> URL: https://issues.apache.org/jira/browse/SPARK-59014
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Priority: Minor
>
> When a DataSourceV2 relation is refreshed / re-resolved,
> {{V2TableUtil.validateCapturedMetadataColumns}} validates the captured
> metadata columns against the current table. It compares the captured metadata
> columns only against the metadata columns the connector still reports, so a
> name conflict that arrives on the *data* side is invisible to it.
> If a data column takes the name of a captured metadata column and the
> connector does not rename conflicts
> ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
> {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
> suppresses the metadata column. A captured reference to it can then no longer
> be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
> the read schema onto the relation output by name and collapses the two
> same-named fields onto a single attribute, so a query that asked for the
> metadata column silently returns the *data* column's values.
> *Expected:* refresh / re-resolution fails with a clear error.
> *Actual:* on a suppressing connector with partial column pruning, the query
> silently returns the data column's values where the metadata column was
> expected (a wrong-results bug); with full pruning it happens to return the
> correct value.
> Connectors that rename conflicting metadata columns are unaffected. The
> {{SupportsMetadataColumns}} contract already recommends that a non-renaming
> source reject such a data-column name, but that is only a recommendation and
> is not enforced, so Spark should validate it.
> *Fix:* detect the conflict in the shared {{validateCapturedMetadataColumns}}
> and raise
> {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
> the suppressed case is rejected; the renaming case keeps working.
> PR: https://github.com/apache/spark/pull/58295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
