[
https://issues.apache.org/jira/browse/SPARK-59014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-59014:
----------------------------
Description:
When a DataSourceV2 relation is refreshed / re-resolved,
{{V2TableUtil.validateCapturedMetadataColumns}} validates its captured
metadata-column attributes against the current table. Two gaps could let
incompatible captured metadata references pass validation.
h3. 1. A data column hides a captured metadata column
The existing check compared captured metadata columns only against the metadata
columns the connector still reports, so a name conflict arriving on the *data*
side was invisible to it.
If a data column takes the name of a captured metadata column and the connector
does not rename conflicts
({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
{{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
suppresses the metadata column. A captured reference to it can then no longer
be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
the read schema onto the relation output by name and collapses the two
same-named fields onto a single attribute, so a query that asked for the
metadata column silently returns the *data* column's values.
*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query
silently returns the data column's values where the metadata column was
expected (a wrong-results bug); with full pruning it happens to return the
correct value.
The {{SupportsMetadataColumns}} contract already recommends that a non-renaming
source reject such a data-column name, but that is only a recommendation and is
not enforced, so Spark should validate it.
h3. 2. A renamed captured metadata column is incorrectly matched by its
physical name
A connector that can rename conflicts keeps a conflicting metadata column
reachable under a different physical attribute name. For example, a data column
and metadata column both logically named {{index}} produce this relation shape:
{code}
AttributeReference.name = "_index"
AttributeReference.metadata.__metadata_col = "index"
relation.table.metadataColumns() = [index]
{code}
The internal {{__metadata_col}} entry stores the metadata column's logical
connector name; it is not a Boolean marker. Previously,
{{extractMetadataColumns}} collected {{AttributeReference.name}}, so it looked
for physical name {{_index}} in {{relation.table.metadataColumns()}}. The
connector reports logical name {{index}}, the lookup returned no match, and the
captured metadata column was omitted from schema compatibility validation. A
later type or nullability change to the logical {{index}} metadata column could
therefore pass silently.
*Expected:* the physical rename remains valid, while the captured logical
metadata column continues to be checked for type and nullability changes.
*Actual:* the renamed captured metadata column is dropped from validation
because its physical name does not match the connector's logical name.
Renaming connectors are unaffected by the first issue's shadowing rejection
because their metadata columns remain reachable. The second issue specifically
restores schema validation for those renamed metadata attributes; an unchanged
renamed metadata column remains valid.
h3. Fix
* Detect a still-reported metadata column hidden by a same-named data column in
the shared {{validateCapturedMetadataColumns}} path and raise
{{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
the suppressed case is rejected; the renaming case keeps working.
* Extract captured metadata attributes with
{{MetadataAttributeWithLogicalName}} and match
{{relation.table.metadataColumns()}} using the logical name stored in
{{__metadata_col}}, so type and nullability changes to renamed metadata columns
are validated.
* Correct the outdated internal comments: {{__metadata_col}} stores the logical
column name, while {{__file_source_metadata_col}} is the Boolean marker.
PR: https://github.com/apache/spark/pull/58295
was:
When a DataSourceV2 relation is refreshed / re-resolved,
{{V2TableUtil.validateCapturedMetadataColumns}} validates the captured metadata
columns against the current table. It compares the captured metadata columns
only against the metadata columns the connector still reports, so a name
conflict that arrives on the *data* side is invisible to it.
If a data column takes the name of a captured metadata column and the connector
does not rename conflicts
({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
{{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
suppresses the metadata column. A captured reference to it can then no longer
be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
the read schema onto the relation output by name and collapses the two
same-named fields onto a single attribute, so a query that asked for the
metadata column silently returns the *data* column's values.
*Expected:* refresh / re-resolution fails with a clear error.
*Actual:* on a suppressing connector with partial column pruning, the query
silently returns the data column's values where the metadata column was
expected (a wrong-results bug); with full pruning it happens to return the
correct value.
Connectors that rename conflicting metadata columns are unaffected. The
{{SupportsMetadataColumns}} contract already recommends that a non-renaming
source reject such a data-column name, but that is only a recommendation and is
not enforced, so Spark should validate it.
*Fix:* detect the conflict in the shared {{validateCapturedMetadataColumns}}
and raise
{{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
the suppressed case is rejected; the renaming case keeps working.
PR: https://github.com/apache/spark/pull/58295
> DataSourceV2 refresh validation misses a data column that hides a captured
> metadata column
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-59014
> URL: https://issues.apache.org/jira/browse/SPARK-59014
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Yan Yan
> Priority: Minor
>
> When a DataSourceV2 relation is refreshed / re-resolved,
> {{V2TableUtil.validateCapturedMetadataColumns}} validates its captured
> metadata-column attributes against the current table. Two gaps could let
> incompatible captured metadata references pass validation.
> h3. 1. A data column hides a captured metadata column
> The existing check compared captured metadata columns only against the
> metadata columns the connector still reports, so a name conflict arriving on
> the *data* side was invisible to it.
> If a data column takes the name of a captured metadata column and the
> connector does not rename conflicts
> ({{SupportsMetadataColumns.canRenameConflictingMetadataColumns()}} returns
> {{false}}, the default), {{LogicalPlan.metadataOutputWithOutConflicts}}
> suppresses the metadata column. A captured reference to it can then no longer
> be resolved. On a partially-pruned scan, {{PushDownUtils.toOutputAttrs}} maps
> the read schema onto the relation output by name and collapses the two
> same-named fields onto a single attribute, so a query that asked for the
> metadata column silently returns the *data* column's values.
> *Expected:* refresh / re-resolution fails with a clear error.
> *Actual:* on a suppressing connector with partial column pruning, the query
> silently returns the data column's values where the metadata column was
> expected (a wrong-results bug); with full pruning it happens to return the
> correct value.
> The {{SupportsMetadataColumns}} contract already recommends that a
> non-renaming source reject such a data-column name, but that is only a
> recommendation and is not enforced, so Spark should validate it.
> h3. 2. A renamed captured metadata column is incorrectly matched by its
> physical name
> A connector that can rename conflicts keeps a conflicting metadata column
> reachable under a different physical attribute name. For example, a data
> column and metadata column both logically named {{index}} produce this
> relation shape:
> {code}
> AttributeReference.name = "_index"
> AttributeReference.metadata.__metadata_col = "index"
> relation.table.metadataColumns() = [index]
> {code}
> The internal {{__metadata_col}} entry stores the metadata column's logical
> connector name; it is not a Boolean marker. Previously,
> {{extractMetadataColumns}} collected {{AttributeReference.name}}, so it
> looked for physical name {{_index}} in {{relation.table.metadataColumns()}}.
> The connector reports logical name {{index}}, the lookup returned no match,
> and the captured metadata column was omitted from schema compatibility
> validation. A later type or nullability change to the logical {{index}}
> metadata column could therefore pass silently.
> *Expected:* the physical rename remains valid, while the captured logical
> metadata column continues to be checked for type and nullability changes.
> *Actual:* the renamed captured metadata column is dropped from validation
> because its physical name does not match the connector's logical name.
> Renaming connectors are unaffected by the first issue's shadowing rejection
> because their metadata columns remain reachable. The second issue
> specifically restores schema validation for those renamed metadata
> attributes; an unchanged renamed metadata column remains valid.
> h3. Fix
> * Detect a still-reported metadata column hidden by a same-named data column
> in the shared {{validateCapturedMetadataColumns}} path and raise
> {{INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH}}. Only
> the suppressed case is rejected; the renaming case keeps working.
> * Extract captured metadata attributes with
> {{MetadataAttributeWithLogicalName}} and match
> {{relation.table.metadataColumns()}} using the logical name stored in
> {{__metadata_col}}, so type and nullability changes to renamed metadata
> columns are validated.
> * Correct the outdated internal comments: {{__metadata_col}} stores the
> logical column name, while {{__file_source_metadata_col}} is the Boolean
> marker.
> PR: https://github.com/apache/spark/pull/58295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
