yyanyy opened a new pull request, #58295: URL: https://github.com/apache/spark/pull/58295
### What changes were proposed in this pull request? `V2TableUtil.validateCapturedMetadataColumns` now detects when a data column has taken the name of a captured metadata column and reports a user-facing error, instead of letting the conflict pass silently. DSv2 relations capture their metadata-column attributes at analysis time and re-validate them when the table is later refreshed / re-resolved. The existing check compared the captured metadata columns only against the metadata columns the connector *still reports*, so a conflict that arrives on the *data* side was invisible. When a data column takes a captured metadata column's name: - If the connector renames conflicting metadata columns (`SupportsMetadataColumns.canRenameConflictingMetadataColumns()` returns `true`), the metadata column stays reachable (renamed) and everything keeps working — this PR does not change that path. - If the connector suppresses the conflict (the default, `false`), `LogicalPlan.metadataOutputWithOutConflicts` drops the metadata column. A captured reference to it can no longer be resolved, and on a partially-pruned scan `PushDownUtils.toOutputAttrs` resolves the read schema by name and collapses the two same-named fields onto one attribute, so a query for the metadata column silently returns the *data* column's values. The new check lives in the shared `validateCapturedMetadataColumns`, so it covers all three callers (metadata refresh, transactional-write re-resolution, and dataframe temp-view-with-plan re-resolution). It fires only for the suppressed case, and only for metadata columns the current table still reports — a column the connector no longer reports is already flagged as *removed* by the existing schema comparison. The error is surfaced through the existing `INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH` condition. ### Why are the changes needed? On a connector that suppresses metadata/data name conflicts (the default), shadowing a captured metadata column with a same-named data column produces a silent wrong result on partially-pruned scans: the data column's values are returned where the query asked for the metadata column. Failing with a clear error is far better than returning wrong data. The `SupportsMetadataColumns` contract already recommends that non-renaming sources reject such data columns. ### Does this PR introduce _any_ user-facing change? Yes. A relation that referenced a metadata column and is later re-resolved against a table where a data column has taken that name now fails with `INCOMPATIBLE_TABLE_CHANGE_AFTER_ANALYSIS.METADATA_COLUMNS_MISMATCH` (message: "`<name>` metadata column is hidden by a data column with the same name"). Previously it either returned wrong results (partially-pruned scans on suppressing connectors) or happened to work depending on pruning. Connectors that rename conflicting metadata columns are unaffected. This tightens a validation gap on the unreleased master branch. ### How was this patch tested? Added unit tests to `V2TableUtilSuite`: a suppressed conflict is rejected (in both `ALLOW_NEW_FIELDS` and `PROHIBIT_CHANGES` modes), detection is case-insensitive, a renamable connector is not affected, and a metadata column the connector no longer reports is reported as removed (not hidden). `catalyst/testOnly *V2TableUtilSuite` passes (53 tests, 0 failures); `catalyst/scalastyle` and `catalyst/Test/scalastyle` are clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
