[
https://issues.apache.org/jira/browse/SPARK-60111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-60111:
----------------------------
Description:
DataSourceV2Relation can be rebuilt without carrying its TreeNode tags in two
paths.
h3. Expected behavior
Rebuilding the same logical relation should preserve its TreeNode tags,
consistent with TreeNode transformation semantics and
LogicalRelation.withMetadataColumns.
h3. Actual behavior
DataSourceV2Relation.withMetadataColumns and the DataSource V2 target-cleaning
branch in Analyzer.ResolveOutputRelation use the case-class copy method
directly. The generated copy method does not call copyTagsFrom, so tags on the
original relation are absent from the rebuilt relation.
h3. Minimal reproduction
Create a DataSourceV2Relation, set a TreeNodeTag on it, and then either add a
metadata column with withMetadataColumns or run ResolveOutputRelation on a
write whose query needs projection. In both cases the returned relation is a
new object and getTagValue returns None.
h3. Root cause and fix
These paths bypass TreeNode.makeCopy and the transform helpers that normally
preserve tags. Explicitly copy tags from the original relation after rebuilding
it, and add regression tests for both paths.
PR: https://github.com/apache/spark/pull/59330
was:
DataSourceV2Relation can be rebuilt without carrying its TreeNode tags in two
paths.
h3. Expected behavior
Rebuilding the same logical relation should preserve its TreeNode tags,
consistent with TreeNode transformation semantics and
LogicalRelation.withMetadataColumns.
h3. Actual behavior
DataSourceV2Relation.withMetadataColumns and the DataSource V2 target-cleaning
branch in Analyzer.ResolveOutputRelation use the case-class copy method
directly. The generated copy method does not call copyTagsFrom, so tags on the
original relation are absent from the rebuilt relation.
h3. Minimal reproduction
Create a DataSourceV2Relation, set a TreeNodeTag on it, and then either add a
metadata column with withMetadataColumns or run ResolveOutputRelation on a
write whose query needs projection. In both cases the returned relation is a
new object and getTagValue returns None.
h3. Root cause and fix
These paths bypass TreeNode.makeCopy and the transform helpers that normally
preserve tags. Explicitly copy tags from the original relation after rebuilding
it, and add regression tests for both paths.
> Preserve tags when rebuilding DataSource V2 relations
> -----------------------------------------------------
>
> Key: SPARK-60111
> URL: https://issues.apache.org/jira/browse/SPARK-60111
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Yan Yan
> Priority: Minor
> Labels: pull-request-available
>
> DataSourceV2Relation can be rebuilt without carrying its TreeNode tags in two
> paths.
> h3. Expected behavior
> Rebuilding the same logical relation should preserve its TreeNode tags,
> consistent with TreeNode transformation semantics and
> LogicalRelation.withMetadataColumns.
> h3. Actual behavior
> DataSourceV2Relation.withMetadataColumns and the DataSource V2
> target-cleaning branch in Analyzer.ResolveOutputRelation use the case-class
> copy method directly. The generated copy method does not call copyTagsFrom,
> so tags on the original relation are absent from the rebuilt relation.
> h3. Minimal reproduction
> Create a DataSourceV2Relation, set a TreeNodeTag on it, and then either add a
> metadata column with withMetadataColumns or run ResolveOutputRelation on a
> write whose query needs projection. In both cases the returned relation is a
> new object and getTagValue returns None.
> h3. Root cause and fix
> These paths bypass TreeNode.makeCopy and the transform helpers that normally
> preserve tags. Explicitly copy tags from the original relation after
> rebuilding it, and add regression tests for both paths.
> PR: https://github.com/apache/spark/pull/59330
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
