[
https://issues.apache.org/jira/browse/HIVE-30160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryu Kobayashi updated HIVE-30160:
---------------------------------
Description:
h2. Summary
Iceberg copy-on-write MERGE does not currently perform the standard
cardinality-violation validation used by native MERGE.
When multiple source rows match the same target row, the copy-on-write rewrite
can apply multiple actions to the same target row instead of raising the
standard cardinality violation error. This behavior should be aligned with
native MERGE.
h2. Root cause
`CopyOnWriteMergeRewriter` currently does not add the standard
cardinality-validation branch implemented by `MergeRewriter`. The native MERGE
rewrite uses the `cardinality_violation` UDF and a dedicated branch writing to
the session-scoped `merge_tmp_table`.
The copy-on-write rewrite instead emits action and replacement-marker branches
directly into the target INSERT/UNION plan. Without a dedicated
cardinality-validation branch, duplicate source matches can flow through the
normal update or delete branches.
The cardinality validation must also account for:
- Iceberg spec ID and partition identity
- Data-file path and row position
- Partitioned and partition-evolved tables
- Row-lineage projections
- CBO-enabled and CBO-disabled execution
- NOT NULL and CHECK constraint processing
The validation branch must not be implemented as a full-width target row
containing NULL values if that would cause constraint enforcement to mask the
intended cardinality error.
h2. Reproduction steps
Create an Iceberg v2 copy-on-write target, insert one row, create a source
containing two rows with the same matching key, and run a matched-update MERGE.
The duplicate source matches should expose the issue.
{code:java}
Create an Iceberg v2 table configured for copy-on-write MERGE:```sql
CREATE TABLE target (a int, b string)
STORED BY ICEBERG
TBLPROPERTIES (
'format-version'='2',
'write.merge.mode'='copy-on-write'
);
Insert one target row and two source rows with the same matching key:INSERT
INTO target VALUES (1, 'original');CREATE TABLE source (a int, b string);INSERT
INTO source VALUES
(1, 'first'),
(1, 'second');
MERGE INTO target t
USING source s
ON t.a = s.a
WHEN MATCHED THEN
UPDATE SET b = s.b;{code}
h2. Impact / exposure
Without cardinality validation, multiple source rows can modify the same target
row in a single copy-on-write MERGE statement.
Potential effects include:
* The target row being updated or deleted more than once.
* The final result depending on execution order.
* Affected-row counts being overstated.
* Different results between copy-on-write and native MERGE.
* A cardinality error being masked by NOT NULL or CHECK constraint enforcement.
Insert-only MERGE statements are out of scope because they do not modify
matched target rows.
was:
h2. Summary
Iceberg copy-on-write MERGE does not currently perform the standard
cardinality-violation validation used by native MERGE.
When multiple source rows match the same target row, the copy-on-write rewrite
can apply multiple actions to the same target row instead of raising the
standard cardinality violation error. This behavior should be aligned with
native MERGE.
h2. Root cause
`CopyOnWriteMergeRewriter` currently does not add the standard
cardinality-validation branch implemented by `MergeRewriter`. The native MERGE
rewrite uses the `cardinality_violation` UDF and a dedicated branch writing to
the session-scoped `merge_tmp_table`.
The copy-on-write rewrite instead emits action and replacement-marker branches
directly into the target INSERT/UNION plan. Without a dedicated
cardinality-validation branch, duplicate source matches can flow through the
normal update or delete branches.
The cardinality validation must also account for:
- Iceberg spec ID and partition identity
- Data-file path and row position
- Partitioned and partition-evolved tables
- Row-lineage projections
- CBO-enabled and CBO-disabled execution
- NOT NULL and CHECK constraint processing
The validation branch must not be implemented as a full-width target row
containing NULL values if that would cause constraint enforcement to mask the
intended cardinality error.
h2. Reproduction steps
# Create an Iceberg v2 table configured for copy-on-write MERGE:
{code:java}
CREATE TABLE target (a int, b string) STORED BY ICEBERG TBLPROPERTIES
('format-version'='2', 'write.merge.mode'='copy-on-write'); {code}
# Insert one target row and duplicate source keys:
{code:java}
INSERT INTO target VALUES (1, 'original'); CREATE TABLE source (a int, b
string); INSERT INTO source VALUES (1, 'first'), (1, 'second');{code}
# Execute the MERGE:
{code:java}
MERGE INTO target t
USING source s
ON t.a = s.a
WHEN MATCHED THEN UPDATE SET b = s.b;{code}
h2. Impact / exposure
Without cardinality validation:
- A target row can be updated more than once by a single MERGE statement.
- The final value may depend on execution order.
- The reported affected-row count can be inflated.
- Copy-on-write and native MERGE can produce different results for the same
input.
- Target constraints may mask the intended cardinality error if validation is
added at the wrong point in the plan.
The issue affects Iceberg copy-on-write MERGE statements with duplicate source
matches. Insert-only MERGE statements should retain their existing behavior
because they do not modify matched target rows.
> Add cardinality violation handling to Iceberg copy-on-write MERGE
> -----------------------------------------------------------------
>
> Key: HIVE-30160
> URL: https://issues.apache.org/jira/browse/HIVE-30160
> Project: Hive
> Issue Type: Bug
> Reporter: Ryu Kobayashi
> Assignee: Ryu Kobayashi
> Priority: Major
>
> h2. Summary
> Iceberg copy-on-write MERGE does not currently perform the standard
> cardinality-violation validation used by native MERGE.
> When multiple source rows match the same target row, the copy-on-write
> rewrite can apply multiple actions to the same target row instead of raising
> the standard cardinality violation error. This behavior should be aligned
> with native MERGE.
> h2. Root cause
> `CopyOnWriteMergeRewriter` currently does not add the standard
> cardinality-validation branch implemented by `MergeRewriter`. The native
> MERGE rewrite uses the `cardinality_violation` UDF and a dedicated branch
> writing to the session-scoped `merge_tmp_table`.
> The copy-on-write rewrite instead emits action and replacement-marker
> branches directly into the target INSERT/UNION plan. Without a dedicated
> cardinality-validation branch, duplicate source matches can flow through the
> normal update or delete branches.
> The cardinality validation must also account for:
> - Iceberg spec ID and partition identity
> - Data-file path and row position
> - Partitioned and partition-evolved tables
> - Row-lineage projections
> - CBO-enabled and CBO-disabled execution
> - NOT NULL and CHECK constraint processing
> The validation branch must not be implemented as a full-width target row
> containing NULL values if that would cause constraint enforcement to mask the
> intended cardinality error.
> h2. Reproduction steps
> Create an Iceberg v2 copy-on-write target, insert one row, create a source
> containing two rows with the same matching key, and run a matched-update
> MERGE. The duplicate source matches should expose the issue.
> {code:java}
> Create an Iceberg v2 table configured for copy-on-write MERGE:```sql
> CREATE TABLE target (a int, b string)
> STORED BY ICEBERG
> TBLPROPERTIES (
> 'format-version'='2',
> 'write.merge.mode'='copy-on-write'
> );
> Insert one target row and two source rows with the same matching key:INSERT
> INTO target VALUES (1, 'original');CREATE TABLE source (a int, b
> string);INSERT INTO source VALUES
> (1, 'first'),
> (1, 'second');
> MERGE INTO target t
> USING source s
> ON t.a = s.a
> WHEN MATCHED THEN
> UPDATE SET b = s.b;{code}
> h2. Impact / exposure
> Without cardinality validation, multiple source rows can modify the same
> target row in a single copy-on-write MERGE statement.
> Potential effects include:
> * The target row being updated or deleted more than once.
> * The final result depending on execution order.
> * Affected-row counts being overstated.
> * Different results between copy-on-write and native MERGE.
> * A cardinality error being masked by NOT NULL or CHECK constraint
> enforcement.
> Insert-only MERGE statements are out of scope because they do not modify
> matched target rows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
