Ryu Kobayashi created HIVE-30147:
------------------------------------
Summary: Align Iceberg copy-on-write MERGE semantics with native
MERGE
Key: HIVE-30147
URL: https://issues.apache.org/jira/browse/HIVE-30147
Project: Hive
Issue Type: Bug
Reporter: Ryu Kobayashi
h2. Summary
The Iceberg copy-on-write MERGE rewrite does not fully preserve the semantics
of the native MERGE implementation.
The affected areas include NULL-key handling, source-only row detection, SQL
three-valued predicate evaluation, `WHEN MATCHED` clause ordering, and
cardinality-violation handling. These issues should be corrected so that
copy-on-write MERGE behaves consistently with native MERGE.
h2. Root cause
The current implementation in `CopyOnWriteMergeRewriter` rewrites MERGE
statements into a source CTE and multiple `UNION ALL` branches.
Several parts of the rewrite rely on derived predicates and string-level
predicate composition:
- `MergeSemanticAnalyzer.OnClauseAnalyzer` derives source-only predicates from
NULL checks on target join columns.
- `CopyOnWriteMergeRewriter` reuses those predicates for the INSERT and
carry-over branches.
- Target/source presence is not represented by an explicit join-side marker
such as the target file path.
- Multiple predicates are concatenated with `AND` and `OR`, which can change
their effective precedence when the original expressions contain OR operators.
- Later `WHEN MATCHED` clauses are gated with `NOT(previousPredicate)`, which
does not implement SQL `IS NOT TRUE` semantics when the previous predicate
evaluates to `UNKNOWN`.
- The copy-on-write rewrite does not currently use the standard
cardinality-violation branch implemented by `MergeRewriter`.
h2. Confirmed / observed behavior
The following cases require regression coverage and correction:
1. A target row whose join key is NULL can be mistaken for a source-only row
when another row in the same data file is matched and the file is rewritten.
2. A target-only row with NULL join keys can satisfy the derived source-only
predicate and produce an unintended INSERT.
3. A matched predicate that evaluates to `UNKNOWN` can prevent a later `WHEN
MATCHED` clause from executing.
4. Unparenthesized combinations of `ON` predicates and clause predicates can
cause source-only rows or unrelated target rows to enter the replacement-marker
path.
5. Multiple source rows matching the same target row can produce duplicate CoW
updates or incorrect affected-row behavior instead of the standard cardinality
violation.
6. On constrained targets, the full-width CoW cardinality branch may interact
with NOT NULL/CHECK enforcement before the intended cardinality error is
reported. This requires explicit validation with both CBO enabled and disabled.
h2. Impact / exposure
These issues affect the correctness and compatibility of Iceberg copy-on-write
MERGE statements.
Potential effects include:
- Silent loss of legitimate target rows.
- Unintended insertion of NULL or source-derived rows.
- Different results from native MERGE for predicates involving NULL or OR
expressions.
- Duplicate or nondeterministic updates when source cardinality is invalid.
- Cardinality errors being masked by constraint violations.
- Inconsistent behavior depending on CBO configuration.
This issue is separate from HIVE-30064's affected-row reporting scope.
HIVE-30064 tracks logical affected-row counts for CoW DML; this issue tracks
the broader correctness and semantic alignment of CoW MERGE rewriting.
The implementation should preserve native MERGE semantics and include
regression coverage for the affected cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
