Akash3121 commented on code in PR #11178:
URL: https://github.com/apache/paimon/pull/11178#discussion_r4581720310
##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeySimpleTableTest.java:
##########
@@ -2080,11 +2080,11 @@ public void
testPartialUpdateRemoveRecordOnSequenceGroup() throws Exception {
getResult(read, toSplits(snapshotReader.read().dataSplits()),
rowToString);
assertThat(result).containsExactlyInAnyOrder("+I[1, 1, 11, 2, 29, 29,
2]");
- // 2. Update Before
+ // 2. Update Before, a lone -U of the remove-record group seq2 removes
the row (issue 6862)
write.write(GenericRow.ofKind(RowKind.UPDATE_BEFORE, 1, 1, 11, 2, 29,
29, 2));
commit.commit(1, write.prepareCommit(true, 1));
result = getResult(read, toSplits(snapshotReader.read().dataSplits()),
rowToString);
- assertThat(result).containsExactlyInAnyOrder("+I[1, 1, NULL, 2, NULL,
NULL, 2]");
+ assertThat(result).isEmpty();
Review Comment:
In `testPartialUpdateRemoveRecordOnSequenceGroup`,
`partial-update.remove-record-on-sequence-group` is only configured for `seq2`,
while Step 2's `UPDATE_BEFORE(1, 1, 11, 2, 29, 29, 2)` and Step 3's
`UPDATE_AFTER(1, 1, 11, 2, 30, 30, 3)` pass non-null values for both `seq1` (`b
= 11, seq1 = 2`) and `seq2`.
Consider:
1. Passing `null, null` for `b, seq1` in Step 2 and Step 3 (e.g.
`UPDATE_BEFORE(1, 1, null, null, 29, 29, 2)` and `UPDATE_AFTER(1, 1, null,
null, 30, 30, 3)`) so this table-level test verifies that `b = 11, seq1 = 2`
from Step 1 is preserved across `seq2`'s `-U` and `+U` commits rather than
re-supplied by Step 3.
2. Optionally keeping a step where `-U` only retracts `seq1`
(`UPDATE_BEFORE(1, 1, 11, 2, null, null, null)`) to verify at the table level
that `-U` on a non-configured sequence group only clears `b` to `NULL` without
deleting the whole row.
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########
Review Comment:
One subtlety with setting `currentDeleteRow = true` on
`RowKind.UPDATE_BEFORE` (both here and at lines 388–390 in
`retractWithSequenceGroup`):
In `add()`, `-U` intentionally preserves existing columns in `row` so a
later `+U` keeps columns written by other streams, whereas `RowKind.DELETE`
(`-D`) re-initializes `row` via `initRow(row, kv.value())` (line 211 / line
381). However, `getResult()` (line 439) always collapses `currentDeleteRow ==
true` into `RowKind.DELETE`:
```java
RowKind rowKind = currentDeleteRow || !meetInsert ? RowKind.DELETE :
RowKind.INSERT;
```
Because `PartialUpdateMergeFunction` is also used to pre-merge records
inside `SortBufferWriteBuffer` (whenever 2+ records for the same key appear in
the same write buffer) and during minor compactions (`dropDelete = false`), any
pre-merge ending with `-U` is flushed to the L0/compacted file as
`RowKind.DELETE` instead of `RowKind.UPDATE_BEFORE`.
For example:
1. Commit 0 (SST 1): Stream 2 writes `+I(1, null, 20)`.
2. Commit 1 (SST 2): Stream 1 writes `+I(1, 10, null)` followed by `-U(1,
10, null)` in the same write buffer before its `+U` arrives in Commit 2.
`SortBufferWriteBuffer` merges the two records via `PartialUpdateMergeFunction`
and flushes `-D(1, 10, null)` (`RowKind.DELETE`) to SST 2.
3. Commit 2 (SST 3): Stream 1 writes `+U(1, 11, null)`.
4. When SST 1, SST 2, and SST 3 are merged on read/compaction, SST 2 now
has `kv.valueKind() == RowKind.DELETE`, which triggers `initRow(row,
kv.value())` and wipes out `f2 = 20` from SST 1 (producing `+I(1, 11, null)`
instead of `+I(1, 11, 20)`).
By contrast, when `-U` is the sole record for that key in Commit 1's write
buffer, `ReducerMergeFunctionWrapper` bypasses `PartialUpdateMergeFunction` and
writes `RowKind.UPDATE_BEFORE` to SST 2, which preserves `f2 = 20`. Should we
preserve `RowKind.UPDATE_BEFORE` (or avoid wiping `row` when the delete state
came from `-U`) across merge boundaries?
##########
paimon-core/src/test/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunctionTest.java:
##########
@@ -1539,6 +1539,78 @@ public void
testRemoveRecordOnDeleteIgnoresUpdateBeforeUntilInsert() {
validate(function, 1, 20);
}
+ @Test
+ public void testRemoveRecordOnDeleteWithTrailingUpdateBefore() {
+ Options options = new Options();
+ options.set("partial-update.remove-record-on-delete", "true");
+ MergeFunction<KeyValue> function =
+ PartialUpdateMergeFunction.factory(
+ options,
+ RowType.of(DataTypes.INT(), DataTypes.INT(),
DataTypes.INT()),
+ ImmutableList.of("f0"))
+ .create();
+ function.reset();
+ add(function, 1, 10, null);
+ add(function, 1, null, 20);
+
+ // issue 6862: a filter in front of the sink drops the +U, so the lone
-U is the last
+ // message of the key and the row must be removed instead of keeping
the stale value
+ add(function, RowKind.UPDATE_BEFORE, 1, 10, null);
+ assertThat(function.getResult().valueKind()).isEqualTo(RowKind.DELETE);
+ // unlike DELETE, -U does not re-initialize the row, so the DELETE
keeps the merged row
+ validate(function, 1, 10, 20);
+
+ // a following +U still updates the merged row, so f2 written by
another writer is kept
+ add(function, RowKind.UPDATE_AFTER, 1, 11, null);
+ assertThat(function.getResult().valueKind()).isEqualTo(RowKind.INSERT);
+ validate(function, 1, 11, 20);
+ }
+
+ @Test
+ public void testRemoveRecordOnDeleteWithOnlyUpdateBefore() {
+ Options options = new Options();
+ options.set("partial-update.remove-record-on-delete", "true");
+ MergeFunction<KeyValue> function =
+ PartialUpdateMergeFunction.factory(
+ options,
+ RowType.of(DataTypes.INT(), DataTypes.INT()),
+ ImmutableList.of("f0"))
+ .create();
+ function.reset();
+
+ // A key that receives only one -U, with no INSERT before it, should
still produce a
+ // DELETE. The remove-record-on-delete docs say UPDATE_BEFORE removes
the whole row.
+ add(function, RowKind.UPDATE_BEFORE, 1, 10);
Review Comment:
Nit: when `add(function, RowKind.UPDATE_BEFORE, 1, 10)` is called with no
prior `INSERT`/`UPDATE_AFTER`, `meetInsert` remains `false`, so `getResult()`
(`currentDeleteRow || !meetInsert ? RowKind.DELETE : RowKind.INSERT`) already
returned `RowKind.DELETE` before this change. Keeping the test is fine, though
the comment might clarify that `!meetInsert` also covers a lone `-U` with no
prior insert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]