Akash3121 commented on code in PR #11178:
URL: https://github.com/apache/paimon/pull/11178#discussion_r4581720310


##########
paimon-core/src/test/java/org/apache/paimon/table/PrimaryKeySimpleTableTest.java:
##########
@@ -2080,11 +2080,11 @@ public void 
testPartialUpdateRemoveRecordOnSequenceGroup() throws Exception {
                 getResult(read, toSplits(snapshotReader.read().dataSplits()), 
rowToString);
         assertThat(result).containsExactlyInAnyOrder("+I[1, 1, 11, 2, 29, 29, 
2]");
 
-        // 2. Update Before
+        // 2. Update Before, a lone -U of the remove-record group seq2 removes 
the row (issue 6862)
         write.write(GenericRow.ofKind(RowKind.UPDATE_BEFORE, 1, 1, 11, 2, 29, 
29, 2));
         commit.commit(1, write.prepareCommit(true, 1));
         result = getResult(read, toSplits(snapshotReader.read().dataSplits()), 
rowToString);
-        assertThat(result).containsExactlyInAnyOrder("+I[1, 1, NULL, 2, NULL, 
NULL, 2]");
+        assertThat(result).isEmpty();

Review Comment:
   In `testPartialUpdateRemoveRecordOnSequenceGroup`, 
`partial-update.remove-record-on-sequence-group` is only configured for `seq2`, 
while Step 2's `UPDATE_BEFORE(1, 1, 11, 2, 29, 29, 2)` and Step 3's 
`UPDATE_AFTER(1, 1, 11, 2, 30, 30, 3)` pass non-null values for both `seq1` (`b 
= 11, seq1 = 2`) and `seq2`.
    
    Consider:
    1. Passing `null, null` for `b, seq1` in Step 2 and Step 3 (e.g. 
`UPDATE_BEFORE(1, 1, null, null, 29, 29, 2)` and `UPDATE_AFTER(1, 1, null, 
null, 30, 30, 3)`) so this table-level test verifies that `b = 11, seq1 = 2` 
from Step 1 is preserved across `seq2`'s `-U` and `+U` commits rather than 
re-supplied by Step 3.
    2. Optionally keeping a step where `-U` only retracts `seq1` 
(`UPDATE_BEFORE(1, 1, 11, 2, null, null, null)`) to verify at the table level 
that `-U` on a non-configured sequence group only clears `b` to `NULL` without 
deleting the whole row.



##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunction.java:
##########


Review Comment:
   One subtlety with setting `currentDeleteRow = true` on 
`RowKind.UPDATE_BEFORE` (both here and at lines 388–390 in 
`retractWithSequenceGroup`):
    
    In `add()`, `-U` intentionally preserves existing columns in `row` so a 
later `+U` keeps columns written by other streams, whereas `RowKind.DELETE` 
(`-D`) re-initializes `row` via `initRow(row, kv.value())` (line 211 / line 
381). However, `getResult()` (line 439) always collapses `currentDeleteRow == 
true` into `RowKind.DELETE`:
    
    ```java
    RowKind rowKind = currentDeleteRow || !meetInsert ? RowKind.DELETE : 
RowKind.INSERT;
    ```
    
    Because `PartialUpdateMergeFunction` is also used to pre-merge records 
inside `SortBufferWriteBuffer` (whenever 2+ records for the same key appear in 
the same write buffer) and during minor compactions (`dropDelete = false`), any 
pre-merge ending with `-U` is flushed to the L0/compacted file as 
`RowKind.DELETE` instead of `RowKind.UPDATE_BEFORE`.
    
    For example:
    1. Commit 0 (SST 1): Stream 2 writes `+I(1, null, 20)`.
    2. Commit 1 (SST 2): Stream 1 writes `+I(1, 10, null)` followed by `-U(1, 
10, null)` in the same write buffer before its `+U` arrives in Commit 2. 
`SortBufferWriteBuffer` merges the two records via `PartialUpdateMergeFunction` 
and flushes `-D(1, 10, null)` (`RowKind.DELETE`) to SST 2.
    3. Commit 2 (SST 3): Stream 1 writes `+U(1, 11, null)`.
    4. When SST 1, SST 2, and SST 3 are merged on read/compaction, SST 2 now 
has `kv.valueKind() == RowKind.DELETE`, which triggers `initRow(row, 
kv.value())` and wipes out `f2 = 20` from SST 1 (producing `+I(1, 11, null)` 
instead of `+I(1, 11, 20)`).
    
    By contrast, when `-U` is the sole record for that key in Commit 1's write 
buffer, `ReducerMergeFunctionWrapper` bypasses `PartialUpdateMergeFunction` and 
writes `RowKind.UPDATE_BEFORE` to SST 2, which preserves `f2 = 20`. Should we 
preserve `RowKind.UPDATE_BEFORE` (or avoid wiping `row` when the delete state 
came from `-U`) across merge boundaries?
   



##########
paimon-core/src/test/java/org/apache/paimon/mergetree/compact/PartialUpdateMergeFunctionTest.java:
##########
@@ -1539,6 +1539,78 @@ public void 
testRemoveRecordOnDeleteIgnoresUpdateBeforeUntilInsert() {
         validate(function, 1, 20);
     }
 
+    @Test
+    public void testRemoveRecordOnDeleteWithTrailingUpdateBefore() {
+        Options options = new Options();
+        options.set("partial-update.remove-record-on-delete", "true");
+        MergeFunction<KeyValue> function =
+                PartialUpdateMergeFunction.factory(
+                                options,
+                                RowType.of(DataTypes.INT(), DataTypes.INT(), 
DataTypes.INT()),
+                                ImmutableList.of("f0"))
+                        .create();
+        function.reset();
+        add(function, 1, 10, null);
+        add(function, 1, null, 20);
+
+        // issue 6862: a filter in front of the sink drops the +U, so the lone 
-U is the last
+        // message of the key and the row must be removed instead of keeping 
the stale value
+        add(function, RowKind.UPDATE_BEFORE, 1, 10, null);
+        assertThat(function.getResult().valueKind()).isEqualTo(RowKind.DELETE);
+        // unlike DELETE, -U does not re-initialize the row, so the DELETE 
keeps the merged row
+        validate(function, 1, 10, 20);
+
+        // a following +U still updates the merged row, so f2 written by 
another writer is kept
+        add(function, RowKind.UPDATE_AFTER, 1, 11, null);
+        assertThat(function.getResult().valueKind()).isEqualTo(RowKind.INSERT);
+        validate(function, 1, 11, 20);
+    }
+
+    @Test
+    public void testRemoveRecordOnDeleteWithOnlyUpdateBefore() {
+        Options options = new Options();
+        options.set("partial-update.remove-record-on-delete", "true");
+        MergeFunction<KeyValue> function =
+                PartialUpdateMergeFunction.factory(
+                                options,
+                                RowType.of(DataTypes.INT(), DataTypes.INT()),
+                                ImmutableList.of("f0"))
+                        .create();
+        function.reset();
+
+        // A key that receives only one -U, with no INSERT before it, should 
still produce a
+        // DELETE. The remove-record-on-delete docs say UPDATE_BEFORE removes 
the whole row.
+        add(function, RowKind.UPDATE_BEFORE, 1, 10);

Review Comment:
   Nit: when `add(function, RowKind.UPDATE_BEFORE, 1, 10)` is called with no 
prior `INSERT`/`UPDATE_AFTER`, `meetInsert` remains `false`, so `getResult()` 
(`currentDeleteRow || !meetInsert ? RowKind.DELETE : RowKind.INSERT`) already 
returned `RowKind.DELETE` before this change. Keeping the test is fine, though 
the comment might clarify that `!meetInsert` also covers a lone `-U` with no 
prior insert.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to