JingsongLi commented on code in PR #1401:
URL: https://github.com/apache/paimon-rust/pull/1401#discussion_r4581250759


##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1707,6 +1722,32 @@ fn sort_merge_stream(
                 tree.update(|a, b| compare_cursors(&cursors, a, 
b).then_with(|| a.cmp(&b)).is_gt());
             }
 
+            // Filter expired compact-input rows before lookup inject / merge
+            // (Java RecordLevelExpire wraps the compact reader so Lookup never
+            // sees expired L0). Historical LevelLookup rows are filtered by
+            // ExpiringLookup when a surviving L0 key still needs before-state.
+            if let Some((expire, now_secs)) = &input_expire {
+                let mut kept = Vec::with_capacity(same_key_rows.len());
+                for row in same_key_rows.drain(..) {
+                    if expire.keeps_merge_row(
+                        &batch_buffer,
+                        &row,
+                        &source_output_col_indices,
+                        &output_schema,
+                        *now_secs,
+                    )? {
+                        kept.push(row);
+                    }
+                }
+                same_key_rows = kept;
+            }
+
+            // Java reader wrap means a fully-expired key never enters merge, 
so
+            // Lookup is not consulted and history is left untouched.
+            if same_key_rows.is_empty() {
+                continue;

Review Comment:
   [P1] Release consumed batches when an entire key expires
   
   This continue skips the `compact_batch_buffer` path below. With no pending 
output, the ordinary `MergeResult::Omit` path releases consumed source batches, 
but a scan of entirely expired keys now keeps every decoded batch in 
`batch_buffer` until the scan ends. A lazy 32-batch × 64-row probe using only 
Weak array references measured peak live batches of 2 for the Omit control and 
32 with input expiration; both produce zero output rows. Thus an expire-only 
rewrite of a large historical file grows with the entire decoded input rather 
than the streaming window and can OOM precisely when it should be discarding 
data. Route empty expired groups through cleanup too, preserving any pending 
output references, and add a multi-batch lifetime regression.



##########
crates/paimon/src/table/compact/primary_key/planner.rs:
##########
@@ -227,10 +227,63 @@ pub(in crate::table::compact) fn 
select_with_force_level_zero(
     Ok(Some((selected, output_level)))
 }
 
+/// Incremental run selection used by execute/commit validation.
+///
+/// Matches planning: prefer UniversalCompaction, then fall back to an
+/// expire-only max-level rewrite when value stats prove every record expired.
+pub(in crate::table::compact) fn select_or_expire(
+    files: Vec<DataFileMeta>,
+    schema: &TableSchema,
+) -> Result<Option<(Vec<DataFileMeta>, i32)>> {
+    if let Some(selected) = select(files.clone(), schema.options())? {

Review Comment:
   [P1] Validate automatic expiration with the same L0 selection policy
   
   Automatic planning calls `select_with_force_level_zero(..., false)` by 
default. If an expired max-level run coexists with a new L0 below the run-count 
trigger, it falls back to an expire-only history task. This validator instead 
calls `select`, which forces L0 promotion, so it selects a different input set 
and rejects its own task.
   
   Public API reproduction: bucket=1, num-levels=3, compaction trigger=5; seed 
and compact an old row to L2 with expiration off; enable 
expire-time=1s/time-field=ts and write-only=false; append a live second key. 
The commit returns `Data commit ... succeeded, but automatic compaction failed` 
caused by `PK compaction inputs do not match incremental run selection`; 
snapshot advances 2→3 and both L2/L0 remain. This makes ordinary writes report 
failure after publication. Persist/reuse the planner's selection mode or 
validate expire-only task membership separately; cover this 
automatic-maintenance case.



##########
crates/paimon/src/table/compact/primary_key.rs:
##########
@@ -381,13 +396,36 @@ pub(super) async fn execute(
     let lookup_changelog = options.try_changelog_producer()? == 
ChangelogProducer::Lookup;
     let output_level = task.output_level();
     let drop_delete = options.deletion_vectors_enabled() && output_level > 0;
-    let reader = new_reader(table, resources.clone())?.with_lookup(lookup);
+    // Java wraps both compact and LevelLookup readers with RecordLevelExpire 
so
+    // LookupChangelogMergeFunction never observes expiration-only deletes.
+    let expire = 
crate::table::record_level_expire::RecordLevelExpire::create(schema)?;
+    let now_secs = crate::table::record_level_expire::current_epoch_secs();
+    let lookup = match (lookup, &expire) {
+        (Some(provider), Some(expire)) => {

Review Comment:
   [P1] Preserve accepted first-row lookup history until it is retired
   
   The expiration wrapper is applied to first-row history as well, so an 
expired accepted row is treated as absent even when its history file remains 
outside the current L0 rewrite. I reproduced this on a table created with 
`merge-engine=first-row` and `changelog-producer=lookup`: seed a large 
max-level history containing key 1/value 10/ts 1, enable expire-time=1s, append 
key 1/value 11 with a live timestamp, then compact only L0. Ordinary query 
still returns value 10 from the retained first-row history, but the incremental 
changelog emits `+I(1,11)`. A downstream consumer therefore diverges from the 
table's current state.
   
   Java explicitly uses an unfiltered, key-only lookup reader for first-row in 
[MergeTreeCompactManagerFactory](https://github.com/apache/paimon/blob/f289b0a2829f8b0c28ae92743f11893893ce1116/paimon-core/src/main/java/org/apache/paimon/mergetree/compact/MergeTreeCompactManagerFactory.java#L334-L346).
 Keep accepted-history lookup behavior for first-row, or retire that history 
coherently before emitting a new INSERT; add a current-state/changelog 
consistency regression.



##########
crates/paimon/src/table/compact/primary_key/planner.rs:
##########
@@ -227,10 +227,63 @@ pub(in crate::table::compact) fn 
select_with_force_level_zero(
     Ok(Some((selected, output_level)))
 }
 
+/// Incremental run selection used by execute/commit validation.
+///
+/// Matches planning: prefer UniversalCompaction, then fall back to an
+/// expire-only max-level rewrite when value stats prove every record expired.
+pub(in crate::table::compact) fn select_or_expire(
+    files: Vec<DataFileMeta>,
+    schema: &TableSchema,
+) -> Result<Option<(Vec<DataFileMeta>, i32)>> {
+    if let Some(selected) = select(files.clone(), schema.options())? {
+        return Ok(Some(selected));
+    }
+    select_expired_max_level(&files, schema)

Review Comment:
   [P2] Keep an unchanged expiration task valid as time advances
   
   Execution and commit validation rerun this selection using `is_expire_file`, 
which reads a fresh wall clock for each call, then require exactly the original 
task's file set. A file that becomes eligible after planning changes the 
recomputed set and invalidates an otherwise immutable task.
   
   Public API probe: two max-level files, A already expired and B just inside 
its retention window; plan the A-only task, perform no writes or metadata 
changes, wait three seconds until B expires, then execute the captured task. It 
fails with `PK compaction inputs do not match incremental run selection`. The 
same drift can happen during a long execution before commit. Capture/reuse a 
selection watermark, or verify that the selected inputs remain valid without 
requiring every newly expired file to join the task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to