JingsongLi commented on code in PR #1401:
URL: https://github.com/apache/paimon-rust/pull/1401#discussion_r4581250759
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1707,6 +1722,32 @@ fn sort_merge_stream(
tree.update(|a, b| compare_cursors(&cursors, a,
b).then_with(|| a.cmp(&b)).is_gt());
}
+ // Filter expired compact-input rows before lookup inject / merge
+ // (Java RecordLevelExpire wraps the compact reader so Lookup never
+ // sees expired L0). Historical LevelLookup rows are filtered by
+ // ExpiringLookup when a surviving L0 key still needs before-state.
+ if let Some((expire, now_secs)) = &input_expire {
+ let mut kept = Vec::with_capacity(same_key_rows.len());
+ for row in same_key_rows.drain(..) {
+ if expire.keeps_merge_row(
+ &batch_buffer,
+ &row,
+ &source_output_col_indices,
+ &output_schema,
+ *now_secs,
+ )? {
+ kept.push(row);
+ }
+ }
+ same_key_rows = kept;
+ }
+
+ // Java reader wrap means a fully-expired key never enters merge,
so
+ // Lookup is not consulted and history is left untouched.
+ if same_key_rows.is_empty() {
+ continue;
Review Comment:
[P1] Release consumed batches when an entire key expires
This continue skips the `compact_batch_buffer` path below. With no pending
output, the ordinary `MergeResult::Omit` path releases consumed source batches,
but a scan of entirely expired keys now keeps every decoded batch in
`batch_buffer` until the scan ends. A lazy 32-batch × 64-row probe using only
Weak array references measured peak live batches of 2 for the Omit control and
32 with input expiration; both produce zero output rows. Thus an expire-only
rewrite of a large historical file grows with the entire decoded input rather
than the streaming window and can OOM precisely when it should be discarding
data. Route empty expired groups through cleanup too, preserving any pending
output references, and add a multi-batch lifetime regression.
##########
crates/paimon/src/table/compact/primary_key/planner.rs:
##########
@@ -227,10 +227,63 @@ pub(in crate::table::compact) fn
select_with_force_level_zero(
Ok(Some((selected, output_level)))
}
+/// Incremental run selection used by execute/commit validation.
+///
+/// Matches planning: prefer UniversalCompaction, then fall back to an
+/// expire-only max-level rewrite when value stats prove every record expired.
+pub(in crate::table::compact) fn select_or_expire(
+ files: Vec<DataFileMeta>,
+ schema: &TableSchema,
+) -> Result<Option<(Vec<DataFileMeta>, i32)>> {
+ if let Some(selected) = select(files.clone(), schema.options())? {
Review Comment:
[P1] Validate automatic expiration with the same L0 selection policy
Automatic planning calls `select_with_force_level_zero(..., false)` by
default. If an expired max-level run coexists with a new L0 below the run-count
trigger, it falls back to an expire-only history task. This validator instead
calls `select`, which forces L0 promotion, so it selects a different input set
and rejects its own task.
Public API reproduction: bucket=1, num-levels=3, compaction trigger=5; seed
and compact an old row to L2 with expiration off; enable
expire-time=1s/time-field=ts and write-only=false; append a live second key.
The commit returns `Data commit ... succeeded, but automatic compaction failed`
caused by `PK compaction inputs do not match incremental run selection`;
snapshot advances 2→3 and both L2/L0 remain. This makes ordinary writes report
failure after publication. Persist/reuse the planner's selection mode or
validate expire-only task membership separately; cover this
automatic-maintenance case.
##########
crates/paimon/src/table/compact/primary_key.rs:
##########
@@ -381,13 +396,36 @@ pub(super) async fn execute(
let lookup_changelog = options.try_changelog_producer()? ==
ChangelogProducer::Lookup;
let output_level = task.output_level();
let drop_delete = options.deletion_vectors_enabled() && output_level > 0;
- let reader = new_reader(table, resources.clone())?.with_lookup(lookup);
+ // Java wraps both compact and LevelLookup readers with RecordLevelExpire
so
+ // LookupChangelogMergeFunction never observes expiration-only deletes.
+ let expire =
crate::table::record_level_expire::RecordLevelExpire::create(schema)?;
+ let now_secs = crate::table::record_level_expire::current_epoch_secs();
+ let lookup = match (lookup, &expire) {
+ (Some(provider), Some(expire)) => {
Review Comment:
[P1] Preserve accepted first-row lookup history until it is retired
The expiration wrapper is applied to first-row history as well, so an
expired accepted row is treated as absent even when its history file remains
outside the current L0 rewrite. I reproduced this on a table created with
`merge-engine=first-row` and `changelog-producer=lookup`: seed a large
max-level history containing key 1/value 10/ts 1, enable expire-time=1s, append
key 1/value 11 with a live timestamp, then compact only L0. Ordinary query
still returns value 10 from the retained first-row history, but the incremental
changelog emits `+I(1,11)`. A downstream consumer therefore diverges from the
table's current state.
Java explicitly uses an unfiltered, key-only lookup reader for first-row in
[MergeTreeCompactManagerFactory](https://github.com/apache/paimon/blob/f289b0a2829f8b0c28ae92743f11893893ce1116/paimon-core/src/main/java/org/apache/paimon/mergetree/compact/MergeTreeCompactManagerFactory.java#L334-L346).
Keep accepted-history lookup behavior for first-row, or retire that history
coherently before emitting a new INSERT; add a current-state/changelog
consistency regression.
##########
crates/paimon/src/table/compact/primary_key/planner.rs:
##########
@@ -227,10 +227,63 @@ pub(in crate::table::compact) fn
select_with_force_level_zero(
Ok(Some((selected, output_level)))
}
+/// Incremental run selection used by execute/commit validation.
+///
+/// Matches planning: prefer UniversalCompaction, then fall back to an
+/// expire-only max-level rewrite when value stats prove every record expired.
+pub(in crate::table::compact) fn select_or_expire(
+ files: Vec<DataFileMeta>,
+ schema: &TableSchema,
+) -> Result<Option<(Vec<DataFileMeta>, i32)>> {
+ if let Some(selected) = select(files.clone(), schema.options())? {
+ return Ok(Some(selected));
+ }
+ select_expired_max_level(&files, schema)
Review Comment:
[P2] Keep an unchanged expiration task valid as time advances
Execution and commit validation rerun this selection using `is_expire_file`,
which reads a fresh wall clock for each call, then require exactly the original
task's file set. A file that becomes eligible after planning changes the
recomputed set and invalidates an otherwise immutable task.
Public API probe: two max-level files, A already expired and B just inside
its retention window; plan the A-only task, perform no writes or metadata
changes, wait three seconds until B expires, then execute the captured task. It
fails with `PK compaction inputs do not match incremental run selection`. The
same drift can happen during a long execution before commit. Capture/reuse a
selection watermark, or verify that the selected inputs remain valid without
requiring every newly expired file to join the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]