JeonDaehong commented on code in PR #1390:
URL: https://github.com/apache/paimon-rust/pull/1390#discussion_r4550411658


##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1872,6 +1887,10 @@ fn compact_batch_buffer(
 
     *batch_buffer = new_buffer;

Review Comment:
   Now that compaction can run before the output is full, this clones every 
live entry on each pass, which is one `Arc` bump per column per entry. Taking 
the old buffer with `std::mem::take` and moving the live entries over would 
avoid that; with many pending materialized rows on a wide table it adds up. 
Might be part of the ~1% on partial-update, though I haven't isolated it.



##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1776,8 +1775,19 @@ fn sort_merge_stream(
                     
Arc::new(output_schema.as_ref().clone().with_metadata(metadata)), 
changelog.columns().to_vec(),
                 ).map_err(|error| Error::DataInvalid { message: 
format!("Invalid changelog batch: {error}"), source: Some(Box::new(error)) })?;
             }
-            if output_indices.is_empty() {
-                compact_batch_buffer(&mut batch_buffer, &mut stream_batch_idx, 
&cursors);
+            // There can be at most one source batch per cursor and per pending

Review Comment:
   Nit on the wording: materialized results are buffer entries too, one per 
pending row for aggregation and partial-update, so this is a bound on 
referenced entries rather than source batches. That's also why the bound grows 
with every merged key on those engines.



##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1897,6 +1916,93 @@ mod tests {
     use std::collections::HashMap;
     use std::sync::Arc;
 
+    #[tokio::test]
+    async fn pending_output_releases_unreferenced_source_batches() {

Review Comment:
   The lookup path (`changelog-producer = lookup`) also pushes a one-row batch 
for a key without a base row, and when that key's result is omitted nothing 
references it afterwards. With this change those get reclaimed too, where main 
kept them until the next flush. Could a case with a `LookupProvider` cover that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to