JeonDaehong commented on code in PR #1390:
URL: https://github.com/apache/paimon-rust/pull/1390#discussion_r4550411658
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1872,6 +1887,10 @@ fn compact_batch_buffer(
*batch_buffer = new_buffer;
Review Comment:
Now that compaction can run before the output is full, this clones every
live entry on each pass, which is one `Arc` bump per column per entry. Taking
the old buffer with `std::mem::take` and moving the live entries over would
avoid that; with many pending materialized rows on a wide table it adds up.
Might be part of the ~1% on partial-update, though I haven't isolated it.
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1776,8 +1775,19 @@ fn sort_merge_stream(
Arc::new(output_schema.as_ref().clone().with_metadata(metadata)),
changelog.columns().to_vec(),
).map_err(|error| Error::DataInvalid { message:
format!("Invalid changelog batch: {error}"), source: Some(Box::new(error)) })?;
}
- if output_indices.is_empty() {
- compact_batch_buffer(&mut batch_buffer, &mut stream_batch_idx,
&cursors);
+ // There can be at most one source batch per cursor and per pending
Review Comment:
Nit on the wording: materialized results are buffer entries too, one per
pending row for aggregation and partial-update, so this is a bound on
referenced entries rather than source batches. That's also why the bound grows
with every merged key on those engines.
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1897,6 +1916,93 @@ mod tests {
use std::collections::HashMap;
use std::sync::Arc;
+ #[tokio::test]
+ async fn pending_output_releases_unreferenced_source_batches() {
Review Comment:
The lookup path (`changelog-producer = lookup`) also pushes a one-row batch
for a key without a base row, and when that key's result is omitted nothing
references it afterwards. With this change those get reclaimed too, where main
kept them until the next flush. Could a case with a `LookupProvider` cover that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]