JeonDaehong commented on code in PR #1390:
URL: https://github.com/apache/paimon-rust/pull/1390#discussion_r4550370463
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1776,8 +1775,19 @@ fn sort_merge_stream(
Arc::new(output_schema.as_ref().clone().with_metadata(metadata)),
changelog.columns().to_vec(),
).map_err(|error| Error::DataInvalid { message:
format!("Invalid changelog batch: {error}"), source: Some(Box::new(error)) })?;
}
- if output_indices.is_empty() {
- compact_batch_buffer(&mut batch_buffer, &mut stream_batch_idx,
&cursors);
+ // There can be at most one source batch per cursor and per pending
+ // output row. A factor of two amortizes compaction while bounding
+ // unrelated batches across a long sequence of omitted keys.
+ let referenced_batch_bound =
num_streams.saturating_add(output_indices.len());
Review Comment:
The bound counts pending rows, not batches, so with full-size batches it
rarely fires. I ran two streams with 1024-row batches through this path: keys
0..1000 kept, then 1000..204800 deleted, batch size 1024. Main and this PR both
peak at 399 of 400 source batches alive. A bound on live batches instead (2x
the buffer length after the last compaction) brings it to 6.
That bound has a cost though: aggregation reads got 24% slower and
partial-update 27% (2M keys, 3 overlapping rounds), since every materialized
row is its own buffer entry and keeps resetting the count. With this PR as is,
deduplicate and aggregation are unchanged, and partial-update is about 1%
slower (1542–1552 vs 1527–1534 ms, 3 alternating rounds). Counting only
`Source` entries might avoid both, but I haven't measured that.
##########
crates/paimon/src/table/sort_merge.rs:
##########
@@ -1897,6 +1916,93 @@ mod tests {
use std::collections::HashMap;
use std::sync::Arc;
+ #[tokio::test]
+ async fn pending_output_releases_unreferenced_source_batches() {
+ use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering};
+ use std::sync::{Mutex, Weak};
+
+ let schema = make_schema();
+ let observed: Arc<Mutex<Vec<Weak<dyn Array>>>> = Arc::default();
+ let peak = Arc::new(AtomicUsize::new(0));
+ let stream = |keys: Vec<i32>, kind: i8, sequence: i64| {
+ let observed = observed.clone();
+ let peak = peak.clone();
+ let schema = schema.clone();
+ futures::stream::iter(keys)
+ .map(move |key| {
+ let mut observed = observed.lock().unwrap();
+ let live = observed
+ .iter()
+ .filter(|batch| batch.strong_count() > 0)
+ .count();
+ peak.fetch_max(live, AtomicOrdering::Relaxed);
+ let value = format!("value-{key}");
+ let batch = make_batch_with_kind(
+ &schema,
+ vec![key],
+ vec![sequence],
+ vec![kind],
+ vec![Some(value.as_str())],
+ );
+ observed.push(Arc::downgrade(batch.column(3)));
+ Ok(batch)
+ })
+ .boxed()
+ };
+ let inserts = stream((0..=65).collect(), RowKind::Insert.to_value(),
1);
+ let deletes = stream(
+ (1..65).filter(|key| *key != 31).collect(),
+ RowKind::Delete.to_value(),
+ 2,
+ );
+ let batches = SortMergeReaderBuilder::new(
+ vec![inserts, deletes],
+ schema,
+ vec![0],
+ 1,
+ 2,
+ vec![],
+ vec![3],
+ make_output_schema(),
+ Box::new(DeduplicateMergeFunction),
+ )
+ .with_batch_size(256)
+ .build()
+ .unwrap()
+ .try_collect::<Vec<_>>()
+ .await
+ .unwrap();
+
+ assert_eq!(batches.len(), 1);
+ let keys = batches[0]
+ .column(0)
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .unwrap();
+ assert_eq!(keys.values().as_ref(), &[0, 31, 65]);
+ let values = batches[0]
+ .column(1)
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .unwrap();
+ assert_eq!(
+ values.iter().collect::<Vec<_>>(),
+ vec![Some("value-0"), Some("value-31"), Some("value-65")]
+ );
+ // Two cursors and at most three pending source rows need only a small
+ // buffer. Unrelated consumed batches must not accumulate with deletes.
+ assert!(
+ peak.load(AtomicOrdering::Relaxed) <= 16,
Review Comment:
The test uses one-row batches, where rows and batches are the same count.
Could it also cover multi-row batches, e.g. 1024 rows with the kept keys
filling most of an output batch? That's the case the bound misses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]