JingsongLi commented on code in PR #1480:
URL: https://github.com/apache/paimon-rust/pull/1480#discussion_r4656346799
##########
crates/paimon/src/table/compact/primary_key/lookup.rs:
##########
@@ -108,17 +108,14 @@ fn source_row(
columns: &[usize],
schema: &SchemaRef,
) -> Result<RecordBatch> {
- let indices = UInt64Array::from(vec![row_idx as u64]);
+ // A zero-copy slice: the reader concatenates changelog rows into batches.
let values = (0..schema.fields().len())
.map(|index| {
- arrow_select::take::take(
- batches[batch_idx].column_for_output(index, columns),
- &indices,
- None,
- )
- .map_err(|error| invalid(format!("Failed to materialize lookup
value: {error}")))
+ batches[batch_idx]
+ .column_for_output(index, columns)
+ .slice(row_idx, 1)
Review Comment:
**[P1] Bound the source buffers held by pending lookup changelog rows**
With lookup changelog, row deduplication and `ignore-update-before` enabled,
an update can enqueue one row that pins the entire input batch. The data output
releases its references after each flush, but `pending_changelog` waits for
`read.batch-size` emitted changes. A sorted rewrite with one changed key per
input batch can therefore retain nearly `read.batch-size` full input batches.
I reproduced this in the compaction merge stream with 128-row batches and 4
KiB binary values. After 40 data outputs, the PR still held 41 L0 payload
buffers (20.5 MiB). Its parent held one (0.5 MiB). At the default 1,024-row
batch size the same pattern can retain about 4 GiB in payloads alone. Please
bound this queue by retained bytes or materialize its rows before releasing a
source batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]