JingsongLi opened a new pull request, #914:
URL: https://github.com/apache/paimon-rust/pull/914
### Purpose
Related to #911: establish shared Rust memory reservations and integrate
them with Reader output ownership as a foundation for resource governance.
Output Arrow buffers can outlive a reader when callers retain batches,
arrays, or slices. Add an opt-in `ResourceContext` so sibling reads share a
budget and reservations follow those retained buffers.
### Brief change log
- Add `ResourceContext`, an external `MemoryPool` adapter, RAII
`MemoryReservation`, current/peak reservation metrics, and `ResourceExhausted`
errors. Failed admission rolls back without consuming budget.
- Add `ReadBuilder::with_resources` and `TableRead::with_resources`,
preserving the context across builder clones, consuming setters, and ordinary,
incremental, row-kind, and audit output paths.
- Retain each Arrow buffer's original owner without copying its contents.
Deduplicate reservations by backing pointer within a context; release them
after the last tracked output alias drops. Reclaim expired registry entries
with amortized sweeps.
- Preserve `ReadBudget` for existing prefetch/concurrency control. This
phase accounts retained output buffers at emission; decoder working memory,
prefetch buffers, scan metadata, merge state, and Writer integration remain
follow-up work. External buffer capacity is estimated from the extent Arrow
exposes.
### Tests
- `cargo test --offline --locked -p paimon --lib --test
reader_resources_test --test incremental_batch_scan_test --test
first_row_scan_test`: 3043 unit tests and 48 integration tests passed; 6
existing tests ignored.
- New coverage includes concurrent admission, failed-reservation rollback,
shared output-buffer accounting, clone/slice and external-owner lifetimes,
cleanup over long streams, context propagation, and terminal exhaustion across
Reader output modes.
- `cargo test --offline --locked -p paimon --doc resource::`: passed.
- `cargo clippy --offline --locked -p paimon --all-targets -- -D warnings`:
passed.
- `cargo fmt --all -- --check`: passed.
Validation used the default `paimon` features. Optional fulltext/Vortex and
warehouse-backed suites were not run.
### API and Format
Adds opt-in Rust resource APIs and an error variant; existing `RecordBatch`
stream signatures and storage formats are unchanged.
```rust
let resources = ResourceContext::builder()
.memory_limit(256 * 1024 * 1024)
.build()?;
let mut builder = table.new_read_builder();
builder.with_resources(resources.clone());
```
Wrapped output buffers may report a smaller Arrow capacity and require
copying to become mutable. Context metrics report the reservations; the
configured limit covers retained output buffers, rather than process RSS or the
complete Reader working set.
### Documentation
Public Rustdoc documents pool callback contracts, shared ownership,
accounting limits, buffer behavior, and a tested reservation example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
