JingsongLi opened a new pull request, #928: URL: https://github.com/apache/paimon-rust/pull/928
### Purpose Table writers currently bypass `ResourceContext`, so buffered write data is invisible to a caller's memory budget. This change connects the write path to the same resource context used by readers, including the DataFusion task memory pool. ### Brief change log - Add optional `with_resources` configuration to table write builders and propagate it through append, key-value, dedicated-format, and postpone writers. - Reserve estimated Arrow batch memory while format writers retain data, and reserve key-value batches until they are flushed. - Make write and commit-preparation failures terminal and clean up uncommitted files, including outputs from other partitions that prepared successfully. ### Tests - `cargo test -p paimon --test writer_resources_test` (10 passed) - `cargo test -p paimon --lib inference_buffer_is_charged_without_triggering_row_group_flush` - `cargo test -p paimon --lib test_postpone_fixed_bucket_builder_modes` - Existing table-write tests for append, partitioned, blob, and reusable prepare-commit paths passed. - `cargo check -p paimon-datafusion`, `cargo fmt --all -- --check`, and `git diff --check` passed. Release-mode write microbenchmark on the in-memory backend: each run wrote 128 prebuilt batches of 8,192 rows (1,048,576 rows) and called `prepare_commit`. Nine measured runs per variant followed two warmups; variants alternated. The comparison isolates the optional resource accounting on the same code revision. | Write mode | Without resources | With resources | Median delta | | --- | ---: | ---: | ---: | | Append | 28.360 ms | 28.242 ms | -0.42% | | Primary key | 70.430 ms | 70.975 ms | +0.77% | The measured differences are within run-to-run variation for this workload. ### API and Format Adds an optional writer builder API; no storage-format change. ### Documentation Updates `ResourceContext` documentation to describe write-memory accounting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
