suxiaogang223 opened a new issue, #1468: URL: https://github.com/apache/paimon-rust/issues/1468
### Search before asking - [x] I searched the open issues. #1386 covers pending-output batch reclamation and #911 concerns writer resource governance. This request concerns a shared reader budget and its C embedding contract. ### Motivation An embedding engine needs to bound a Paimon logical read across concurrent splits. The v0.5.0 core has `ResourceContext` and a Rust `MemoryPool` trait; C callers can create a local byte budget and attach it to a reader. The documented coverage is incomplete, so this is a request to extend that contract, not a claim that the current accounting is an allocator or RSS limit. Source baseline: v0.5.0, code commit `bb6389a8b430e897e038ff7395b6a7763d785a7a`. Two uncovered paths prevent an engine from treating this budget as a bound on logical merge work: 1. ORC reads do not charge this reservation budget. 2. Sort-merge can retain consumed input batches while a small pending output remains below the batch size. The reservation used by a Parquet decoder does not cover that subsequent retention. One key with many versions and variable-size aggregation state require their own budget coverage too. ### Reproduction and observations I ran a C ABI reader linked against the Doris master thirdparty prebuilt published on 2026-11-19, SHA256 `4fb0e17f73db3ebf9edd0ce95bdbb8e3e37c95cdbe1e06f7e2ee8ff9917318f8`. Paimon data and serialized DataSplits were produced by Java Paimon 2.1.0. These are SDK execution probes, not a Doris integrated-reader benchmark. Basic inputs: two commits containing `(1,11),(2,22)` and then `(1,7)`, bucket 1, write-only, deletion vectors disabled. Deduplicate and partial-update returned `(1,7),(2,22)`; aggregation/sum returned `(1,18),(2,22)`. A compacted first-row control returned `(1,11),(2,22)`. Parquet and ORC results matched the Java reader. Attaching a zero-byte resource context: - Parquet failed with resource-exhausted, e.g. `Cannot reserve 118 bytes ... limit 0`. - ORC still returned the rows, with `peak_reserved_memory_bytes=0`, for all four engine controls. A second input exercises #1386: two overlapping files, primary key `id`, string value of 1024 bytes, first commit INSERT keys 0..32768, second commit DELETE keys 1..32768. The only surviving row is key 0. Writer options include `parquet.block.size=65536`, `parquet.page.size=8192`, `write-only=true`, `deletion-vectors.enabled=false`. Each probe runs in a fresh process, consumes/releases each exported Arrow batch immediately, and releases reader/read/plan/builder/table before reading resource metrics. Only `read.batch-size` changes between runs; no files or splits change. | Format | Batch size | Local byte limit | Rows | Peak reservation bytes | Process peak RSS KiB | |---|---:|---:|---:|---:|---:| | Parquet | 1 | 2097152 | 1 | 44020 | 27236 | | Parquet | 32 | 2097152 | 1 | 44020 | 95996 | | ORC | 1 | 0 | 1 | 0 | 23984 | | ORC | 32 | 0 | 1 | 0 | 93308 | RSS includes runtime/decoder overhead and is not an exact measurement of retained merge allocations. The large difference with unchanged reservation metrics, together with the source retention path, shows why these metrics cannot enforce the desired merge budget. Batch size 1 is a control, not a proposed fix. This probe does not establish bounds for same-key versions, aggregation state, concurrency, or cancellation. The C call sequence is: ```c paimon_table_from_schema_json(table_path, schema_json, "db", "probe", "main", NULL, 0); paimon_resource_context_create(memory_limit_bytes); paimon_table_new_read_builder(table); paimon_read_builder_with_resources(builder, context); paimon_plan_from_split_bytes(java_split_bytes, length); paimon_read_builder_new_read(builder); paimon_table_read_to_arrow(read, plan, 0, paimon_plan_num_splits(plan)); // Iterate paimon_record_batch_reader_next, invoke Arrow release callbacks, // then paimon_arrow_batch_free. Free all handles on every exit. ``` ### Proposed scope - Retain shared, ownership-bound reservations with decoded input buffers for as long as cursors, same-key groups, or pending output refer to them. Do not release the accounting when a decoder advances while merge state still owns those buffers. Avoid charging the same aliased buffer twice. - Reclaim losing batches independently of filling an output batch (#1386), preserving cursor/output references and result order. - Cover converted keys/user sequences, same-key version collections, materialized rows, and variable-size aggregation state, including oversized single keys. Reject work that cannot be admitted before retaining it. Document temporary/output ownership separately. - Extend equivalent admission/accounting to ORC, or explicitly expose format/operation coverage so embedding engines can reject an unsupported budget contract before reading. - Add an additive, versioned C constructor for the existing host `MemoryPool` contract. Specify reserve/release callbacks, context destruction, lifetime through cloned readers and Arrow buffers, and calls from Rust runtime threads. Share one host budget across concurrent split readers; do not assign each split the full query limit. - Provide a thread-safe cancellation token checked during I/O and merge loops, including a synchronous `next` waiting on the stream. Concurrently freeing a busy reader must remain invalid. A 0.5.0-compatible backport would preserve the existing symbols and local-budget behavior and add new entry points. A downstream integration would need a prebuilt containing those changes; merely linking the unmodified 0.5.0 library and calling `with_resources` does not complete the contract. ### Validation Use deterministic reservation/ownership counters in addition to RSS. Cover early survivor plus long deleted tail, all-deleted input, one key spanning many batches/files, all four merge engines, variable-size aggregates, concurrent readers sharing a small budget, EOF/LIMIT, I/O and conversion failures, and cancellation while a read is blocked. Results and errors must agree with the unbudgeted read; every reservation must return to zero after cleanup. References: [resource coverage](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/crates/paimon/src/resource/mod.rs), [sort-merge](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/crates/paimon/src/table/sort_merge.rs), [C resource API](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/bindings/c/src/resource.rs), [synchronous C next](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/bindings/c/src/table.rs#L1060). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
