suxiaogang223 opened a new issue, #1468:
URL: https://github.com/apache/paimon-rust/issues/1468

   ### Search before asking
   
   - [x] I searched the open issues. #1386 covers pending-output batch 
reclamation and #911 concerns writer resource governance. This request concerns 
a shared reader budget and its C embedding contract.
   
   ### Motivation
   
   An embedding engine needs to bound a Paimon logical read across concurrent 
splits. The v0.5.0 core has `ResourceContext` and a Rust `MemoryPool` trait; C 
callers can create a local byte budget and attach it to a reader. The 
documented coverage is incomplete, so this is a request to extend that 
contract, not a claim that the current accounting is an allocator or RSS limit.
   
   Source baseline: v0.5.0, code commit 
`bb6389a8b430e897e038ff7395b6a7763d785a7a`.
   
   Two uncovered paths prevent an engine from treating this budget as a bound 
on logical merge work:
   
   1. ORC reads do not charge this reservation budget.
   2. Sort-merge can retain consumed input batches while a small pending output 
remains below the batch size. The reservation used by a Parquet decoder does 
not cover that subsequent retention. One key with many versions and 
variable-size aggregation state require their own budget coverage too.
   
   ### Reproduction and observations
   
   I ran a C ABI reader linked against the Doris master thirdparty prebuilt 
published on 2026-11-19, SHA256 
`4fb0e17f73db3ebf9edd0ce95bdbb8e3e37c95cdbe1e06f7e2ee8ff9917318f8`. Paimon data 
and serialized DataSplits were produced by Java Paimon 2.1.0. These are SDK 
execution probes, not a Doris integrated-reader benchmark.
   
   Basic inputs: two commits containing `(1,11),(2,22)` and then `(1,7)`, 
bucket 1, write-only, deletion vectors disabled. Deduplicate and partial-update 
returned `(1,7),(2,22)`; aggregation/sum returned `(1,18),(2,22)`. A compacted 
first-row control returned `(1,11),(2,22)`. Parquet and ORC results matched the 
Java reader.
   
   Attaching a zero-byte resource context:
   
   - Parquet failed with resource-exhausted, e.g. `Cannot reserve 118 bytes ... 
limit 0`.
   - ORC still returned the rows, with `peak_reserved_memory_bytes=0`, for all 
four engine controls.
   
   A second input exercises #1386: two overlapping files, primary key `id`, 
string value of 1024 bytes, first commit INSERT keys 0..32768, second commit 
DELETE keys 1..32768. The only surviving row is key 0. Writer options include 
`parquet.block.size=65536`, `parquet.page.size=8192`, `write-only=true`, 
`deletion-vectors.enabled=false`.
   
   Each probe runs in a fresh process, consumes/releases each exported Arrow 
batch immediately, and releases reader/read/plan/builder/table before reading 
resource metrics. Only `read.batch-size` changes between runs; no files or 
splits change.
   
   | Format | Batch size | Local byte limit | Rows | Peak reservation bytes | 
Process peak RSS KiB |
   |---|---:|---:|---:|---:|---:|
   | Parquet | 1 | 2097152 | 1 | 44020 | 27236 |
   | Parquet | 32 | 2097152 | 1 | 44020 | 95996 |
   | ORC | 1 | 0 | 1 | 0 | 23984 |
   | ORC | 32 | 0 | 1 | 0 | 93308 |
   
   RSS includes runtime/decoder overhead and is not an exact measurement of 
retained merge allocations. The large difference with unchanged reservation 
metrics, together with the source retention path, shows why these metrics 
cannot enforce the desired merge budget. Batch size 1 is a control, not a 
proposed fix. This probe does not establish bounds for same-key versions, 
aggregation state, concurrency, or cancellation.
   
   The C call sequence is:
   
   ```c
   paimon_table_from_schema_json(table_path, schema_json, "db", "probe", 
"main", NULL, 0);
   paimon_resource_context_create(memory_limit_bytes);
   paimon_table_new_read_builder(table);
   paimon_read_builder_with_resources(builder, context);
   paimon_plan_from_split_bytes(java_split_bytes, length);
   paimon_read_builder_new_read(builder);
   paimon_table_read_to_arrow(read, plan, 0, paimon_plan_num_splits(plan));
   // Iterate paimon_record_batch_reader_next, invoke Arrow release callbacks,
   // then paimon_arrow_batch_free. Free all handles on every exit.
   ```
   
   ### Proposed scope
   
   - Retain shared, ownership-bound reservations with decoded input buffers for 
as long as cursors, same-key groups, or pending output refer to them. Do not 
release the accounting when a decoder advances while merge state still owns 
those buffers. Avoid charging the same aliased buffer twice.
   - Reclaim losing batches independently of filling an output batch (#1386), 
preserving cursor/output references and result order.
   - Cover converted keys/user sequences, same-key version collections, 
materialized rows, and variable-size aggregation state, including oversized 
single keys. Reject work that cannot be admitted before retaining it. Document 
temporary/output ownership separately.
   - Extend equivalent admission/accounting to ORC, or explicitly expose 
format/operation coverage so embedding engines can reject an unsupported budget 
contract before reading.
   - Add an additive, versioned C constructor for the existing host 
`MemoryPool` contract. Specify reserve/release callbacks, context destruction, 
lifetime through cloned readers and Arrow buffers, and calls from Rust runtime 
threads. Share one host budget across concurrent split readers; do not assign 
each split the full query limit.
   - Provide a thread-safe cancellation token checked during I/O and merge 
loops, including a synchronous `next` waiting on the stream. Concurrently 
freeing a busy reader must remain invalid.
   
   A 0.5.0-compatible backport would preserve the existing symbols and 
local-budget behavior and add new entry points. A downstream integration would 
need a prebuilt containing those changes; merely linking the unmodified 0.5.0 
library and calling `with_resources` does not complete the contract.
   
   ### Validation
   
   Use deterministic reservation/ownership counters in addition to RSS. Cover 
early survivor plus long deleted tail, all-deleted input, one key spanning many 
batches/files, all four merge engines, variable-size aggregates, concurrent 
readers sharing a small budget, EOF/LIMIT, I/O and conversion failures, and 
cancellation while a read is blocked. Results and errors must agree with the 
unbudgeted read; every reservation must return to zero after cleanup.
   
   References: [resource 
coverage](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/crates/paimon/src/resource/mod.rs),
 
[sort-merge](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/crates/paimon/src/table/sort_merge.rs),
 [C resource 
API](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/bindings/c/src/resource.rs),
 [synchronous C 
next](https://github.com/apache/paimon-rust/blob/bb6389a8b430e897e038ff7395b6a7763d785a7a/bindings/c/src/table.rs#L1060).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to