JingsongLi opened a new pull request, #129:
URL: https://github.com/apache/paimon-vector-index/pull/129

   Wide distance bands can return millions of hits, but `max_working_bytes` 
excludes output. The previous collector could also keep staged rows and the 
final CSR arrays alive together.
   
   Add a separate positive `max_result_bytes` budget, defaulting to 64 MiB for 
the whole call across all queries. Account for result metadata, producer and 
staged capacities, replacement allocations, and simultaneous CSR construction; 
fail explicitly instead of truncating. Expose synchronous chunk consumers 
across Rust, C, C++, Java/JNI, and Python so large outputs can be processed 
during scanning. Producers transfer budgeted blocks into materialized results 
without another payload copy and avoid per-hit output-lock contention. Preserve 
the first consumer error across parallel cancellation, reject explicit empty 
Python filters, and flush full and final partial chunks correctly.
   
   The existing C parameter prefix remains supported; missing appended budget 
fields use the default. Bindings check the result capability version. Large 
materialized queries may now require a higher budget or the consumer API. 
Native result budgeting excludes resident index data, traversal scratch, 
decoded filters, binding copies, and consumer-retained data. Callbacks are 
serialized, may run on native workers, and already-delivered chunks remain 
delivered if a later step fails.
   
   Validation:
   - `cargo test --workspace`, `cargo fmt --all -- --check`, and `cargo clippy 
--all-targets --workspace -- -D warnings`.
   - Python suite: 459 passed, 1 skipped; C/C++ smoke tests and Java/JNI range 
tests passed on macOS arm64.
   - Regression tests cover budget boundaries, 255/256/257-hit block 
boundaries, low-budget fallback, multiple workers, filtering, reentry, original 
error causes, full-chunk backpressure, and final partial chunks.
   - Independent parallel probes preserved OS error codes and custom error 
sources in 1,000 runs each.
   - A 10-million-hit IVF-FLAT probe rejected materialization at the default 
budget and consumed every unique hit under a 1 MiB native result budget in 
10,000 chunks of 1,000 hits.
   
   Cross-platform builds and process RSS peaks were not measured locally.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to