JingsongLi opened a new pull request, #129: URL: https://github.com/apache/paimon-vector-index/pull/129
Wide distance bands can return millions of hits, but `max_working_bytes` excludes output. The previous collector could also keep staged rows and the final CSR arrays alive together. Add a separate positive `max_result_bytes` budget, defaulting to 64 MiB for the whole call across all queries. Account for result metadata, producer and staged capacities, replacement allocations, and simultaneous CSR construction; fail explicitly instead of truncating. Expose synchronous chunk consumers across Rust, C, C++, Java/JNI, and Python so large outputs can be processed during scanning. Producers transfer budgeted blocks into materialized results without another payload copy and avoid per-hit output-lock contention. Preserve the first consumer error across parallel cancellation, reject explicit empty Python filters, and flush full and final partial chunks correctly. The existing C parameter prefix remains supported; missing appended budget fields use the default. Bindings check the result capability version. Large materialized queries may now require a higher budget or the consumer API. Native result budgeting excludes resident index data, traversal scratch, decoded filters, binding copies, and consumer-retained data. Callbacks are serialized, may run on native workers, and already-delivered chunks remain delivered if a later step fails. Validation: - `cargo test --workspace`, `cargo fmt --all -- --check`, and `cargo clippy --all-targets --workspace -- -D warnings`. - Python suite: 459 passed, 1 skipped; C/C++ smoke tests and Java/JNI range tests passed on macOS arm64. - Regression tests cover budget boundaries, 255/256/257-hit block boundaries, low-budget fallback, multiple workers, filtering, reentry, original error causes, full-chunk backpressure, and final partial chunks. - Independent parallel probes preserved OS error codes and custom error sources in 1,000 runs each. - A 10-million-hit IVF-FLAT probe rejected materialization at the default budget and consumed every unique hit under a 1 MiB native result budget in 10,000 chunks of 1,000 hits. Cross-platform builds and process RSS peaks were not measured locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
