JingsongLi opened a new pull request, #123: URL: https://github.com/apache/paimon-vector-index/pull/123
Large IVF-RQ partitions could bypass the 64 MiB batch limit, malformed headers could drive large allocations, and JNI output was limited by a single Java byte array. Release jobs also lacked a shared source-version check and runtime validation of their final wheels. ## Changes - Stream oversized IVF-RQ lists in aligned scan blocks, including a bounded delta-ID window. Preserve top-K, range, filtered, batch, and continued-probe results while accounting for the ID window in the allocation budget. - Validate header-derived PQ/RQ memory before construction, use fallible allocations, defer rotation creation, and commit loaded metadata only after complete success so retries cannot retain partial arrays. - Write JNI output in reusable 1 MiB chunks, update position after each successful chunk, and preserve the original Java exception. - Gate the release orchestrator and independent language workflows on tag, Cargo workspace/path-dependency, Python, and Maven version consistency. RC tags use the existing clean source-version policy. - Install each final repaired wheel in an isolated environment and exercise native loading, training, writing, reading, and searching before artifact upload and publication. ## Validation - `cargo test --workspace --no-fail-fast`: 737 passed, 2 existing ignored. - `python3 -m unittest discover -s tools/tests -p 'test_*.py' -q`: 26 passed, including final/RC tags, mismatches, and non-tag inputs. - `cargo clippy --all-targets --workspace -- -D warnings`, formatting, license-header, and binary legal-inventory checks passed. - Java API tests and native chunk/boundary tests passed, including `-Xcheck:jni`, buffer reuse, byte order, and original exception identity. - A locally built and delocate-repaired macOS arm64 wheel passed isolated installation and the end-to-end smoke test. Linux x86-64/aarch64 and Windows final-wheel runtime validation is configured as a required step in the release workflows; it still needs the corresponding CI runners. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
