shyjsarah commented on PR #73: URL: https://github.com/apache/paimon-vector-index/pull/73#issuecomment-5238287630
Diagnostic follow-up: - Confirmed the PR #73 IVF-PQ batch diagnostic path was executing in EMR Spark, including the filtered batch route across all 14 shards. - Root cause of the missing messages: the EMR log collector did not capture writes made directly to the Rust/native stderr descriptor. - Added a JNI bridge that captures the opt-in IVF-PQ diagnostic line on the search thread and mirrors the same line through Java `System.err`, while retaining the existing native stderr write. - EMR smoke job `jr-375b9ff8d65f53f3` succeeded: 14/14 filtered batch searches emitted complete diagnostics and there were no native errors. - The smoke showed `nq=100`, `k=3`, fixed `nprobe=16`, and `incomplete_queries=0` on all 14 shards. Payload read time dominated filter and PQ scan time. Minimal validation after cleanup: - `cargo fmt --all -- --check` - focused diagnostic capture tests (3 passed) - `cargo check -p paimon-vindex-jni` - `mvn -f java/pom.xml test` The final change is contained in `paimon-vector-index`; no Paimon source change is required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
