wangzhigang1999 opened a new pull request, #95: URL: https://github.com/apache/paimon-vector-index/pull/95
PRs currently have no automated comparison of vector index performance against their base revision. This adds an informational benchmark check with a compact Recall/QPS report for IVF-FLAT, IVF-SQ, IVF-PQ, IVF-RQ and DiskANN. Both revisions build the same candidate-provided benchmark driver with Rust stable on the same runner. The fixed synthetic workload uses 10,000 64D vectors, 2,048 queries and two Rayon threads. Six fresh-process samples per version/index alternate execution order; warm sequential and batch timing each run for at least one second. The report compares medians and flags QPS drops above 10%/20% or recall drops above 1/3 percentage points. Performance changes are advisory, not merge gates. The existing benchmark gains optional warm timing and CSV fields, plus a fix for retained ground-truth allocations. Original first-pass Recall/I/O metrics are preserved. The comparison script validates samples, enforces a shared 20-minute budget, and retains raw data and failure logs. Results are published in the Actions summary. Same-repository PRs also update one bot comment through a separate job that only reads report text and skips changed head/base revisions. **External fork PRs, including this PR, use the Actions summary and artifacts; automatic comments are not supported for those PRs in this version.** No privileged PR-code execution or additional token is introduced. Validation on the exact submitted head: - [Full CI passed](https://github.com/wangzhigang1999/paimon-vector-index/actions/runs/34136756202), including Rust, C/C++, JNI and Python checks. - [Benchmark and comment publication passed](https://github.com/wangzhigang1999/paimon-vector-index/actions/runs/34136756151); all 60 samples completed. - [Example report in the fork PR](https://github.com/wangzhigang1999/paimon-vector-index/pull/1#issuecomment-5571649591). Identical library inputs produced unchanged recall and QPS differences between -0.7% and +0.9%; this is repeatability calibration, not an index performance improvement. - Eight Python tests cover calculations, advisory thresholds, invalid/incomplete samples and timeout handling. This is a small warm-cache regression check, not a large-scale or object-storage benchmark. Production index algorithms are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
