TheR1sing3un opened a new pull request, #9754: URL: https://github.com/apache/paimon/pull/9754
### Purpose Vector search scores indexed rows using the metric persisted in the index file, but raw fallback and refinement infer their metric from query/table options and otherwise default to L2. Vindex writers default to inner product, and explicitly supplied build options need not remain in table options. Merging these scores can return the wrong top-k. For example, index `[2, 0]` with the default IVF-FLAT metric, append `[3, 0]` without indexing it, and search for `[1, 0]` in `full` mode. The indexed row receives score `2.0`, while the raw row previously received L2 score `0.2`. The raw row now receives inner-product score `3.0` and ranks first. Read the persisted metric from each opened index reader and use it for raw fallback and original-vector refinement in single, batch and primary-key searches. Persisted metadata takes precedence over table options, which may have changed since the index was built. Explicit query metric conflicts and incompatible metrics across searched index shards raise a `ValueError` instead of merging incomparable scores. Close readers on metadata/metric validation failures and reset the resolved metric for each read call. For raw-only searches with a known vindex type and no configured metric, use the writer's inner-product default rather than L2 or a metric inferred from another column. Raw searches without a known vindex type keep their existing fallback behavior. ### Tests The native mixed-search regression fails on the base revision with `3.0 != 0.2` and passes with this change. ```sh python -m pytest pypaimon/tests/vector_metric_consistency_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/primary_key_global_index_golden_test.py -q ``` **89 passed, 1 skipped**, using Python 3.9.6, PyArrow 19.0.1 and paimon-vindex 0.4.0. The skip is the unrelated primary-key full-text fixture test because paimon-ftindex is unavailable; native vector tests and the Java-produced primary-key vector fixture ran. - Real Paimon/Parquet and native IVF-FLAT coverage for default inner product and build-only inner-product/cosine/L2 settings, mixed indexed/raw results, single/batch refinement, changed table options, incompatible query/shard metrics, and top-k consistency before/after index construction. - Metric-resolution checks without the native vindex dependency cover query option aliases, other-column settings, all five vindex defaults, reader closure on failures, and metric reset between read calls. - Java-produced primary-key vector indexes retain identical refinement results when table metric options change. - Changed files pass repository-configured flake8 and `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
