TheR1sing3un opened a new pull request, #9759: URL: https://github.com/apache/paimon/pull/9759
### Purpose Raw vector fallback and refinement currently materialize Python vector lists and execute a Python loop for every vector dimension. Score regular FLOAT vectors in bounded NumPy blocks, covering Arrow list/large-list/fixed-size-list columns, candidate refinement, and primary-key raw/refined results. Candidate filtering still happens before scoring; unsupported and null-containing blocks retain scalar handling. L2 and cosine use float64 left-to-right accumulation to preserve the scalar reduction order. Inner product applies Python sum to precomputed products, preserving its runtime-specific summation behavior. Existing Top-K tie-breaking rules remain unchanged. Blocks target at most 1024 rows and 8 MiB per float64 matrix (with at least one vector); the Arrow read itself is still materialized. ### Tests - `python -m pytest pypaimon/tests/vector_scoring_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/primary_key_global_index_golden_test.py pypaimon/tests/primary_key_index_definitions_test.py -q`: 101 passed, 1 skipped because paimon_ftindex is unavailable. - Exact score checks cover random vectors, cancellation, close scores, zero norms, Arrow slices/chunks/list layouts, nulls, dimension failures, candidate filtering and ties. Java-generated primary-key data files exercise both raw scoring and native-index refinement against the scalar path. - Flake8, license-header checks and `git diff --check` passed. ### Benchmark macOS arm64, Python 3.9. Four queries over 8,192 vectors x 384 dimensions, FLOAT fixed-size lists in Zstd Parquet, Top-K=10. Each variant runs in a fresh process; values are medians of three runs. Timing includes Parquet reads, Arrow conversion, scoring and Top-K. Arrow read machinery is warmed before timing. Manifest planning and ANN search are excluded. The blocked-scalar ablation uses bounded conversion but disables vectorized arithmetic, separating its memory benefit from the arithmetic speedup. | Metric | Variant | Four-query time (s) | Peak RSS (MiB) | |---|---|---:|---:| | L2 | Original scalar | 2.198696 | 450.344 | | L2 | Blocked scalar | 2.109683 | 342.562 | | L2 | Vectorized | 0.062453 | 335.391 | | Cosine | Original scalar | 2.635863 | 441.766 | | Cosine | Blocked scalar | 2.573783 | 339.500 | | Cosine | Vectorized | 0.089766 | 322.234 | | Inner product | Original scalar | 2.134443 | 451.984 | | Inner product | Blocked scalar | 2.031288 | 341.406 | | Inner product | Vectorized | 0.187022 | 337.109 | All 27 runs matched Top-K row IDs and the binary representation of their scores within each metric. These workloads show about 35x / 29x / 11x faster fallback read-and-score execution and about 25-27% lower process peak RSS. These are not whole-query speedups including ANN/planning. A 64-row x 8-dimension smoke comparison also preserved result bits across all variants. Reproduce: ```shell python -m pypaimon.benchmark.vector_scoring_bench --output /tmp/scoring.json python -m pypaimon.benchmark.vector_scoring_bench --rows 64 --dimension 8 --repeats 1 --output /tmp/scoring-small.json ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
