TheR1sing3un opened a new pull request, #10157: URL: https://github.com/apache/paimon/pull/10157
### Purpose Allow Python to build native vector indexes on deletion-vector-enabled data-evolution tables, including tables created with the multimodal API's defaults. The Python builder still rejected these tables after #8930 enabled the corresponding JVM build and pinned Python vector reads to the query snapshot. The build keeps the existing physical row-ID coverage, including logically deleted rows. Query-side live-row filtering applies the pinned snapshot's deletion vectors before top-K selection. The allowance is limited to the five existing native vector index types on unaware-bucket data-evolution tables; other generic index types retain the DV restriction. The new partitioned-DV regression also exposed that `partitions` predicates were built with full-table field positions, then evaluated against partition rows in existing index manifests. Build those predicates against the partition schema, matching the drop-index path. Add regression coverage for deleted head/middle/tail/all rows, NULL vectors, overlapping column files, multiple shards, incremental builds, partition selection, snapshot consistency, and concurrent deletion or replacement of source row IDs. Existing Ray and scalar-filter tests now build vector indexes directly on DV-enabled tables. Document the default multimodal build/delete/search workflow. ### Tests - Python 3.11: 197 tests passed (plus 27 subtests) across `vector_index_dv_build_test.py`, `global_index_build_test.py`, `vector_filter_exactness_test.py`, `vector_search_filter_test.py`, `search_snapshot_test.py`, and `vindex_vector_index_test.py`. - Real local Ray cluster: 27 selected single/batch vector tests passed, covering indexed/raw queries, deletion filtering, refinement and snapshots. - Real paimon-vindex 0.5.0 smoke checks for IVF-FLAT, IVF-PQ, IVF-SQ, IVF-RQ and DiskANN: build after deletion, delete again after build, then single/batch searches exclude the deleted IDs. - Restoring the old DV guard in-process makes the new build regression fail at the original rejection. - Flake8, Python 3.6 syntax parsing, `git diff --check`, compilation of the added MDX section, and execution of its Python example passed. The source-row replacement regression uses Python overwrite; this validation does not include a live cross-engine JVM materialized-compaction race. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
