JingsongLi opened a new pull request, #8293:
URL: https://github.com/apache/paimon/pull/8293
## Summary
This PR flattens `paimon-vector` into a single jar module backed by
`paimon-vector-index-java` 0.1.0, removing the old `paimon-vector-jni`
submodule. It also adds Python read support for vindex vector indexes through
`paimon-vindex` 0.1.0 and covers Java-write/Python-read interop.
## Changes
- Convert `paimon-vector` from a parent module with `paimon-vector-index`
and `paimon-vector-jni` children into one module that depends on
`paimon-vector-index-java` 0.1.0.
- Move the vector global index factories/readers/writers into the top-level
`paimon-vector` source tree and load the native library from the released Java
artifact.
- Add a Python `paimon-vindex` vector global index reader and wire it into
vector search reader dispatch for `ivf-flat`, `ivf-pq`, `ivf-hnsw-flat`, and
`ivf-hnsw-sq`.
- Add optional Python packaging/dev dependency entries for
`paimon-vindex==0.1.0`.
- Add Java/Python mixed tests: Java writes a vindex-backed vector index and
Python reads/searches it; `run_mixed_tests.sh` now runs that path on Python >=
3.9.
## Testing
- [x] `python -m py_compile
paimon-python/pypaimon/table/source/vector_search_read.py
paimon-python/pypaimon/globalindex/vindex/vindex_vector_global_index_reader.py
paimon-python/pypaimon/tests/e2e/java_py_read_write_test.py`
- [x] `bash -n paimon-python/dev/run_mixed_tests.sh`
- [x] `git diff --check`
- [x] `mvn -pl paimon-vector -am -Pfast-build -DfailIfNoTests=false
-Dtest=NativeVectorGlobalIndexerFactoryTest,SeekableStreamVectorIndexInputTest,NativeVectorGlobalIndexTest,org.apache.paimon.JavaPyE2ETest#testVindexVectorIndexWrite
test -Drun.e2e.tests=true`
- [x] `PYTHONPATH=/tmp/paimon-vindex-site:${PYTHONPATH:-} python -m pytest
java_py_read_write_test.py::JavaPyReadWriteTest::test_read_vindex_vector_index
-v`
- [x] `mvn -pl paimon-vector -am -DskipTests compile`
## Notes
`paimon-vector` has not been released yet, so this intentionally does not
preserve compatibility with the old two-submodule layout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
