ChaomingZhangCN opened a new pull request, #438: URL: https://github.com/apache/paimon-cpp/pull/438
### Purpose Linked issue: None. Add dedicated Vortex storage for a single top-level VECTOR column in append-only tables, building on the Vortex format support in #376. With `vector.file.format=vortex`, ordinary columns are written to Parquet main files and the VECTOR column is written to `.vector.vortex` files. Both files cover the same row range and are read together through the existing data-evolution reader. The dedicated mode requires: ```text file.format=parquet vector.file.format=vortex bucket=-1 row-tracking.enabled=true data-evolution.enabled=true ``` - Support full-row INSERTs with exactly one top-level VECTOR and at least one ordinary non-partition column. - Roll main/VECTOR files together when either size target or the row-count target is reached; preserve matching row ranges and clean up both files on write/close failure. - Add `vector.target-file-size`, defaulting to `target-file-size`. - Reject unsupported dedicated-mode combinations: primary-key tables, custom/partial write schemas, multiple or nested VECTOR columns, and BLOB columns. Independent VECTOR file rolling is outside this implementation. Separately, allow VECTOR columns in ordinary `file.format=vortex` tables. Validate visible VECTOR elements using ancestor validity before normalizing hidden NULL slots for Vortex import, so NULL STRUCT/LIST ancestors do not cause valid writes to fail. Visible NULL elements are still rejected for non-nullable VECTOR elements. This is a scoped implementation of dedicated VECTOR storage; it does not claim complete feature parity with Paimon Java. ### Tests Java interoperability is tested using checked-in files actually written and read back by Paimon Java `2.2-SNAPSHOT` at commit `7d6d146bbaf33a349dc83c3caf71334bd551e3d5` (Vortex JNI `0.73.0`). The fixture contains two append commits, six rows including two NULL vectors, and two Parquet/Vortex file pairs. Its generator, runtime dependency POM, provenance, and reproduction commands are included. `DataEvolutionTableTest.TestReadJavaDedicatedVectorStore` reads the fixture without rewriting it and verifies full rows, ordinary-column and VECTOR-only projections, row ranges crossing commits, and paired file metadata (`firstRowId`, row counts, and `writeCols`). `VortexFileFormatTest.WriteThenReadVectorsWithNullAncestors` covers NULL STRUCT/LIST parents, batch boundaries, multiple read batch sizes, and rejection of the same invalid elements when the ancestors become visible. The NULL-ancestor regression fails before the fix and passes afterward. Local validation on macOS aarch64, C++17 Debug, with Vortex enabled: | Validation | Result | | --- | --- | | Core feature regression: rolling VECTOR writer, schema validation, options, append writer, row tracking, paths, BLOB rolling, and data evolution utilities | 118 passed at feature commit `dff3266` | | Complete `paimon-data-evolution-table-test` after the fix | 294 passed, 12 skipped by format/parameter guards | | Complete `paimon-vortex-format-test` after the fix | 26 passed | | `paimon-write-and-read-inte-test --gtest_filter='Vortex*'` after the fix | 4 passed | | Java fixture generation and Java readback assertions | Passed | | Applicable changed-file pre-commit hooks and `git diff --check` | Passed | Whole-file clang-format was skipped to preserve unrelated existing formatting; new C++ files and modified lines were checked with clang-format `20.1.8`. The existing local macOS build uses temporary dependency/link workarounds and still emits duplicate-library and deployment-target linker warnings. Linux CI, the repository-wide `unittest` target, sanitizers, and C++-written data read by Java have not been run. ### API and Format - Add exported option constants `Options::VECTOR_FILE_FORMAT` and `Options::VECTOR_TARGET_FILE_SIZE` in `include/paimon/defs.h`. - Reuse existing `.vector.vortex` naming and data-evolution metadata (`writeCols`, `firstRowId`, and row counts); no new manifest version or protocol is introduced. - Dedicated storage is selected explicitly by `vector.file.format=vortex`; schemas and write contexts are validated before using this writer. ### Documentation - Update `docs/source/user_guide/data_types.rst` with Vortex VECTOR support, dedicated-mode prerequisites, limitations, and paired rolling behavior. - Add fixture provenance and reproducible Java generation instructions in `test/test_data/vortex/dedicated_vector_java_compat.db/dedicated_vector_java_compat/README.md`. ### Generative AI tooling Generated-by: OpenAI Codex (codex-cli 0.162.0-alpha.17.2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
