JunRuiLee opened a new pull request, #423: URL: https://github.com/apache/paimon-rust/pull/423
## Purpose Add read-path support for dedicated `.vector.<format>` files in data-evolution splits, including **rolling (multi-segment)** vector columns. When a writer rolls a vector column into multiple segments (upstream `DedicatedFormatRollingFileWriter` uses an independent `VECTOR_TARGET_FILE_SIZE`), the reader must reassemble them. This mirrors upstream Paimon's `VectorFileBunch` non-pushdown semantics in `DataEvolutionSplitRead.java`. This is a real gap, not a deferred feature: even with no Rust vector writer, the reader must be able to consume Java-written rolled tables. ## Changes All changes are confined to the read path (`crates/paimon/src/table/data_evolution_reader.rs`, plus a small detector in `blob_file_writer.rs`). - **Classify & route vector files**: `is_vector_store_file_name` detector; `normalize_merge_group` classifies `.vector.` files into their own group, excludes them from the merge anchor, and forces the column-merge path. - **`VectorBunch`** (modeled on the existing `BlobBunch`): aggregates rolled segments belonging to one logical vector source, keyed by `(schema_id, format_suffix, normalized_write_cols)`. `add` enforces continuity/dedup (same-`first_row_id` lower-seq -> ignore, overlap lower-seq -> ignore, gap -> error, row-count overflow -> error, key-identity mismatch -> error). - **Segment ordering**: `normalize_merge_group` sorts vector segments by `first_row_id` asc, then `max_sequence_number` desc, and no longer validates rolled segments (which are slices) against the normal-file row range. - **`normalized_write_cols`**: computed per vector file in `load_file_infos` (write columns sorted by field position; missing `write_cols` -> `DataInvalid`), so differently-ordered raw write columns that normalize equal aggregate into one bunch. - **`build_source_plan`**: aggregates same-key segments into a single `FieldSource::VectorBunch`, with a bunch-granularity duplicate-field-id guard and a bunch row-count check symmetric to the blob check. A single vector file is simply a bunch of length 1 -- one code path, no special-casing. - **Read**: `VectorBunch` is read via the same sequential-concat path as `BlobBunch`; each segment carries its real `first_row_id`/`row_count`, so the existing `to_local_row_ranges` mechanism clips absolute row ranges per segment correctly. ## Out of scope Write-side changes; blob fallback-placeholder optimization; predicate/vector pushdown into vector files; upstream's `rowIdPushDown` branch (non-pushdown semantics only). ## Tests - Unit: `VectorBunch::add` guards (gap / overlap-lower-seq / same-first_row_id dedup / row-count overflow / key-identity mismatch); `normalize_merge_group` multi-segment sort and removal of the vector range check; `normalize_vector_write_cols` sort-by-position and missing/unknown-column errors; `build_source_plan` aggregation (incl. differently-ordered write cols collapsing to one bunch) and cross-bunch duplicate-field-id rejection. - Integration (end-to-end): normal `data.parquet` + 3 rolled `.vector.parquet` segments reassemble into the correct column in order (including a null row); `row_ranges` selecting rows **across a segment boundary** return the correct subset. All 41 `data_evolution_reader` tests pass; clippy clean. > Note: this builds on the earlier dedicated-vector classification/routing commits; the diff against `main` includes those as they have not yet landed upstream. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
