JingsongLi opened a new pull request, #9112: URL: https://github.com/apache/paimon/pull/9112
## What changed - add `process_row_id_ranges` to plan logical file-group ranges and invoke a user processor sequentially in target-sized batches - keep overlapping base, data-evolution, BLOB, and VECTOR files in one indivisible range group - fix row-id updates of columns added after existing data by retaining `_ROW_ID` as a row-count anchor - document a resumable Ray embedding backfill from BLOB payloads with one commit per approximately one million rows ## Why Long-running transforms such as embedding generation need bounded commits and application-owned distributed processing. The new API provides stable row-id/file boundaries while leaving Ray reads, model execution, commits, and retry policy to the processor. The row-count anchor fix is required for first-time backfills of a newly added nullable VECTOR column; without it, reading only a column absent from old files yielded a zero-row table. ## Validation - `python -m pytest -q pypaimon/tests/ray_process_row_id_ranges_test.py pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest::test_backfill_vector_column_added_after_existing_rows` — 6 passed, 7 subtests passed - `python -m pytest -q pypaimon/tests/table_update_test.py` — 96 passed, 14 subtests passed - `python -m pytest -q pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest -k 'not vortex'` — 5 passed, 1 skipped, 1 deselected - changed Python files compile with `py_compile` - the embedding documentation example parses with `ast.parse` The full vector test class was not completed locally because the installed Vortex native extension segfaults under Python 3.13 in the existing Vortex-specific test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
