jianguotian opened a new pull request, #9100: URL: https://github.com/apache/paimon/pull/9100
### Purpose Writing `BundleRecords` currently requires converting a vector batch into `InternalRow` objects and processing the rows one by one, even when the reader and format writer support the same vector representation. This PR introduces a format-neutral, opt-in contract for synchronously passing borrowed vector batches to compatible format writers. Existing bundle implementations continue to use the row-by-row path by default, so current Paimon behavior is unchanged. ### Changes - Add `BundleRecords.isDirectWriteBundle()` as an opt-in capability; its default value is `false`. - Expose borrowed Arrow batches through `ArrowVectorizedRecordIterator` and carry the final output `RowType` in `ArrowBundleRecords`. - Add `ArrowBatchReader.readVectorizedBatch()` for independent batch wrappers while preserving wrapper reuse in the existing `readBatch()` path. - Add a guarded direct-bundle path in `RowDataFileWriter` only when there are no auxiliary writers, row tracking, or statistics that require per-record collection. - Update generated sequence numbers using the exact bundle row count after a direct write. - Preserve row-by-row fallback for unmarked bundles, per-record statistics, row tracking, and file indexes. - Document the borrowed Arrow-buffer lifetime: the writer must consume the bundle synchronously before `writeBundle()` returns. ### Tests - `ArrowBatchReaderTest`: 2 tests passed, covering reusable row-batch wrappers and independent vectorized-batch wrappers. - `RowDataFileWriterTest`: 6 tests passed, covering the direct path, extractor-based statistics, and all guarded fallback paths. - Maven Checkstyle, Spotless, and Enforcer checks passed in the focused reactor build. - `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
