zjw1111 opened a new pull request, #10101: URL: https://github.com/apache/paimon/pull/10101
### Purpose V2 file-index containers now support table writes and index rewrites when an index payload or the container exceeds 2 GiB. Previously, both production paths assembled the complete container in a `ByteArrayOutputStream`, so selecting V2 did not remove that size limit. The container writer accepts payloads through `writeIndex` and `FileIndexWriter.writeTo(OutputStream)`. `DataFileIndexWriter` and `FileIndexProcessor` write through a spillable output stream: small containers stay embedded in file metadata, while larger ones become independent index files. V2 records 64-bit payload positions and lengths in its footer; readers continue to accept V1 and V2. The plugin reader interface now takes a `long` payload length. Existing built-in index writers still serialize individual payloads as byte arrays, and V1 keeps its header-first, in-memory write behavior. V1 remains the default format. ### Tests - `FileIndexFormatFormatTest`: `testV2TotalSizeOverTwoGiBAndLegacyBackendAtLargePosition`, `testV2StreamedPayloadOverInt32`, and `testV2CopyPayloadOverInt32` cover large positions, a single large streamed payload, and copying that payload without a large byte array. `testIncrementalWriteAndCopyPayload` covers V1/V2 incremental writes, and `testV2RejectInvalidTailMagic` covers a damaged trailer. - `DataFileIndexWriterTest`: `testSpillableIndexOutputStream` covers the embed/spill boundary; `testV2TableWriterStreamsPayloadOverTwoGiB` writes a table index above 2 GiB; `testV2FailedWriteDeletesPartialIndexFile` checks cleanup after a failed write. - `FileIndexProcessorTest`: `testV2RewritesContainerOverTwoGiB` rewrites a table index above 2 GiB. The map-key rewrite test also runs with V1 and V2. - `CoreOptionsTest.testFileIndexInManifestThreshold` covers threshold bounds, and `FileIndexPredicateCloseTest.testFileStatusFailureDoesNotOpenStream` covers reader construction failure. - `FileIndexesTableTest` runs embedded and external index cases with both V1 and V2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
