JingsongLi commented on PR #10030: URL: https://github.com/apache/paimon/pull/10030#issuecomment-5770620512
I do not see an end-to-end user path for the >2 GiB V2 container in this PR. The new large-file test calls `FileIndexFormat.createWriter` directly with a synthetic sparse stream, but the two production writers changed here (`DataFileIndexWriter` and `FileIndexProcessor`) still serialize the complete container into `ByteArrayOutputStream`; the rewrite path also calls `readAll()`. The PR itself leaves streaming writes as a TODO. Consequently setting `file-index.format.version=2` does not let a Paimon table create or rewrite an index beyond the current int-sized in-memory boundary, while it introduces a new persisted format and changes public reader/writer signatures. Please bring V2 back together with a streaming table write/rewrite path and an integration test that builds and queries an index beyond the V1 boundary. Closing this PR for now on end-to-end value grounds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
