Gabriel39 opened a new pull request, #9243: URL: https://github.com/apache/paimon/pull/9243
### Purpose Add a thread-safe `ReadBatchSizeController` so integrations can adapt the physical read batch size while a reader is running. - Allocate Parquet and ORC vectors once using the controller's immutable maximum batch size. - Snapshot the requested batch size at the start of each physical batch, keeping updates within `[1, maxBatchSize]`. - Share one controller instance through table reads, split reads, format contexts, and all lazily created readers. - Keep already-started and asynchronously prefetched batches on their previous requested size. - Preserve existing reader APIs and behavior when no controller is configured. Shrinking the requested size only changes the logical row count; it does not resize vectors. Each in-flight physical batch retains vectors sized for the maximum, so asynchronous ORC prefetch can increase memory usage by the number of pooled batches. ### Tests - `mvn -pl paimon-core -am -DfailIfNoTests=false -Dtest=ReadBatchSizeControllerTest,ParquetReadWriteTest#testDynamicReadBatchSize,OrcReaderFactoryTest#testDynamicReadBatchSize,RawFileSplitReadTest#testTableReadSharesDynamicBatchSizeController,AsyncRecordReaderTest#testPrefetchedBatchesMayUsePreviousRequestedSize,PrimaryKeyVectorPositionReaderTest#testKeyValueTableReadRoutesPhysicalSplitToRawReader test` - `mvn -pl paimon-format -DfailIfNoTests=false -Dtest=ParquetReadWriteTest,OrcReaderFactoryTest test` - `mvn -pl paimon-core -am -DfailIfNoTests=false -Dtest=RawFileSplitReadTest,AsyncRecordReaderTest test` - `mvn -pl paimon-core -am -DskipTests verify` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
