morningman commented on PR #68885: URL: https://github.com/apache/doris/pull/68885#issuecomment-6100680816
Closing this PR. Master dropped the forced Paimon read batch on purpose. #65046 set `read.batch-size` to the session `batch_size` on every JNI read, and #65955 removed that so that a `read.batch-size` set on the table, the catalog or the query takes effect; since then master reads with Paimon's default of 1024 rows. The default this PR ported came back on branch-4.1 with #65868, which had changed that same line to `copyWithoutTimeTravel` for schema evolution before #66176 backported the removal, and #66297 later limited it to readers that set nothing. As the review points out, the default is not free either. It fixes Paimon's physical batch at the session `batch_size` (8160 by default) for the whole split, while adaptive batch sizing may ask for far fewer rows per block. Paimon's ORC and Parquet readers hold one batch per open file in the BE's embedded JVM, so for wide rows that batch grows eightfold, and neither branch-4.x nor this PR measured a gain to justify it. A user who wants larger batches can set `read.batch-size` on the table, the catalog or the query. The chain-table finding does not depend on this PR: master's manifest-parallelism bound rebuilds a chain table's `ChainGroupReadTable` as a plain fallback pair whenever the FE cap is below the BE's core count, and `dropCatalogLoader` on the FE rebuilds the pair branch by branch as well. That will be checked and fixed in a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
