calluw commented on issue #39444: URL: https://github.com/apache/arrow/issues/39444#issuecomment-1891826511
Yes, that does explain why the max batch size that I could get to work was `2^11` (`16 == 2^4` -> `2^4 * 2^11 == 2^15`). It also explains why this didn't vary with increased row size, since `2^15` is the turning point at which multiple row groups are written. Still not clear why `2^17` and above couldn't be made to work with batch size `2^11` since it should matter, but I can confirm that setting `batch_readahead=0` does work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
