tustvold commented on issue #4973: URL: https://github.com/apache/arrow-rs/issues/4973#issuecomment-1774062770
A brief look at the linked code has a max row group size of 100? This will lead to a huge number of pages, the metadata for which must be buffered before it can all be flushed when writing the file footer. This would be my guess as to what is occurring. A more typical row group limit would be 100,000 or more, and this will lead to more reasonably sized pages. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
