StuartHadfield commented on issue #36303: URL: https://github.com/apache/arrow/issues/36303#issuecomment-1607666771
Hmmm so after being basically stable at 1GB usage until 20m rows, the 20m-30m part has increased memory consumption to about 1.6GB. It seems likely that using a 500k row limit will OOM my container. In this case, is my only option for making writes more memory efficient to use a lower row limit per file? Or perhaps if I lower the max number of open files, that might help too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
