westonpace commented on issue #36303: URL: https://github.com/apache/arrow/issues/36303#issuecomment-1631644231
To be clear, the part I don't understand is how a max_rows_per_file setting would have any real impact on memory usage. I would expect data to be written to the file immediately. I think, perhaps a more general but unpleasant answer, is that we should in theory be able to run with 2GB headroom but it will require testing, profiling, and some tweaking of the C++ code and nobody has done that yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
