folkvir commented on issue #44295: URL: https://github.com/apache/arrow/issues/44295#issuecomment-2605050536
It could be related to this issue because of the example I use here. But the issues are not exactly the same. Here it appears to be a problem with how metadata are handled. Due to the row_group_size set to 1 the metadata are collected and written at the very end of the export. And with a row_group_size set to 1 the metadata collected are huge! It also seems they are not correctly garbaged at the end of the process because the memory usage is not garbaged correctly at the end of the export. For "fixing" our issues, we set at least a row_group_size around 10 000 to reduce the memory usage and the number of collected metadata. It seems to be almost stable with such values. Hence, also reducing the size of the footer. I don't have other advice to give. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
