LukasBec commented on issue #11043: URL: https://github.com/apache/arrow/issues/11043#issuecomment-913493475
Thanks for the answer! > One thing to note is that you are creating 400 columns even if you are only using 100 of them. This makes the row group metadata about 4x larger than it needs to be. I am actually also using all 400 columns but i see that my example is a little confusing. > This is probably just an artifact of the way you are measuring memory usage. The VSZ/RSS numbers tend to grow in chunks because malloc will try and reduce the # of allocations it needs to make. Ah yes i forgot about that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
