yossibm commented on issue #13949: URL: https://github.com/apache/arrow/issues/13949#issuecomment-1225337819
3.2GB is accross all the parquet files, but if created without the dictionary encoding it was around 11GB, so I suspected it loaded the entire files. I have noticed the 64M row groups and tried with much lower sizes, such as 128, but it had the same effect. anyway, I couldn't afford to invest more time in this so yesterday I have converted all of my files (which are much more then 20) to feather and it works fine. Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
