alamb commented on PR #20839: URL: https://github.com/apache/datafusion/pull/20839#issuecomment-4038072624
> Thanks @Dandandan just yesterday I was looking into datafusion parquet reads takes more memory comparing to jvm+native and came to some hotspots, one of them as you mentioned in `get_byte_ranges` I think some of the changes we are planning with "Morsels" may make it easier to control memory usage (e.g. not buffer entire row groups). However, I don't have a concrete example to show yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
