jayzhan211 commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-6383079505
@fwojciec Sorry for the slow reply, and thanks for the repro. I reproduced it on main: for a `Partitioned` join over Parquet, the memory still allocated after the join's stream is dropped (with the plan alive) is 16 MB at 1M rows and 153 MB at 10M rows, with 0 bytes reserved. `CollectLeft` keeps its map reserved through the shared build, and inputs that don't accept the dynamic filter don't show it, so it comes from the filter holding the `Partitioned` maps. I'd go with your first option: keep a reservation together with the map, so the memory stays charged for as long as anything holds the map. That fixes the accounting however long the plan lives. Your second option would free the memory sooner, but it changes when the filter is updated, which #26641 is also touching, so I'd leave it for a follow-up. Please go ahead with a separate issue and a PR on main, and I'm happy to review. A separate issue is also easier to track from the memory accounting discussion in #25650 than a comment in this epic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
