thinkharderdev commented on issue #13831: URL: https://github.com/apache/datafusion/issues/13831#issuecomment-2553813843
> This is a good find. > > Given your description it sounds like the storage for the group values is what is taking the memory > > ```sql > group by truncated_time, k8s_deployment_name, message > ``` > > The aggregate operator does account for the memory stored in the groups here: > > https://github.com/apache/datafusion/blob/63ce4865896b906ca34fcbf85fdc55bff3080c30/datafusion/physical-plan/src/aggregates/row_hash.rs#L900-L899 > > However, I believe the memory accounting is only updated after processing an entire batch of values. > > So for example, if you are using a batch of 8000 rows and each row has values 8k, that means at least 256 MB will be allocated (and since your query has 3 columns that may be even higher) > > I can think of two possible solutuions: > > 1. Use a smaller batch size for such queries so the memory accounting is more fine grained > 2. Leave more "slop" in the configured limits (e.g. set the maximum memory limit to be 500MB less than you actually have, for example) I think part of the issue is that we are accounting for the memory used by the actual values but the underlying `Vec` will always grow by 2x when it needs to grow. So in scenarios where we have a lot of group values the memory accounting can diverge significantly from what we are actually allocating from the OS -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
