comphead commented on issue #14510: URL: https://github.com/apache/datafusion/issues/14510#issuecomment-2637366054
That is nice @berkaysynnada I love it. To implement it we have to use a `memory reservation` in most of intensive operations, like Sorting, Join, hash, scan, etc, now we have quite few operations supporting memory reservation, the spill support even fewer. To break down this into manageable pieces we need to - figure out what parts of execution plan we wanna monitor - create a subissue to integrate memory reservation for each part of exec plan - Display it back to the user. We already have a metrics builder on top of the `ExecutionPlan` and we can access metrics through `EXPLAIN ANALYZE` so probably this can be a substitute for the builtin function? Should we output per partition or aggregated, max value or mean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
