wiedld commented on issue #16904: URL: https://github.com/apache/datafusion/issues/16904#issuecomment-3211276732
@alamb -- I feel like we need a new epic to delineate the different needs, and then consider whether we have solutions to address these needs. I'm going to take a shot at consolidating the identified needs: * more fine grained memory tracking, per https://github.com/apache/datafusion/issues/16904#issuecomment-3193801681 (**this ticket**) * This may be about tags or "sub-consumers" under a MemoryConsumer. * Or in some cases, perhaps using the existing `MemoryConsumer` with certain terminology (prefixes) at sites of use within the execution plans. * etc * provide APIs for memory tracking in non-error situations (a.k.a. Runtime Monitoring), per https://github.com/apache/datafusion/issues/14510 * imo, this expands beyond the existing [report_top](https://github.com/apache/datafusion/blob/2f61429eef12b3326f50b838f58a50738c78cc8d/datafusion/execution/src/memory_pool/pool.rs#L325), and is more aligned with the `ConsumerMemoryMetrics` in this open PR: https://github.com/apache/datafusion/pull/17021 * provide CLI tooling for doing the memory profiling * e.g. the `\memory_profiling` in the same PR referenced above: https://github.com/apache/datafusion/pull/17021 * consider how memory is accounted for in shared data buffers * current datafusion ticket: https://github.com/apache/datafusion/issues/16841 * perhaps building on work done for this epic in arrow-rs: https://github.com/apache/arrow-rs/issues/8137 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
