cetra3 commented on PR #4582: URL: https://github.com/apache/datafusion-comet/pull/4582#issuecomment-4667144199
We've been using https://github.com/cetra3/thresher ourselves to handle something similar with datafusion. Namely, we log a trace when the threshold is reached with the heap dump of jemalloc, based upon [the example](https://github.com/cetra3/thresher/blob/main/examples/jemalloc.rs) It's helped us so far nail down some gnarly memory allocation related bugs with our compaction process. I'm hoping to expand this to queries soon -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
