c21 commented on pull request #32198: URL: https://github.com/apache/spark/pull/32198#issuecomment-828537591
> this doesn't do any sort of memory tracking right? Yes. It seems to me there's no way to track the memory usage accurately because writer is using on-heap memory. And we need memory usage information available to retrieve from each individual writer implementation (Parquet, ORC, Aveo, etc), which is not the case right now. One immature idea though is to look at executor JVM heap memory usage (which I think should already be captured). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
