liupc opened a new pull request #23610: [SPARK-26684]Add logs when allocating large memory for PooledByteBufAllocator URL: https://github.com/apache/spark/pull/23610 ## What changes were proposed in this pull request? Currently, Spark use `PooledByteBufAllocator` to allocate memory for channel reading. However, the allocated heap/offheap memory size is not tracked. Sometimes, this make it difficult to find out the cause of OOM failures(for instance, direct memory oom). we have to use more advanced tools like MAT or pmap to locate the cause. Actually, we can add some logs for `PooledByteBufAllocator` when allocating large memory, which can facilitate the debugging. ## How was this patch tested? NA Please review http://spark.apache.org/contributing.html before opening a pull request.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
