[ https://issues.apache.org/jira/browse/SPARK-25008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-25008. ---------------------------------- Resolution: Incomplete > Add memory mode info to showMemoryUsage in TaskMemoryManager > ------------------------------------------------------------ > > Key: SPARK-25008 > URL: https://issues.apache.org/jira/browse/SPARK-25008 > Project: Spark > Issue Type: Task > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Ankur Gupta > Priority: Major > Labels: bulk-closed > > TaskMemoryManager prints the current memory usage information before throwing > an OOM exception which is helpful in debugging issues. This log does not have > the memory mode information which can be also useful to quickly determine > which memory users need to increase. > This JIRA is to add that information to showMemoryUsage method of > TaskMemoryManager. > Current logs: > {code} > 18/07/03 17:57:16 INFO memory.TaskMemoryManager: Memory used in task 318 > 18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by > org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@7f084d1b: > 1024.0 KB > 18/07/03 17:57:16 INFO memory.TaskMemoryManager: Acquired by > org.apache.spark.shuffle.sort.ShuffleExternalSorter@713d50f2: 32.0 KB > 18/07/03 17:57:16 INFO memory.TaskMemoryManager: 0 bytes of memory were used > by task 318 but are not associated with specific consumers > 18/07/03 17:57:16 INFO memory.TaskMemoryManager: 1081344 bytes of memory are > used for execution and 306201016 bytes of memory are used for storage > 18/07/03 17:57:16 ERROR executor.Executor: Exception in task 86.0 in stage > 49.0 (TID 318) > java.lang.OutOfMemoryError: Unable to acquire 326284160 bytes of memory, got > 3112960 > at > org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:127) > at > org.apache.spark.shuffle.sort.ShuffleExternalSorter.acquireNewPageIfNecessary(ShuffleExternalSorter.java:359) > at > org.apache.spark.shuffle.sort.ShuffleExternalSorter.insertRecord(ShuffleExternalSorter.java:382) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriter.insertRecordIntoSorter(UnsafeShuffleWriter.java:246) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriter.write(UnsafeShuffleWriter.java:167) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org