Georg Heiler created SPARK-18887:
------------------------------------
Summary: Executor OOM due to tungsten memory leak in external
sorter
Key: SPARK-18887
URL: https://issues.apache.org/jira/browse/SPARK-18887
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.2
Environment: -master local[*] --driver-memory=14G --conf
spark.memory.fraction=0.8, 0.4 and 0.6 show all the same problem
Reporter: Georg Heiler
When performing a collect /write.parquet on
http://stackoverflow.com/questions/41169873/spark-dynamic-dag-is-a-lot-slower-and-different-from-hard-coded-dag
I can observe problems with Tungstens memory management. The problem is
similar to https://issues.apache.org/jira/browse/SPARK-14363 which should be
fixed in 2.0.0 - but I am using 2.0.2.
Playing with spark.memory.fraction did not fix the issue.
6/12/15 19:15:56 WARN TaskMemoryManager: leak 64.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@62d91577
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@264a5b7b in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@18b52a29 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@49965a33 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@44c939b8 in task 15307
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@a074a26 in task 15307
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@7ad8ad95 in task 15312
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@ab0aec8 in task 15307
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@64014c38 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@10e44240 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@21b1d617 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@5a8d57d0 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@e91953 in task 15314
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@4d33cd6d in task 15310
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@30aa29c9 in task 15310
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@69979ca3 in task 15310
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@646416fd in task 15310
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@3403b035 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@64712b7 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@29e97d06 in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@19ea8c3c in task 15308
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@5cc91c89 in task 15312
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@6f792272 in task 15312
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@73a12102 in task 15312
16/12/15 19:15:56 WARN TaskMemoryManager: leak a page:
org.apache.spark.unsafe.memory.MemoryBlock@37eff7b9 in task 15310
16/12/15 19:15:56 ERROR Executor: Exception in task 0.0 in stage 272.0 (TID
15307)
java.lang.OutOfMemoryError: Unable to acquire 1252 bytes of memory, got 0
at
org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:129)
at
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:377)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
