[jira] [Resolved] (SPARK-27417) CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
[
https://issues.apache.org/jira/browse/SPARK-27417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yangpengyu resolved SPARK-27417.
Resolution: Fixed
> CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory
> in their stop() methods
> ---
>
> Key: SPARK-27417
> URL: https://issues.apache.org/jira/browse/SPARK-27417
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
>Reporter: yangpengyu
>Assignee: Josh Rosen
>Priority: Critical
> Fix For: 1.6.0
>
>
> I discovered multiple leaks of shuffle memory while working on my memory
> manager consolidation patch, which added the ability to do strict memory leak
> detection for the bookkeeping that used to be performed by the
> ShuffleMemoryManager. This uncovered a handful of places where tasks can
> acquire execution/shuffle memory but never release it, starving themselves of
> memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
> memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
> {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing
> its resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-27417) CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
[
https://issues.apache.org/jira/browse/SPARK-27417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yangpengyu closed SPARK-27417.
--
> CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory
> in their stop() methods
> ---
>
> Key: SPARK-27417
> URL: https://issues.apache.org/jira/browse/SPARK-27417
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
>Reporter: yangpengyu
>Assignee: Josh Rosen
>Priority: Critical
> Fix For: 1.6.0
>
>
> I discovered multiple leaks of shuffle memory while working on my memory
> manager consolidation patch, which added the ability to do strict memory leak
> detection for the bookkeeping that used to be performed by the
> ShuffleMemoryManager. This uncovered a handful of places where tasks can
> acquire execution/shuffle memory but never release it, starving themselves of
> memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
> memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
> {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing
> its resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-27417) CLONE - ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
yangpengyu created SPARK-27417:
--
Summary: CLONE - ExternalSorter and ExternalAppendOnlyMap should
free shuffle memory in their stop() methods
Key: SPARK-27417
URL: https://issues.apache.org/jira/browse/SPARK-27417
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
Reporter: yangpengyu
Assignee: Josh Rosen
Fix For: 1.6.0
I discovered multiple leaks of shuffle memory while working on my memory
manager consolidation patch, which added the ability to do strict memory leak
detection for the bookkeeping that used to be performed by the
ShuffleMemoryManager. This uncovered a handful of places where tasks can
acquire execution/shuffle memory but never release it, starving themselves of
memory.
Problems that I found:
* {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
memory.
* BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
{{CompletionIterator}}.
* {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing its
resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11293) ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their stop() methods
[
https://issues.apache.org/jira/browse/SPARK-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813495#comment-16813495
]
yangpengyu commented on SPARK-11293:
I hit the sam problem when I run TPCH test on spark1.6.0.
my dataset scale is SF=1000，
Environment as follows:
1Master 3 Worker,
onHeapMemory=10g,
offHeapMemory=20g,
24threads/Worker
the query3 and query17 detected memory leak.Some logs are as follows:
9/04/09 21:57:59 ERROR Executor: Managed memory leak detected; size = 536870912
bytes, TID = 2685
41 19/04/09 21:58:16 WARN TaskMemoryManager: leak 512.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@385b3b3f
42 19/04/09 21:58:16 ERROR Executor: Managed memory leak detected; size =
536870912 bytes, TID = 2683
43 19/04/09 21:58:16 WARN TaskMemoryManager: leak 512.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@35be7b55
44 19/04/09 21:58:16 ERROR Executor: Managed memory leak detected; size =
536870912 bytes, TID = 2703
45 19/04/09 21:58:20 WARN TaskMemoryManager: leak 512.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@50f93582
46 19/04/09 21:58:20 ERROR Executor: Managed memory leak detected; size =
536870912 bytes, TID = 2709
47 19/04/09 21:58:21 WARN TaskMemoryManager: leak 512.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@28e3ec7a
48 19/04/09 21:58:21 ERROR Executor: Managed memory leak detected; size =
536870912 bytes, TID = 2723
49 19/04/09 21:59:50 WARN TaskMemoryManager: leak 512.0 MB memory from
org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@5b2f5dbc
50 19/04/09 21:59:50 ERROR Executor: Managed memory leak detected; size =
536870912 bytes, TID = 2687
51 19/04/09 22:00:50 WARN TransportChannelHandler: Exception in connection
from hw083/172.18.11.83:42989
> ExternalSorter and ExternalAppendOnlyMap should free shuffle memory in their
> stop() methods
> ---
>
> Key: SPARK-11293
> URL: https://issues.apache.org/jira/browse/SPARK-11293
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.3.1, 1.4.1, 1.5.1, 1.6.0
>Reporter: Josh Rosen
>Assignee: Josh Rosen
>Priority: Critical
> Fix For: 1.6.0
>
>
> I discovered multiple leaks of shuffle memory while working on my memory
> manager consolidation patch, which added the ability to do strict memory leak
> detection for the bookkeeping that used to be performed by the
> ShuffleMemoryManager. This uncovered a handful of places where tasks can
> acquire execution/shuffle memory but never release it, starving themselves of
> memory.
> Problems that I found:
> * {{ExternalSorter.stop()}} should release the sorter's shuffle/execution
> memory.
> * BlockStoreShuffleReader should call {{ExternalSorter.stop()}} using a
> {{CompletionIterator}}.
> * {{ExternalAppendOnlyMap}} exposes no equivalent of {{stop()}} for freeing
> its resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
