[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608047#comment-16608047 ] Michael Spector commented on SPARK-23670: - Reported: SPARK-25380 > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Assignee: Myroslav Lisniak >Priority: Major > Fix For: 2.3.1, 2.4.0 > > Attachments: Screen Shot 2018-09-06 at 23.19.56.png, heap.png, > heapdump_OOM.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606355#comment-16606355 ] Marcelo Vanzin commented on SPARK-23670: Yep, looks like just huge plans. Do you mind opening a separate bug? That's not really a memory leak, but probably good to have some control over how much data is kept here if the plans can get that large. > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Assignee: Myroslav Lisniak >Priority: Major > Fix For: 2.3.1, 2.4.0 > > Attachments: Screen Shot 2018-09-06 at 23.19.56.png, heap.png, > heapdump_OOM.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606354#comment-16606354 ] Michael Spector commented on SPARK-23670: - !Screen Shot 2018-09-06 at 23.19.56.png|width=756,height=340! > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Assignee: Myroslav Lisniak >Priority: Major > Fix For: 2.3.1, 2.4.0 > > Attachments: Screen Shot 2018-09-06 at 23.19.56.png, heap.png, > heapdump_OOM.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[
https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606343#comment-16606343
]
Marcelo Vanzin commented on SPARK-23670:
Do you mind listing the {{SparkPlanGraphNodeWrapper}} and
{{SparkPlanGraphWrapper}} instances in your heap dump and checking their
retained size?
>From the (truncated) screenshot you posted it just looks like the plan
>descriptions are huge. This seems like a separate issue; could be a feature
>request to truncate large plans in the UI.
> Memory leak of SparkPlanGraphWrapper in sparkUI
> ---
>
> Key: SPARK-23670
> URL: https://issues.apache.org/jira/browse/SPARK-23670
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Myroslav Lisniak
>Assignee: Myroslav Lisniak
>Priority: Major
> Fix For: 2.3.1, 2.4.0
>
> Attachments: heap.png, heapdump_OOM.png
>
>
> Memory leak on driver for a long time running application. We have
> application using structured streaming and running 48 hours. But driver fails
> with out of memory after 25 hours. After investigating heap dump we found
> that most of the memory was occupied with a lot of *SparkPlanGraphWrapper*
> objects inside *InMemoryStore*.
> Application was run with option:
> --driver-memory 4G
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606295#comment-16606295 ] Michael Spector commented on SPARK-23670: - Not many: 7, 7 and 49. > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Assignee: Myroslav Lisniak >Priority: Major > Fix For: 2.3.1, 2.4.0 > > Attachments: heap.png, heapdump_OOM.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[
https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606151#comment-16606151
]
Marcelo Vanzin commented on SPARK-23670:
How many instances of the following classes do you have in your dump:
- {{SQLExecutionUIData}}
- {{SparkPlanGraphWrapper}}
- {{SparkPlanGraphNodeWrapper}}
With the fix I'd expect the first two to be capped at 1000 (unless you changed
the config), but the latter may still be high if you have complicated queries.
That could explain high memory usage.
> Memory leak of SparkPlanGraphWrapper in sparkUI
> ---
>
> Key: SPARK-23670
> URL: https://issues.apache.org/jira/browse/SPARK-23670
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.3.0
>Reporter: Myroslav Lisniak
>Assignee: Myroslav Lisniak
>Priority: Major
> Fix For: 2.3.1, 2.4.0
>
> Attachments: heap.png, heapdump_OOM.png
>
>
> Memory leak on driver for a long time running application. We have
> application using structured streaming and running 48 hours. But driver fails
> with out of memory after 25 hours. After investigating heap dump we found
> that most of the memory was occupied with a lot of *SparkPlanGraphWrapper*
> objects inside *InMemoryStore*.
> Application was run with option:
> --driver-memory 4G
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605642#comment-16605642 ] Michael Spector commented on SPARK-23670: - The bug still happens in Apache Spark 2.3.1: !heapdump_OOM.png|width=1595,height=966! These char[] objects occupy 88% of driver memory. > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Assignee: Myroslav Lisniak >Priority: Major > Fix For: 2.3.1, 2.4.0 > > Attachments: heap.png, heapdump_OOM.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23670) Memory leak of SparkPlanGraphWrapper in sparkUI
[ https://issues.apache.org/jira/browse/SPARK-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397104#comment-16397104 ] Apache Spark commented on SPARK-23670: -- User 'myroslavlisniak' has created a pull request for this issue: https://github.com/apache/spark/pull/20813 > Memory leak of SparkPlanGraphWrapper in sparkUI > --- > > Key: SPARK-23670 > URL: https://issues.apache.org/jira/browse/SPARK-23670 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.0 >Reporter: Myroslav Lisniak >Priority: Major > Attachments: heap.png > > > Memory leak on driver for a long time running application. We have > application using structured streaming and running 48 hours. But driver fails > with out of memory after 25 hours. After investigating heap dump we found > that most of the memory was occupied with a lot of *SparkPlanGraphWrapper* > objects inside *InMemoryStore*. > Application was run with option: > --driver-memory 4G -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
