[
https://issues.apache.org/jira/browse/SPARK-54753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xihuan updated SPARK-54753:
---------------------------
Description:
For Apache Spark *4.0.1* local mode, the memory is not released after long time
running, while downgrade the spark version {*}v3.5.6{*}, there is no issue.
The issue can be reproduced with simple test cases:
```
```
was:
For Apache Spark *4.0.1* local mode, the memory is not released after long time
running, while downgrade the spark version {*}v3.5.6{*}, there is no issue.
We got the eclipse memory analyzer to check the heap dump.
h3. Problem Suspect 1
9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
*org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
occupy *972,232,520 (61.58%)* bytes.
Most of these instances are referenced from one instance of
{*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is referenced
by {*}org.apache.spark.sql.artifact.ArtifactManager @ 0x80a14948{*}, loaded by
{*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
Significant stack frames and local variables
* jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
The stacktrace of this Thread is available.
*Keywords*
* org.apache.hadoop.conf.Configuration
* org.springframework.boot.loader.launch.LaunchedClassLoader
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
* org.apache.spark.sql.artifact.ArtifactManager
* jdk.internal.ref.CleanerImpl.run()V
* CleanerImpl.java:140
h3. Problem Suspect 2
12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
*Keywords*
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
Could anyone help provide any insights on the issue? Thanks.
> memory leak in Apache Spark 4.0.1
> ---------------------------------
>
> Key: SPARK-54753
> URL: https://issues.apache.org/jira/browse/SPARK-54753
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.1
> Reporter: xihuan
> Priority: Major
>
> For Apache Spark *4.0.1* local mode, the memory is not released after long
> time running, while downgrade the spark version {*}v3.5.6{*}, there is no
> issue.
> The issue can be reproduced with simple test cases:
> ```
> ```
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
