[
https://issues.apache.org/jira/browse/SPARK-54753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xihuan updated SPARK-54753:
---------------------------
Description:
For Apache Spark *4.0.1* local mode, the memory is not released after long time
running, while downgrade the spark version {*}v3.5.6{*}, there is no issue.
We got the eclipse memory analyzer to check the heap dump.
h3. Problem Suspect 1
9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
*org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
occupy *972,232,520 (61.58%)* bytes.
Most of these instances are referenced from one instance of
{*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is referenced
by {*}org.apache.spark.sql.artifact.ArtifactManager @ 0x80a14948{*}, loaded by
{*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
Significant stack frames and local variables
* jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
The stacktrace of this Thread is available.
*Keywords*
* org.apache.hadoop.conf.Configuration
* org.springframework.boot.loader.launch.LaunchedClassLoader
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
* org.apache.spark.sql.artifact.ArtifactManager
* jdk.internal.ref.CleanerImpl.run()V
* CleanerImpl.java:140
h3. Problem Suspect 2
12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
*Keywords*
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
Could anyone help provide any insights on the issue? Thanks.
was:
For Apache Spark *4.0.1* local mode, the memory is not released after long time
running, while downgrade the spark version {*}v3.5.6{*}, there is no issue.
We got the eclipse memory analyzer to check the heap dump.
h3. Problem Suspect 1
9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
*org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
occupy *972,232,520 (61.58%)* bytes.
Most of these instances are referenced from one instance of
{*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is referenced
by {*}org.apache.spark.sql.artifact.ArtifactManager @ 0x80a14948{*}, loaded by
{*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
Significant stack frames and local variables
* jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
The stacktrace of this Thread is available. [See
stacktrace|file:///C:/Users/xihuan/AppData/Local/Temp/2/report8526993352425331840/pages/27.html].
[See stacktrace with involved local
variables|file:///C:/Users/xihuan/AppData/Local/Temp/2/report8526993352425331840/pages/28.html].
*Keywords*
* org.apache.hadoop.conf.Configuration
* org.springframework.boot.loader.launch.LaunchedClassLoader
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
* org.apache.spark.sql.artifact.ArtifactManager
* jdk.internal.ref.CleanerImpl.run()V
* CleanerImpl.java:140
h3. Problem Suspect 2
12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
*Keywords*
* jdk.internal.ref.CleanerImpl$PhantomCleanableRef
Could anyone help provide any insights on the issue? Thanks.
> possible memory leak in spark 4.0.1
> -----------------------------------
>
> Key: SPARK-54753
> URL: https://issues.apache.org/jira/browse/SPARK-54753
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.1
> Reporter: xihuan
> Priority: Major
>
> For Apache Spark *4.0.1* local mode, the memory is not released after long
> time running, while downgrade the spark version {*}v3.5.6{*}, there is no
> issue.
> We got the eclipse memory analyzer to check the heap dump.
> h3. Problem Suspect 1
> 9,109 instances of {*}org.apache.hadoop.conf.Configuration{*}, loaded by
> *org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888*
> occupy *972,232,520 (61.58%)* bytes.
> Most of these instances are referenced from one instance of
> {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*}, loaded by {*}<system
> class loader>{*}, which occupies *48 (0.00%)* bytes. The instance is
> referenced by {*}org.apache.spark.sql.artifact.ArtifactManager @
> 0x80a14948{*}, loaded by
> {*}org.springframework.boot.loader.launch.LaunchedClassLoader @ 0x80098888{*}.
> Thread *jdk.internal.misc.InnocuousThread @ 0x866c1710 Cleaner-1* has a local
> variable or reference to *jdk.internal.ref.CleanerImpl @ 0x866c1898* which is
> on the shortest path to {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef @
> 0x861b7638{*}. The thread *jdk.internal.misc.InnocuousThread @ 0x866c1710
> Cleaner-1* keeps local variables with total size *184 (0.00%)* bytes.
> Significant stack frames and local variables
> * jdk.internal.ref.CleanerImpl.run()V (CleanerImpl.java:140)
> ** jdk.internal.ref.CleanerImpl @ 0x866c1898 retains 24 (0.00%) bytes
> The stacktrace of this Thread is available.
> *Keywords*
> * org.apache.hadoop.conf.Configuration
> * org.springframework.boot.loader.launch.LaunchedClassLoader
> * jdk.internal.ref.CleanerImpl$PhantomCleanableRef
> * org.apache.spark.sql.artifact.ArtifactManager
> * jdk.internal.ref.CleanerImpl.run()V
> * CleanerImpl.java:140
> h3. Problem Suspect 2
> 12,291 instances of {*}jdk.internal.ref.CleanerImpl$PhantomCleanableRef{*},
> loaded by *<system class loader>* occupy *432,466,008 (27.39%)* bytes.
> *Keywords*
> * jdk.internal.ref.CleanerImpl$PhantomCleanableRef
>
> Could anyone help provide any insights on the issue? Thanks.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
