: 2021年9月23日 2:57
收件人: Russell Spitzer
抄送: Sean Owen ; user
主题: Re: Possibly a memory leak issue in Spark
I'm seeing millions of Stage / Job / Task data
jcmd GC.class_histogram shows
--
6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
25: 3765152 180727296
I'm seeing millions of Stage / Job / Task data
jcmd GC.class_histogram shows
--
6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper
25: 3765152 180727296 org.apache.spark.status.StageDataWrapper
88:2322559290200
As Sean said I believe you want to be setting
spark.ui.retainedJobs 1000How many jobs the Spark UI and status APIs
remember before garbage collecting. This is a target maximum, and fewer
elements may be retained in some circumstances. 1.2.0
spark.ui.retainedStages 1000How many