Re: Possibly a memory leak issue in Spark

2021-09-24 Thread tenglei
: 2021年9月23日 2:57 收件人: Russell Spitzer 抄送: Sean Owen ; user 主题: Re: Possibly a memory leak issue in Spark I'm seeing millions of Stage / Job / Task data jcmd GC.class_histogram shows -- 6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper 25: 3765152 180727296

Re: Possibly a memory leak issue in Spark

2021-09-22 Thread Kohki Nishio
I'm seeing millions of Stage / Job / Task data jcmd GC.class_histogram shows -- 6: 7835346 2444627952 org.apache.spark.status.TaskDataWrapper 25: 3765152 180727296 org.apache.spark.status.StageDataWrapper 88:2322559290200

Re: Possibly a memory leak issue in Spark

2021-09-22 Thread Russell Spitzer
As Sean said I believe you want to be setting spark.ui.retainedJobs 1000How many jobs the Spark UI and status APIs remember before garbage collecting. This is a target maximum, and fewer elements may be retained in some circumstances. 1.2.0 spark.ui.retainedStages 1000How many