Josh Rosen created SPARK-4495:
---------------------------------
Summary: Memory leak in JobProgressListener due to
`spark.ui.retainedJobs` not being used properly
Key: SPARK-4495
URL: https://issues.apache.org/jira/browse/SPARK-4495
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.2.0
Reporter: Josh Rosen
Assignee: Josh Rosen
Priority: Blocker
In the JobProgressListener, we never actually remove jobs from the jobIdToData
map. In fact, there's a somewhat embarrassing typo: _retailed_ jobs:
{code}
val retailedJobs = conf.getInt("spark.ui.retainedJobs", DEFAULT_RETAINED_JOBS)
{code}
This was introduced by my patch for [SPARK-2321]: I should have added a test
that set {{spark.ui.retainedJobs}} to some low threshold and checked that we
only reported information for that many jobs; there was already a corresponding
"test LRU eviction of stages" test that I should have just copied and modified.
This only affects the unreleased 1.2.0 version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
