[
https://issues.apache.org/jira/browse/SPARK-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-4495:
------------------------------
Summary: Memory leak in JobProgressListener due to `spark.ui.retainedJobs`
not being used (was: Memory leak in JobProgressListener due to
`spark.ui.retainedJobs` not being used properly)
> Memory leak in JobProgressListener due to `spark.ui.retainedJobs` not being
> used
> --------------------------------------------------------------------------------
>
> Key: SPARK-4495
> URL: https://issues.apache.org/jira/browse/SPARK-4495
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Blocker
>
> In the JobProgressListener, we never actually remove jobs from the
> jobIdToData map. In fact, there's a somewhat embarrassing typo: _retailed_
> jobs:
> {code}
> val retailedJobs = conf.getInt("spark.ui.retainedJobs",
> DEFAULT_RETAINED_JOBS)
> {code}
> This was introduced by my patch for [SPARK-2321]: I should have added a test
> that set {{spark.ui.retainedJobs}} to some low threshold and checked that we
> only reported information for that many jobs; there was already a
> corresponding "test LRU eviction of stages" test that I should have just
> copied and modified.
> This only affects the unreleased 1.2.0 version.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
