GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/6220
[SPARK-7689] Deprecate spark.cleaner.ttl
With the introduction of `ContextCleaner` (in #126), I think there's no
longer any reason for users to enable the MetadataCleaner /
`spark.cleaner.ttl`. This patch removes the last remaining documentation for
`spark.cleaner.ttl` and logs a deprecation warning if it is used.
I think that this configuration used to be relevant for Spark Streaming
jobs, but I think that's no longer the case since the latest Streaming docs
have removed all mentions of `spark.cleaner.ttl` (see
https://github.com/apache/spark/pull/4956/files#diff-dbee746abf610b52d8a7cb65bf9ea765L1817,
for example). The TTL-based cleaning is not safe and may prematurely clean
resources that are still being used, leading to confusing errors (such as
https://issues.apache.org/jira/browse/SPARK-5594), so it generally should not
be enabled (see
http://apache-spark-user-list.1001560.n3.nabble.com/is-spark-cleaner-ttl-safe-td2557.html
for an old, related discussion).
The only use-case that I can think of is super-long-lived Spark REPLs where
you're worried about orphaning RDDs or broadcast variables in your REPL history
and having them never get cleaned up, but I don't know that anyone uses
`spark.cleaner.ttl` for this in practice.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark SPARK-7689
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6220.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6220
----
commit f6e3d4546fa4f1ff75da167630f8bcaf53def609
Author: Josh Rosen <[email protected]>
Date: 2015-05-17T17:45:01Z
Deprecate spark.cleaner.ttl
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
