[
https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17215015#comment-17215015
]
Haoyuan Wang edited comment on SPARK-33000 at 10/15/20, 9:40 PM:
-----------------------------------------------------------------
[~nchammas] That'd be good. I was planning to take this opportunity to finally
touch some open source work. But realistically, I don't know if I'll get time.
Cleaning checkpointed file is done by
[ContextCleaner|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ContextCleaner.scala].
It's triggered by a JVM hook. In this case, a WeakReference hook. These hooks
need to be registered before JVM is launched.
As for checkpoint dir, it's just a property of SparkContext. So it can(have to)
be set after object is initiated.
There are a few more properties like this, that has to be set before
SparkContext initiation, mostly due to utilization of JVM hooks. Those are not
documented neither, however I don't have full list. But we are fixing one of
them!
was (Author: caowang888):
[~nchammas] That'd be good. I was planning to take this opportunity to finally
touch some open source work. But realistically, I don't know if I'll get time.
Cleaning checkpointed file is done by
[ContextCleaner|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ContextCleaner.scala].
It's triggered by a JVM hook. In this case, a WeakReference hook. These hooks
need to be registered before JVM is launched.
As for checkpoint dir, it's just a property of SparkContext. So it can(have to)
be set after object is initiated.
There are a few more properties like this, that has to be set before
SparkContext initiation, mostly due to utilization of JVM hooks. But these are
not documented neither, however I don't have full list. But we are fixing one
of them!
> cleanCheckpoints config does not clean all checkpointed RDDs on shutdown
> ------------------------------------------------------------------------
>
> Key: SPARK-33000
> URL: https://issues.apache.org/jira/browse/SPARK-33000
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.6
> Reporter: Nicholas Chammas
> Priority: Minor
>
> Maybe it's just that the documentation needs to be updated, but I found this
> surprising:
> {code:python}
> $ pyspark
> ...
> >>> spark.conf.set('spark.cleaner.referenceTracking.cleanCheckpoints', 'true')
> >>> spark.sparkContext.setCheckpointDir('/tmp/spark/checkpoint/')
> >>> a = spark.range(10)
> >>> a.checkpoint()
> DataFrame[id: bigint]
>
> >>> exit(){code}
> The checkpoint data is left behind in {{/tmp/spark/checkpoint/}}. I expected
> Spark to clean it up on shutdown.
> The documentation for {{spark.cleaner.referenceTracking.cleanCheckpoints}}
> says:
> {quote}Controls whether to clean checkpoint files if the reference is out of
> scope.
> {quote}
> When Spark shuts down, everything goes out of scope, so I'd expect all
> checkpointed RDDs to be cleaned up.
> For the record, I see the same behavior in both the Scala and Python REPLs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
