nchammas opened a new pull request #31742:
URL: https://github.com/apache/spark/pull/31742
### What changes were proposed in this pull request?
This PR fixes Spark's behavior when
`spark.cleaner.referenceTracking.cleanCheckpoints` is set to `true`.
Previously, checkpoint data would be left around when Spark is shut down. With
this PR, Spark now cleans up checkpoint data as expected.
### Why are the changes needed?
Spark's behavior does not match the description of the
`spark.cleaner.referenceTracking.cleanCheckpoints` configuration.
> Controls whether to clean checkpoint files if the reference is out of
scope.
When Spark is shut down, all references go out of scope. These changes bring
Spark's behavior in line with the intended (and documented) behavior.
### Does this PR introduce _any_ user-facing change?
Yes. Previously, if you set `cleanCheckpoints` to `true`, checkpoint some
data, and then immediately shut down Spark, that checkpoint data will be left
intact. With this PR, that checkpoint data will now be cleaned up.
### How was this patch tested?
I tested this PR manually as follows:
```sh
# Cleanup checkpoint directory and start Spark.
trash ./checkpoint/
./bin/pyspark --conf 'spark.cleaner.referenceTracking.cleanCheckpoints=true'
```
```python
# Checkpoint some data and exit.
spark.sparkContext.setCheckpointDir('./checkpoint/')
spark.range(3).checkpoint()
import sys; sys.exit()
```
```sh
# Check that there is no checkpoint data in the checkpoint directory.
ll ./checkpoint/*
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
