Github user szhem commented on the issue:
https://github.com/apache/spark/pull/19410
I've tested the mentioned checkpointers with
`spark.cleaner.referenceTracking.cleanCheckpoints` set to `true` and without
explicit checkpoint files removal.
It seems that there are somewhere hard references remained - old checkpoint
files are not deleted at all and it seems that ContextCleaner.doCleanCheckpoint
is never called.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
