Github user MrMathias commented on the issue:
https://github.com/apache/spark/pull/20759
If anyone are interested; checkpointing is only really documented in the
scope of Spark streaming, however checkpointing it is extensively used in the
ml package.
I think the best solution is to add warnings or even errors to the
procedures that uses checkpointing, when a checkpoint directory is not set.
Silently ignoring checkpointInterval on heavily iterative procedures that
easily causes memory errors, is not a good default behavior.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
