HyukjinKwon commented on a change in pull request #35737:
URL: https://github.com/apache/spark/pull/35737#discussion_r820009353
##########
File path: python/pyspark/rdd.py
##########
@@ -465,6 +465,21 @@ def getCheckpointFile(self) -> Optional[str]:
return checkpointFile.get() if checkpointFile.isDefined() else None
+ def cleanShuffleDependencies(self, blocking: bool = False) -> None:
+ """
+ Removes an RDD's shuffles and it's non-persisted ancestors.
+
+ When running without a shuffle service, cleaning up shuffle files
enables downscaling.
+ If you use the RDD after this call, you should checkpoint and
materialize it first.
+
+ .. versionadded:: 3.3.0
+
+ Notes
Review comment:
Oh also we should add it in the list at
https://github.com/apache/spark/blob/master/python/docs/source/reference/pyspark.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
