ulysses-you commented on a change in pull request #815: URL: https://github.com/apache/incubator-kyuubi/pull/815#discussion_r670942229
########## File path: docs/tools/spark_block_cleaner.md ########## @@ -0,0 +1,97 @@ +<div align=center> + + + +</div> + +# Kubernetes tools spark-block-cleaner + +## Requirements +You'd better have cognition upon the following things when you want to use spark-block-cleaner. + +* Read this article +* An active Kubernetes cluster +* [Kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) + +## Purpose + +When running Spark On Kubernetes, we encountered such a situation that after using hostPath volume local-dir, the usage rate of the directory storing shuffle files remained high. + +So an additional tool is needed to clear the accumulated block files. + +## Usage + +### Build Block Cleaner Docker Image +```shell + docker build ./tools/spark-block-cleaner/kubernetes/docker +``` + +### Modify spark-block-cleaner.yml +You need to modify the spark-block-cleaner.yml to fit your current environment. Review comment: `spark-block-cleaner.yml` -> `**spark-block-cleaner.yml**` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
