Prasheel3001 opened a new issue #4771: URL: https://github.com/apache/hudi/issues/4771
**_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at [email protected]. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** When yarn kills the containers abruptly for any reason while hudi stage is in progress then the spill path created by hudi on the disk is not cleaned and as a result of which the nodes on the cluster start running out of space. We need to clear the spill path manually to free out disk space. **To Reproduce** Steps to reproduce the behavior: Problem would be reproducible if hudi stage fails with containers getting killed abruptly. **Expected behavior** Hudi should clear the spill path even if the containers are killed abruptly by yarn for any reason. **Environment Description** * Hudi version : 0.7.0 * Spark version : 2.4.6 * Hive version : * Hadoop version : * Storage (HDFS/S3/GCS..) : HDFS * Running on Docker? (yes/no) : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
