ewoodbury commented on PR #3879: URL: https://github.com/apache/celeborn/pull/3879#issuecomment-6283329248
Pushed an update: For HDFS `cleanFile` now deletes first and only stops tracking the file once all of its files (data, index, .success, .sorted) are gone. Otherwise it logs a warning and throws, so `DestroyWorkerSlots` replies PARTIAL_SUCCESS and the driver retries. Shuffle expiry cleans it up if the retries also fail. One change from my earlier comment: the writer is still dropped up front since it can only delete once. Retries delete through the file info instead. I added tests for repeated failures, a leftover index file, a successful retry, and expiry during a failed delete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
