Yida Wu created IMPALA-13677:
--------------------------------
Summary: Cleanup of s3 scratch files on abnormal executor exit
Key: IMPALA-13677
URL: https://issues.apache.org/jira/browse/IMPALA-13677
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Yida Wu
Assignee: Yida Wu
Currently, when an executor spills data to remote storage, scratch files would
remain in the storage if the executor exits abnormally or is terminated after
the graceful shutdown deadline.
Immediate removal might be challenging, but we could consider adding an extra
thread in the coordinator to handle cleanup of leftover scratch files in remote
storage. Since s3 is the most common scenario, this task may specifically focus
on handling the leftover scratch files in s3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
