[
https://issues.apache.org/jira/browse/IMPALA-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yida Wu updated IMPALA-13677:
-----------------------------
Description:
Currently, when an executor spills data to a remote storage, scratch files
would remain in the remote storage if the executor exits abnormally or is
terminated after the graceful shutdown deadline.
Immediate removal might be challenging, but we could consider adding an extra
thread in the coordinator to handle cleanup of leftover scratch files in remote
storage. Since s3 is the most common scenario, this task may specifically focus
on handling the leftover scratch files in s3.
was:
Currently, when an executor spills data to remote storage, scratch files would
remain in the storage if the executor exits abnormally or is terminated after
the graceful shutdown deadline.
Immediate removal might be challenging, but we could consider adding an extra
thread in the coordinator to handle cleanup of leftover scratch files in remote
storage. Since s3 is the most common scenario, this task may specifically focus
on handling the leftover scratch files in s3.
> Cleanup of s3 scratch files on abnormal executor exit
> -----------------------------------------------------
>
> Key: IMPALA-13677
> URL: https://issues.apache.org/jira/browse/IMPALA-13677
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Critical
>
> Currently, when an executor spills data to a remote storage, scratch files
> would remain in the remote storage if the executor exits abnormally or is
> terminated after the graceful shutdown deadline.
> Immediate removal might be challenging, but we could consider adding an extra
> thread in the coordinator to handle cleanup of leftover scratch files in
> remote storage. Since s3 is the most common scenario, this task may
> specifically focus on handling the leftover scratch files in s3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
