[
https://issues.apache.org/jira/browse/TEZ-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe reassigned TEZ-3877:
-------------------------------
Assignee: Jason Lowe
Summary: Delete unordered spill files once merge is done (was: Delete
spill files once merge is done)
Offline Rohini pointed me to the UnorderedKVWriter, and indeed the intermediate
spill files are *not* being deleted after being merged like they are for the
ordered case. Updated the JIRA summary accordingly.
> Delete unordered spill files once merge is done
> -----------------------------------------------
>
> Key: TEZ-3877
> URL: https://issues.apache.org/jira/browse/TEZ-3877
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rohini Palaniswamy
> Assignee: Jason Lowe
>
> I see that spill files are not deleted right after merge completes. We
> should do that as it takes up a lot of space and we can't afford that wastage
> when Tez takes up a lot of shuffle space with complex DAGs. [~jlowe] told me
> they are only cleaned up after application completes as they are written in
> app directory and not container directory. That also has to be done so that
> they are cleaned up by node manager during task failures or container crashes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
