[
https://issues.apache.org/jira/browse/CRUNCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458870#comment-16458870
]
Josh Wills commented on CRUNCH-669:
-----------------------------------
Added the patch for this, which simply adds a config option to disable the tmp
dir delete inside of finalize().
> Add an option to preserve Crunch temp directories
> -------------------------------------------------
>
> Key: CRUNCH-669
> URL: https://issues.apache.org/jira/browse/CRUNCH-669
> Project: Crunch
> Issue Type: Improvement
> Reporter: Josh Wills
> Priority: Major
> Attachments: CRUNCH-669.patch
>
>
> I have a problem where a Crunch client can potentially get killed through no
> fault of its own (e.g., an Airflow task failing a heartbeat check), which
> will kill the client, but leave the MR job itself still running. The problem
> is that the finalize() operation can delete the temp directory that contains
> the config info for Crunch, which will make any subsequent tasks fail to run
> to completion b/c their config info has been removed, or because some temp
> output in the directory that would have been useful can be deleted. I'd like
> to add an option to override the automatic delete of the temp directory
> during the finalize() operation for these situations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
