[
https://issues.apache.org/jira/browse/GOBBLIN-2152?focusedWorklogId=934316&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-934316
]
ASF GitHub Bot logged work on GOBBLIN-2152:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Sep/24 17:52
Start Date: 11/Sep/24 17:52
Worklog Time Spent: 10m
Work Description: Will-Lo merged PR #4047:
URL: https://github.com/apache/gobblin/pull/4047
Issue Time Tracking
-------------------
Worklog Id: (was: 934316)
Time Spent: 1h 20m (was: 1h 10m)
> Gobblin Temporal Jobs should be properly cleaning up writer staging and
> output dirs
> -----------------------------------------------------------------------------------
>
> Key: GOBBLIN-2152
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2152
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: William Lo
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Gobblin HDFS writers write to a staging directory, which renames to an output
> directory, before committing data to the final destination. This is so that
> commits are atomic and not partially done (unless configured).
> Gobblin needs to clean up these directories in order to manage HDFS space
> properly so that failed jobs do not maintain their files around. It is
> critical to ensure that deletion is only done to folders that are configured
> at a job level, so that any folders that may be shared between jobs or
> concurrently are not deleted mid-execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
