[
https://issues.apache.org/jira/browse/GOBBLIN-2152?focusedWorklogId=934143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-934143
]
ASF GitHub Bot logged work on GOBBLIN-2152:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Sep/24 05:47
Start Date: 11/Sep/24 05:47
Worklog Time Spent: 10m
Work Description: abhishekmjain commented on code in PR #4047:
URL: https://github.com/apache/gobblin/pull/4047#discussion_r1753155396
##########
gobblin-temporal/src/test/java/org/apache/gobblin/temporal/ddm/activity/impl/GenerateWorkUnitsImplTest.java:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.gobblin.temporal.ddm.activity.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import org.apache.gobblin.source.workunit.BasicWorkUnitStream;
+import org.apache.gobblin.source.workunit.MultiWorkUnit;
+import org.apache.gobblin.source.workunit.WorkUnit;
+import org.apache.gobblin.source.workunit.WorkUnitStream;
+
+
+public class GenerateWorkUnitsImplTest {
+
+ @Test
+ public void testFetchesWorkDirsFromWorkUnits() {
+ List<WorkUnit> workUnits = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ WorkUnit workUnit = new WorkUnit();
+ workUnit.setProp("writer.staging.dir", "/tmp/jobId/task-staging/" + i);
+ workUnit.setProp("writer.output.dir", "/tmp/jobId/task-output/" + i);
+ workUnit.setProp("qualitychecker.row.err.file",
"/tmp/jobId/row-err/file");
+ workUnit.setProp("qualitychecker.clean.err.dir", "true");
+ workUnits.add(workUnit);
+ }
+ WorkUnitStream workUnitStream = new BasicWorkUnitStream.Builder(workUnits)
+ .setFiniteStream(true)
+ .build();
+ Set<String> output =
GenerateWorkUnitsImpl.calculateWorkDirsToCleanup(workUnitStream);
+ Assert.assertEquals(output.size(), 11);
+ }
+
+ @Test
+ public void testFetchesWorkDirsFromMultiWorkUnits() {
+ List<WorkUnit> workUnits = new ArrayList<>();
+ for (int i = 0; i < 5; i++) {
+ MultiWorkUnit multiWorkUnit = new MultiWorkUnit();
+ for (int j = 0; j < 3; j++) {
+ WorkUnit workUnit = new WorkUnit();
+ workUnit.setProp("writer.staging.dir", "/tmp/jobId/task-staging/");
+ workUnit.setProp("writer.output.dir", "/tmp/jobId/task-output/");
+ workUnit.setProp("qualitychecker.row.err.file",
"/tmp/jobId/row-err/file");
+ workUnit.setProp("qualitychecker.clean.err.dir", "true");
+ multiWorkUnit.addWorkUnit(workUnit);
+ }
+ workUnits.add(multiWorkUnit);
+ }
Review Comment:
Can we add a test where the staging and/or output dirs are different? The
logic for multiworkunit special handling is not getting tested well here
because of Set merging all duplicate paths into one.
Issue Time Tracking
-------------------
Worklog Id: (was: 934143)
Time Spent: 1h 10m (was: 1h)
> Gobblin Temporal Jobs should be properly cleaning up writer staging and
> output dirs
> -----------------------------------------------------------------------------------
>
> Key: GOBBLIN-2152
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2152
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: William Lo
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Gobblin HDFS writers write to a staging directory, which renames to an output
> directory, before committing data to the final destination. This is so that
> commits are atomic and not partially done (unless configured).
> Gobblin needs to clean up these directories in order to manage HDFS space
> properly so that failed jobs do not maintain their files around. It is
> critical to ensure that deletion is only done to folders that are configured
> at a job level, so that any folders that may be shared between jobs or
> concurrently are not deleted mid-execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
