[
https://issues.apache.org/jira/browse/GOBBLIN-1380?focusedWorklogId=549928&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549928
]
ASF GitHub Bot logged work on GOBBLIN-1380:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/21 22:02
Start Date: 08/Feb/21 22:02
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#3220:
URL: https://github.com/apache/gobblin/pull/3220#discussion_r572405809
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
}
}
+ /**
+ * Thread that runs retention on failed dags based on their original start
time (from flow execution ID).
+ */
+ public static class FailedDagRetentionThread implements Runnable {
+ private final DagStateStore failedDagStateStore;
+ private final Map<String, Dag<JobExecutionPlan>> failedDags;
+ private final long failedDagRetentionTime;
+
+ FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String,
Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+ this.failedDagStateStore = failedDagStateStore;
+ this.failedDags = failedDags;
+ this.failedDagRetentionTime = failedDagRetentionTime;
+ }
+
+ @Override
+ public void run() {
+ try {
+ log.info("start clean");
+ for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter =
this.failedDags.entrySet().iterator(); iter.hasNext();) {
+ Map.Entry<String, Dag<JobExecutionPlan>> entry = iter.next();
+ if (this.failedDagRetentionTime > 0L
+ && System.currentTimeMillis() >
DagManagerUtils.getFlowExecId(entry.getValue()) + this.failedDagRetentionTime) {
+ log.info("cleaning");
Review comment:
Whoops, I added those logs for testing and meant to remove them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 549928)
Time Spent: 2h 40m (was: 2.5h)
> Add retention to failed dag state store
> ---------------------------------------
>
> Key: GOBBLIN-1380
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1380
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
