[
https://issues.apache.org/jira/browse/GOBBLIN-1380?focusedWorklogId=549930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549930
]
ASF GitHub Bot logged work on GOBBLIN-1380:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/21 22:03
Start Date: 08/Feb/21 22:03
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#3220:
URL: https://github.com/apache/gobblin/pull/3220#discussion_r572406510
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
}
}
+ /**
+ * Thread that runs retention on failed dags based on their original start
time (from flow execution ID).
+ */
+ public static class FailedDagRetentionThread implements Runnable {
+ private final DagStateStore failedDagStateStore;
+ private final Map<String, Dag<JobExecutionPlan>> failedDags;
+ private final long failedDagRetentionTime;
+
+ FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String,
Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+ this.failedDagStateStore = failedDagStateStore;
+ this.failedDags = failedDags;
+ this.failedDagRetentionTime = failedDagRetentionTime;
+ }
+
+ @Override
+ public void run() {
+ try {
+ log.info("start clean");
+ for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter =
this.failedDags.entrySet().iterator(); iter.hasNext();) {
Review comment:
I think it is unsafe to remove elements from a collection that you are
iterating right? In the for loop I am removing elements from `failedDags`.
I did it this way as per
https://stackoverflow.com/questions/223918/iterating-through-a-collection-avoiding-concurrentmodificationexception-when-re,
although I realize now I still did it incorrectly, I should be calling
`iter.remove()` in the loop
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 549930)
Time Spent: 2h 50m (was: 2h 40m)
> Add retention to failed dag state store
> ---------------------------------------
>
> Key: GOBBLIN-1380
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1380
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
