[
https://issues.apache.org/jira/browse/GOBBLIN-1380?focusedWorklogId=549925&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-549925
]
ASF GitHub Bot logged work on GOBBLIN-1380:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 08/Feb/21 21:58
Start Date: 08/Feb/21 21:58
Worklog Time Spent: 10m
Work Description: htran1 commented on a change in pull request #3220:
URL: https://github.com/apache/gobblin/pull/3220#discussion_r572402034
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
}
}
+ /**
+ * Thread that runs retention on failed dags based on their original start
time (from flow execution ID).
+ */
+ public static class FailedDagRetentionThread implements Runnable {
+ private final DagStateStore failedDagStateStore;
+ private final Map<String, Dag<JobExecutionPlan>> failedDags;
+ private final long failedDagRetentionTime;
+
+ FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String,
Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+ this.failedDagStateStore = failedDagStateStore;
+ this.failedDags = failedDags;
+ this.failedDagRetentionTime = failedDagRetentionTime;
+ }
+
+ @Override
+ public void run() {
+ try {
+ log.info("start clean");
+ for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter =
this.failedDags.entrySet().iterator(); iter.hasNext();) {
+ Map.Entry<String, Dag<JobExecutionPlan>> entry = iter.next();
+ if (this.failedDagRetentionTime > 0L
+ && System.currentTimeMillis() >
DagManagerUtils.getFlowExecId(entry.getValue()) + this.failedDagRetentionTime) {
Review comment:
Can retrieve the time once outside the loop so that everything is
cleaned up relative to the same time point.
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
}
}
+ /**
+ * Thread that runs retention on failed dags based on their original start
time (from flow execution ID).
+ */
+ public static class FailedDagRetentionThread implements Runnable {
+ private final DagStateStore failedDagStateStore;
+ private final Map<String, Dag<JobExecutionPlan>> failedDags;
+ private final long failedDagRetentionTime;
+
+ FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String,
Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+ this.failedDagStateStore = failedDagStateStore;
+ this.failedDags = failedDags;
+ this.failedDagRetentionTime = failedDagRetentionTime;
+ }
+
+ @Override
+ public void run() {
+ try {
+ log.info("start clean");
+ for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter =
this.failedDags.entrySet().iterator(); iter.hasNext();) {
Review comment:
Can use the following for loop construct.
for (Map.Entry<String, Dag<JobExecutionPlan>> entry :
this.failedDags.entrySet()) {
##########
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##########
@@ -1176,6 +1192,39 @@ private synchronized void cleanUpDag(String dagId) {
}
}
+ /**
+ * Thread that runs retention on failed dags based on their original start
time (from flow execution ID).
+ */
+ public static class FailedDagRetentionThread implements Runnable {
+ private final DagStateStore failedDagStateStore;
+ private final Map<String, Dag<JobExecutionPlan>> failedDags;
+ private final long failedDagRetentionTime;
+
+ FailedDagRetentionThread(DagStateStore failedDagStateStore, Map<String,
Dag<JobExecutionPlan>> failedDags, long failedDagRetentionTime) {
+ this.failedDagStateStore = failedDagStateStore;
+ this.failedDags = failedDags;
+ this.failedDagRetentionTime = failedDagRetentionTime;
+ }
+
+ @Override
+ public void run() {
+ try {
+ log.info("start clean");
+ for (Iterator<Map.Entry<String, Dag<JobExecutionPlan>>> iter =
this.failedDags.entrySet().iterator(); iter.hasNext();) {
+ Map.Entry<String, Dag<JobExecutionPlan>> entry = iter.next();
+ if (this.failedDagRetentionTime > 0L
+ && System.currentTimeMillis() >
DagManagerUtils.getFlowExecId(entry.getValue()) + this.failedDagRetentionTime) {
+ log.info("cleaning");
Review comment:
Will this flood the log? How about keeping a count and only printing one
line per execution of run()?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 549925)
Time Spent: 2.5h (was: 2h 20m)
> Add retention to failed dag state store
> ---------------------------------------
>
> Key: GOBBLIN-1380
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1380
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Jack Moseley
> Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
