[
https://issues.apache.org/jira/browse/GOBBLIN-1970?focusedWorklogId=893939&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-893939
]
ASF GitHub Bot logged work on GOBBLIN-1970:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 05/Dec/23 02:14
Start Date: 05/Dec/23 02:14
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3841:
URL: https://github.com/apache/gobblin/pull/3841#discussion_r1414758146
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/DagActionStoreChangeMonitor.java:
##########
@@ -107,6 +115,37 @@ protected void assignTopicPartitions() {
return;
}
+ /**
+ * Load all actions from the DagActionStore to process any missed actions
during service startup
+ */
+ private void initializeMonitor() {
+ // TODO: exclude the dag actions that have a lease in the arbiter store if
this is processing too many events
Review Comment:
I'll remove this for now unless we see other problems but basically the idea
was I can do a join of dag actions with completed leases in the lease arbiter
table to exclude actions that have already been completed but it's only an
interim soln if this doesn't work and kind of hacky
Issue Time Tracking
-------------------
Worklog Id: (was: 893939)
Time Spent: 1h 10m (was: 1h)
> Consolidate Processing Dag Actions to Single Code Path
> ------------------------------------------------------
>
> Key: GOBBLIN-1970
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1970
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We have similar code in the DagActionStoreChangeMonitor and DagManager to
> process dag actions from the DagActionStore. There have been small
> discrepancies between the code in each area leading to unexpected bugs in how
> the actions are processed, so to fix forward and make this easier to maintain
> with code in one place we consolidate all logic relating to the dag action
> processing to the DagActionStoreChangeMonitor. The processing in the change
> monitor is working correctly, while the handling of launch events in the
> DagManager is failing compilation most likely due to an error loading the job
> templates when initializing the compiler but we're not able to identify the
> exact issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
