[
https://issues.apache.org/jira/browse/GOBBLIN-1837?focusedWorklogId=863162&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-863162
]
ASF GitHub Bot logged work on GOBBLIN-1837:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 01/Jun/23 00:32
Start Date: 01/Jun/23 00:32
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3700:
URL: https://github.com/apache/gobblin/pull/3700#discussion_r1212444910
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/DagActionStore.java:
##########
@@ -27,7 +27,9 @@
public interface DagActionStore {
enum DagActionValue {
KILL,
- RESUME
+ RESUME,
+ // TODO: potentially combine this enum with {@link
SchedulerLeaseDeterminationStore.FlowActionType}
+ LAUNCH
Review Comment:
Yes, that is correct. The host that wins the lease is only the one who
passes that job along to the DagManager to launch (they do this by persisting
in mysql). In the near future, only the leader host will actually run the job
but eventually there should be a non-blocking, multi-active DagManager as well
that will work similarly to win the lease of running the job.
Issue Time Tracking
-------------------
Worklog Id: (was: 863162)
Time Spent: 6h 10m (was: 6h)
> Implement multi-active, non blocking for leader host
> ----------------------------------------------------
>
> Key: GOBBLIN-1837
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1837
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> This task will include the implementation of non-blocking, multi-active
> scheduler for each host. It will NOT include metric emission or unit tests
> for validation. That will be done in a separate follow-up ticket. The work in
> this ticket includes
> * define a table to do scheduler lease determination for each flow's trigger
> event and related methods to execute actions on this table
> * update DagActionStore schema and DagActionStoreMonitor to act upon new
> "LAUNCH" type events in addition to KILL/RESUME
> * update scheduler/orchestrator logic to apply the non-blocking algorithm
> when "multi-active scheduler mode" is enabled, otherwise submit events
> directly to the DagManager after receiving a scheduler trigger
> * implement the non-blocking algorithm, particularly handling reminder
> events if another host is in the process of securing the lease for a
> particular flow trigger
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
