[
https://issues.apache.org/jira/browse/GOBBLIN-1837?focusedWorklogId=862845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-862845
]
ASF GitHub Bot logged work on GOBBLIN-1837:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 30/May/23 23:04
Start Date: 30/May/23 23:04
Worklog Time Spent: 10m
Work Description: umustafi commented on code in PR #3700:
URL: https://github.com/apache/gobblin/pull/3700#discussion_r1210912630
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java:
##########
@@ -310,6 +316,17 @@ public void orchestrate(Spec spec) throws Exception {
flowCompilationTimer.get().stop(flowMetadata);
}
+ // If multi-active scheduler is enabled do not pass onto DagManager,
otherwise scheduler forwards it directly
+ if (this.isMultiActiveSchedulerEnabled) {
+ String flowExecutionId =
flowMetadata.get(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD);
+ boolean leaseAttemptSucceeded =
schedulerLeaseAlgoHandler.handleNewTriggerEvent(jobProps, flowGroup, flowName,
+ flowExecutionId,
SchedulerLeaseDeterminationStore.FlowActionType.LAUNCH, triggerTimestampMillis);
+ _log.info("scheduler attempted lease on flowGroup: %s, flowName: %s,
flowExecutionId: %s, LAUNCH event for "
Review Comment:
This is a great suggestion. It'll also be useful to keep the prefix and
formatting constant across log events for the scheduler lease obtaining so we
can easily grep to understand what happened for each host.
`Multi-active scheduler lease attempt: [flowGroup: 'foo-grp', flowName:
'bar-fn', execId: 1234, dagAction: LAUNCH] - triggerEvent: 98765 -
SUCCESS/FAILURE `
Issue Time Tracking
-------------------
Worklog Id: (was: 862845)
Time Spent: 1h 40m (was: 1.5h)
> Implement multi-active, non blocking for leader host
> ----------------------------------------------------
>
> Key: GOBBLIN-1837
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1837
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-service
> Reporter: Urmi Mustafi
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This task will include the implementation of non-blocking, multi-active
> scheduler for each host. It will NOT include metric emission or unit tests
> for validation. That will be done in a separate follow-up ticket. The work in
> this ticket includes
> * define a table to do scheduler lease determination for each flow's trigger
> event and related methods to execute actions on this table
> * update DagActionStore schema and DagActionStoreMonitor to act upon new
> "LAUNCH" type events in addition to KILL/RESUME
> * update scheduler/orchestrator logic to apply the non-blocking algorithm
> when "multi-active scheduler mode" is enabled, otherwise submit events
> directly to the DagManager after receiving a scheduler trigger
> * implement the non-blocking algorithm, particularly handling reminder
> events if another host is in the process of securing the lease for a
> particular flow trigger
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
