[
https://issues.apache.org/jira/browse/GOBBLIN-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hanghang Liu updated GOBBLIN-1692:
----------------------------------
Summary: Make GobblinHelixJobScheduler stop Helix workflow asynchronously
(was: Make GobblinHelixJobScheduler stop Helix workflow synchronously)
> Make GobblinHelixJobScheduler stop Helix workflow asynchronously
> ----------------------------------------------------------------
>
> Key: GOBBLIN-1692
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1692
> Project: Apache Gobblin
> Issue Type: Improvement
> Components: gobblin-cluster
> Reporter: Hanghang Liu
> Assignee: Hung Tran
> Priority: Major
>
> When handleUpdateJobConfigArrival, a new job config gets posted,
> GobblinHelixJobScheduler will firstly stop and delete the old job, and try to
> spin up the updated helix workflow.
> The job scheduler will try to do the stop synchronically with a default 10
> seconds timeout setting. However, this stop constantly running longer than
> the timeout for Helix, causing the job state not correctly updated as
> stopped. Thus, when construct the GobblinHelixJobLauncher, we will have the
> previous job in a wrong state as jobRunningMap is not updated yet, causing
> the new job won’t being launched. So we always see this log: {{{}Job {} will
> not be executed because other jobs are still running{}}}.
> We can make the job delete asynchronized, and let waitForJobCompletion method
> to ensure the job status get updated correctly eventually.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
