vinoyang commented on FLINK-7641:

Hi [~StephanEwen] , about this issue, we implemented a feature : In v1.3.2 and 
v1.4.0, for standalone cluster (HA mode) deployment environment, Job Manager 
can hot standby for running jobs.


More details : Different from Flink's default behavior ( jm leader election 
will trigger all jobs fail then recovery), in our inner version we implemented 
all jobs which under RUNNING state will still keep RUNNING (other non-RUNNING 
and non-TERMINAL jobs also follow the default processing way) when jm leader 
election and we have verified this feature in our production environment. In 
our opinion, if active jm failure, all RUNNING jobs fail and recover, the cost 
is very expensive, especially most RUNNING jobs communicate with jm just for 
checkpointing and updating accmulator snapshot, exection state and some other 
non-critical purpose.


Implementation details: We do snapshot for running jobs' execution graph (just 
some key information) and persist to Zookeeper. And the snapshot will be 
synchronized to all standby jms at runtime. If active jm is failure, the new jm 
leader can also index all running jobs with their execution graph snapshot. 
Moreover we also snapshoted slot resource information.


However, in an awkward position, Flink will squint towards flip-6, the job and 
jobmanager will be one-to-one in session(standalone) mode. So the active jm's 
failure cost will be less than old version. But for each RUNNING job, it still 
can reduce the cost of failing and recovery.


Do you think this function has value in Flink's future plan. Hope for listening 
your comment. Thanks.


> Loss of JobManager in HA mode should not cause jobs to fail
> -----------------------------------------------------------
>                 Key: FLINK-7641
>                 URL: https://issues.apache.org/jira/browse/FLINK-7641
>             Project: Flink
>          Issue Type: Improvement
>          Components: JobManager
>    Affects Versions: 1.3.2
>            Reporter: Elias Levy
>            Assignee: vinoyang
>            Priority: Major
> Currently if a standalone cluster of JobManagers is configured in 
> high-availability mode and the master JM is lost, the job executing in the 
> cluster will be restarted.  This is less than ideal.  It would be best if the 
> jobs could continue to execute without restarting while one of the spare JMs 
> becomes the new master, or in the worse case, the jobs are paused while the 
> JM election takes place.

This message was sent by Atlassian JIRA

Reply via email to