[
https://issues.apache.org/jira/browse/AURORA-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renan DelValle updated AURORA-1944:
-----------------------------------
Description:
Using Apache Curator as the Zookeeper library causes an issue where Aurora is
unable to elect a leader if Zookeeper loses quorum for an extended period of
time.
Scheduler seems to crash around:
{{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave
leadership: org.apache.aurora.common.zookeeper.SingletonService$LeaveException:
Failed to abdicate leadership of group at /aurora/scheduler}}
When the init system brings the scheduler back up, it is unable to elect a
leader if ZK is still down.
Specifically, the redirect monitor fails:
{{E0802 14:09:37.063 [RedirectMonitor STARTING,
GuavaUtils$LifecycleShutdownListener] Service: RedirectMonitor [FAILED] failed
unexpectedly. Triggering shutdown.}}
Leading to every scheduler showing the following:
{{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor
in host set, will not redirect despite not being leader.}}
Once the scheduler enters this state, it is unable to snap out of it until it
is manually restarted.
was:
Using Apache Curator as the Zookeeper library causes an issue where Aurora is
unable to elect a leader if Zookeeper loses quorum for an extended period of
time.
Scheduler seems to crash around:
{{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to leave
leadership: org.apache.aurora.common.zookeeper.SingletonService$LeaveException:
Failed to abdicate leadership of group at /aurora/scheduler }}
When the init system brings the scheduler back up, it is unable to elect a
leader if ZK is still down.
Once the scheduler enters this state, it is unable to snap out of it until it
is manually restarted.
> Aurora is unable to elect leader after losing ZK for an extended period of
> time
> -------------------------------------------------------------------------------
>
> Key: AURORA-1944
> URL: https://issues.apache.org/jira/browse/AURORA-1944
> Project: Aurora
> Issue Type: Bug
> Components: Scheduler
> Environment: Running on 0.17.0
> Reporter: Renan DelValle
> Attachments: aurora-0.log, aurora-1.log, aurora-2.log
>
>
> Using Apache Curator as the Zookeeper library causes an issue where Aurora is
> unable to elect a leader if Zookeeper loses quorum for an extended period of
> time.
> Scheduler seems to crash around:
> {{W0802 14:01:14.436 [TaskEventBatchWorker, SchedulerLifecycle] Failed to
> leave leadership:
> org.apache.aurora.common.zookeeper.SingletonService$LeaveException: Failed to
> abdicate leadership of group at /aurora/scheduler}}
> When the init system brings the scheduler back up, it is unable to elect a
> leader if ZK is still down.
> Specifically, the redirect monitor fails:
> {{E0802 14:09:37.063 [RedirectMonitor STARTING,
> GuavaUtils$LifecycleShutdownListener] Service: RedirectMonitor [FAILED]
> failed unexpectedly. Triggering shutdown.}}
> Leading to every scheduler showing the following:
> {{W0802 14:16:34.646 [qtp576711849-43, LeaderRedirect] No serviceGroupMonitor
> in host set, will not redirect despite not being leader.}}
> Once the scheduler enters this state, it is unable to snap out of it until it
> is manually restarted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
