Karthik Kambatla commented on YARN-4438:

bq. And because ZKRMStateStore is currently in active service, it cannot be 
simply moved to AlwaysOn service. So, I'd like to do it separately to minimize 
the core change in this jira.
Fine with separate JIRA. Not sure I understand why ZKRMStateStore needs to be 
an AlwaysOn service. 

bq. I'd like to change this part for RM to not refresh the configs if shared 
storage based config provider is not enabled.
I was never a fan of the shared-storage-configuration stuff. Now that we have 
it, don't think we can get rid of it until Hadoop 4. How would this change 
look? The RM has an instance of the elector; every time we transition to 
active, will either the RM or the elector check if 
shared-storage-config-provider is enabled and call refresh? 

But yeah, I do see the point of calling these methods directly from RM. 

bq. To avoid a busy loop and rejoining immediately. 
If we rejoin immediately, one of the RMs would become Active. It is not like 
the RM is going to use the cycles for anything else if we sleep. Is the concern 
that Curator may be biased in picking an RM in certain conditions? 

bq. What do you mean by force give-up ? exit RM ?
If leaderLatch.close() throws an exception, when does Curator realize the RM is 
not participating in the election anymore? If not, it might keep electing the 
same RM active? How do we handle this, and how long of a wait is okay? 

bq. Even though RM remains at standby, all services should be already shutdown, 
so there's no harm to the end users ?
Agree, there is no harm. My concern is about availability - having one of the 
RMs active "most" of the time. 

bq. I have one question about ActiveStandbyCheckThread. if we make zkStateStore 
and elector to share the same zkClient, do we still need the 
ActiveStandbyCheckThread ? the elector itself should get notification when the 
connection is lost.

Are you referring to the VerifyActiveStatusThread? The RM loses leadership; the 
connection can be restored even if it loses. We could actively go stop the 
store if it hasn't already stopped. The store would have already gotten fenced, 
so we don't run the risk of corrupting the store. So, you are right, we might 
not need that thread.

bq. This is currently what EmbeddedElectorService is doing. If the leadership 
is already lost from zk's perspective, the other RM should take up the 
You are right, it isn't a big deal. Just realized EmbeddedElectorService does 
the same today. Haven't seen Curator's LeaderLatch code. What happens if this 
RM is subsequently elected leader? Does the transition to Active succeed just 
fine? Or, is it possible it gets stuck in a way it can't transition to active? 
If it gets into such a situation, we should consider crashing it altogether. 

bq. I think leaderLatch could never be null ?
Seeing all the NPEs we have in RM/Scheduler, I would like for us to err on the 
side of caution and do null-checks. If not, we at least need to make it 
consistent everywhere. 

bq. Why does it need to be called outside of if (state == 
HAServiceProtocol.HAServiceState.ACTIVE) ? This is a fresh start, it does not 
need to call reinitiialize.
You are right. Sorry for the noise, clearly it has been a while since I looked 
at this code. 

> Implement RM leader election with curator
> -----------------------------------------
>                 Key: YARN-4438
>                 URL: https://issues.apache.org/jira/browse/YARN-4438
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-4438.1.patch, YARN-4438.2.patch, YARN-4438.3.patch
> This is to implement the leader election with curator instead of the 
> ActiveStandbyElector from common package,  this also avoids adding more 
> configs in common to suit RM's own needs. 

This message was sent by Atlassian JIRA

Reply via email to