Hey Zhijie, Thanks, that's what I suspected. We are using the AMRMClient(Async), so if you're using YARN 2.4.0 with HA RM, Samza should just work.
Regarding work-preserving AM, here's the ticket that's tracking this feature in Samza: https://issues.apache.org/jira/browse/SAMZA-133 Cheers, Chris On 9/11/14 3:51 PM, "Zhijie Shen" <[email protected]> wrote: >W.R.T HA RM, AM doesn't need to make any change if it is using >AMRMClient(Async), which will automatically take care of the work required >when RM failover happens. > >One thing that may interest Samza is the work-preserving AM restarting. If >Samza AM crash for some reason, when it restarts, it can try to get the >old >containers back, to continue the work. To achieve this, Samza needs some >logic optimization on the AM side. > >On Thu, Sep 11, 2014 at 2:45 PM, Chris Riccomini < >[email protected]> wrote: > >> Hey Ethan, >> >> Yes, we plan to support this, but haven't done any testing with it yet. >> >> The main question that I have is whether supporting HA RM requires >>Samza's >> AM to have code changes, or whether the YARN AM client will >>transparently >> handle RM failover. Until we run some tests on this (or are told >> otherwise), I just don't know (and haven't had time to investigate). >>Does >> anyone know the answer to this question? >> >> Cheers, >> Chris >> >> On 9/11/14 1:55 PM, "Ethan Setnik" <[email protected]> wrote: >> >> >Yarn 2.4.0 brings support for high availability configurations by >> >specifying a cluster of resource managers and a state store via >>zookeeper. >> > >> > >> >"When there are multiple RMs, the configuration (yarn-site.xml) used by >> >clients and nodes is expected to list all the RMs. Clients, >> >ApplicationMasters (AMs) and NodeManagers (NMs) try connecting to the >>RMs >> >in a round-robin fashion until they hit the Active RM. If the Active >>goes >> >down, they resume the round-robin polling until they hit the "new" >> >Active.² >> > >> > >> >Does Samza have any plans to support Yarn HA configurations? >> >> > > >-- >Zhijie Shen >Hortonworks Inc. >http://hortonworks.com/ > >-- >CONFIDENTIALITY NOTICE >NOTICE: This message is intended for the use of the individual or entity >to >which it is addressed and may contain information that is confidential, >privileged and exempt from disclosure under applicable law. If the reader >of this message is not the intended recipient, you are hereby notified >that >any printing, copying, dissemination, distribution, disclosure or >forwarding of this communication is strictly prohibited. If you have >received this communication in error, please contact the sender >immediately >and delete it from your system. Thank You.
