Thanks for the clarification! I was able to test it out with Samza as well and get it working. I look forward to this patch https://issues.apache.org/jira/browse/YARN-4758 as it would simplify things a lot.
Appreciate your help. On Wed, Aug 12, 2020 at 6:51 AM [email protected] <[email protected]> wrote: > Bharath, > I just want to clarify a couple of things. The yarn distributed shell is > its own framework, and it does support preserving containers across AM > restart. I have tested this. But the MapReduce framework does not support > this feature (see https://issues.apache.org/jira/browse/MAPREDUCE-6608). > I spoke with Jon Eagles, and he believes that the Tez framework does > support the container-preserving feature. > > On Tuesday, August 11, 2020, 5:53:32 PM CDT, Bharath Kumara Subramanian < > [email protected]> wrote: > > Thanks Eric & Wilfred. > To give you some context, I work on Apache Samza and we have streaming as a > service offering on top of YARN. > Ideally, we would like to ensure in the event of AM restarts, the container > it spawned can still be taken over by the new AM. > > "Keep containers across application attempts" seems like an option that > might work for us. > Let me investigate and play with the parameter. > > Appreciate your quick response. > > Cheers, > Bharath > > > On Tue, Aug 11, 2020 at 6:31 AM Eric Payne <[email protected]> > wrote: > > > Bharath, > > > > while there is no concept of HA AM in YARN, some frameworks do support > > preserving containers across AM restarts. > > In the yarn distributed shell, for example, you can set the > > "-keep_containers_across_application_attempts" > > parameter. > > > > -Eric > > > > > > On Monday, August 10, 2020, 7:18:18 PM CDT, Bharath Kumara Subramanian < > > [email protected]> wrote: > > > > Hi, > > > > I am looking for more documentation/information on AM high availability. > I > > looked through the documentation and found resources on RM high > > availability but none for AM. > > > > I understand, YARN has provisions to restart AM in case of failure up to > a > > configured number of attempts. However, I wanted to know if YARN has an > > active/standby option for AM. I would like to avoid bringing up the > > processing containers again in the event of AM failure and have the > standby > > AM take over the managing my application. > > > > Thanks, > > Bharath > > >
