Hi Wangda, Can this feature be disabled? Is it on or off by default? We're 1 month from the target release for beta1, so I don't want to introduce risk to existing code paths. TSv2 and S3Guard and YARN Federation are all okay in that regard.
I'm also not clear on what work is remaining, there are a lot of unresolved subtasks still under YARN-3926. In terms of compatibility, the design doc talks about some PB changes. Are these stable? Is there documentation somewhere that explains what APIs are stable or unstable for users? I'm going to start a separate discussion about beta1 scope. I think this is the fourth merge proposal this week, and this amount of code movement makes me very nervous considering that beta1 is only a month out. Best, Andrew On Thu, Aug 17, 2017 at 8:27 PM, Wangda Tan <wheele...@gmail.com> wrote: > +hdfs/common/mr > > On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan <wheele...@gmail.com> wrote: > > > Hi all, > > > > I want to hear your thoughts of merging YARN resource profile branch into > > trunk in the next few weeks. The goal is to get it in for Hadoop 3.0 > beta1. > > > > *Regarding to testing:* > > We did extensive tests for the feature in the last several months. > > Comparing to latest trunk. > > - For SLS benchmark: We didn't see observable performance gap from > > simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+ > > containers allocated per second. > > - For microbenchmark: We use performance test cases added by YARN-6775, > it > > shows around 5% performance regression comparing to trunk. > > > > *Regarding to API stability: * > > Most new added @Public APIs are @Unstable (We're going to convert some > new > > added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we > want > > to get this included by beta1 so we get some feedbacks before declaring > > stable API. > > > > There're few pending cleanups under YARN-3926 umbrella JIRA. Besides > these > > cleanups, this feature works from end-to-end, we will do another > iteration > > of end-to-end tests after cleanup patches got committed. > > > > We would love to get your thoughts before opening a voting thread. > > > > Special thanks to a team of folks who worked hard and contributed towards > > this efforts including design discussion / patch / reviews, etc.: Varun > > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu, > > Karthik Kambatla, Jason Lowe, Arun Suresh. > > > > Thanks, > > Wangda Tan > > >
