Hi Subru, Thanks for starting the discussion.
We (LinkedIn) have an immediate need for resource types and native GPU support. Given we are running 2.7 on our main clusters, we decided to avoid deploying hadoop 3.x on our machine learning clusters (and having to support two very different hadoop versions). Since for us there is considerable risk and work involved in upgrading to hadoop 3, I think having a branch-2.10 bridge release for porting important hadoop 3 features to branch-2 is a good idea. Thanks, Jonathan Hung On Mon, Feb 26, 2018 at 2:37 PM, Subru Krishnan <su...@apache.org> wrote: > Folks, > > We (i.e. Microsoft) have started stabilization of 2.9 for our production > deployment. During planning, we realized that we need to backport 3.x > features to support GPUs (and more resource types like network IO) natively > as part of the upgrade. We'd like to share that work with the community. > > Instead of stabilizing the base release and cherry-picking fixes back to > Apache, we want to work publicly and push fixes directly into > trunk/.../branch-2 for a stable 2.10.0 release. Our goal is to create a > bridge release for our production clusters to the 3.x series and to address > scalability problems in large clusters (N*10k nodes). As we find issues, we > will file JIRAs and track resolution of significant regressions/faults in > wiki. Moreover, LinkedIn also has committed plans for a production > deployment of the same branch. We welcome broad participation, particularly > since we'll be stabilizing relatively new features. > > The exact list of features we would like to backport in YARN are: > > - Support for Resource types [1][2] > - Native support for GPUs[3] > - Absolute Resource configuration in CapacityScheduler [4] > > > With regards to HDFS, we are currently looking at mainly fixes to Router > based Federation and Windows specific fixes which should anyways flow > normally. > > Thoughts? > > Thanks, > Subru/Arun > > [1] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27786.html > [2] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28281.html > [3] https://issues.apache.org/jira/browse/YARN-6223 > [4] https://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg28772.html >
