+1 (binding) LinkedIn has been running an internal version of OrgQueue for almost 2 years. It is an essential feature for running YARN scheduler on clusters shared by multiple internal organizations.
I have participated in the YARN-5734 design and have regularly synced with Jonathan on development and implementation details. I believe the feature is of decent quality now and is ready to merge into trunk. On Mon, Oct 2, 2017 at 11:09 AM Jonathan Hung <jyhung2...@gmail.com> wrote: > Hi all, > > From discussion at [1], I'd like to start a vote to merge feature branch > YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending > Monday Oct 9 at 11:00AM PDT. > > This branch adds a framework to the scheduler to allow scheduler > configuration mutation on the fly, including a REST and CLI interface, and > an interface for the scheduler configuration backing store. Currently the > capacity scheduler implements this framework. > > Umbrella is here (YARN-5734 > <https://issues.apache.org/jira/browse/YARN-5734>), jenkins build is here > ( > YARN-7241 <https://issues.apache.org/jira/browse/YARN-7241>). All required > tasks for this feature are committed. Since this feature changes RM only, > we have tested this on a local RM setup with a suite of configuration > changes with no issue so far. > > Key points: > - The feature is turned off by default, and must be explicitly configured > to turn on. When turned off, the behavior reverts back to the original file > based mechanism for changing scheduler configuration (i.e. yarn rmadmin > -refreshQueues). > - The framework was designed in a way to be extendable to other schedulers > (most notably FairScheduler). > - A pluggable ACL policy (YARN-5949 > <https://issues.apache.org/jira/browse/YARN-5949>) allows admins > fine-grained control for who can change what configurations. > - The configuration storage backend is also pluggable. Currently an > in-memory, leveldb, and zookeeper implementation are supported. > > There were 15 subtasks completed for this feature. > > Huge thanks to everyone who helped with reviews, commits, guidance, and > technical discussion/design, including Carlo Curino, Xuan Gong, Subru > Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod > Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou. > > [1] > > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E > > Jonathan Hung > -- Zhe Zhang Apache Hadoop Committer http://zhe-thoughts.github.io/about/ | @oldcap
