+1
On Mon, Jan 13, 2020 at 1:47 PM Arpit Agarwal <aagar...@cloudera.com.invalid> wrote: > +1 > (binding) > > > On Jan 13, 2020, at 9:44 AM, Bertrand Delacretaz < > bdelacre...@codeconsult.ch> wrote: > > > > Hi, > > > > On Fri, Jan 10, 2020 at 6:47 PM Vinod Kumar Vavilapalli > > <vino...@apache.org> wrote: > >> I'd like to call a vote on accepting YuniKorn into the Apache > Incubator... > > > > +1 > > > > I'm copying the proposal text below, we usually do that to get > > complete mail archives. > > > > -Bertrand > > > > > > YuniKorn proposal > > > > Abstract > > > > YuniKorn is a standalone resource scheduler responsible for scheduling > > batch jobs and long-running services on large scale distributed > > systems running in on-premises environments as well as different > > public clouds. > > Proposal > > > > YuniKorn ['ju:nikɔ:n] is a unified resource scheduler aiming to > > achieve fine-grained resource sharing for various workloads > > efficiently on a large scale, multi-tenant and cloud-native > > environments. YuniKorn brings a unified, cross-platform scheduling > > experience for mixed workloads, with support for but not limited to, > > Apache™ Hadoop® YARN and Kubernetes. YuniKorn is a made-up word > > (credit to Vinod Kumar Vavilapalli) - it’s made up of Y for Apache™ > > Hadoop® YARN, K for K8s, Uni for Unified, and its pronunciation is the > > same as “Unicorn” > > > > Currently, YuniKorn is an open-source project with Apache 2.0 license. > > The source code is hosted as a git-repo under github.com/cloudera > > domain. We would like to share it with the ASF and expand the > > community to a wider range of users and contributors. > > Background > > > > Enterprise users run their workloads on different platforms such as > > Apache™ Hadoop® YARN and Kubernetes. They need to work with different > > resource schedulers in order to plan their workloads to run on these > > platforms efficiently. The scheduler implementations are fragmented, > > and not optimized to balance existing use-cases like batch workloads > > along with new needs such as cloud-native architecture, autoscaling, > > etc. We need a single resource planning/management framework to manage > > resources on different platforms using the same semantics, in order to > > address all the important resource management requirements. > > Rationale > > > > There is no solution that exists now to address the needs of having a > > unified resource scheduling experiences across platforms. That makes > > it difficult to manage workloads running on different environments, > > from on-premise to Cloud. YuniKorn aims to satisfy these needs. > > YuniKorn is designed around the following principles: > > > > 1) Support different environments > > > > As the compute platforms are evolving quickly, there are more and more > > challenges appears in on-prem, cloud or hybrid environments. YuniKorn > > aims to bring unified scheduling experiences across multiple > > environments with enhanced scheduling capabilities. > > > > 2) Support extensive type of workloads > > > > To improve the efficiency of the computing platform, a key idea is to > > run different types of applications, like long-running services and > > batch jobs, on shared resources. YuniKorn is an effort to address all > > the scheduling features needed for such mixed workload environments. > > > > 3) Benefit both big-data and cloud-native communities > > > > A resource scheduler needs to be capable of supporting mixed > > workloads, both batch and long-running services. This is the key to > > improving cluster utilization, and to reduce the complexity of > > dev-ops. By creating a common scheduler that is decoupled from the > > container platforms underneath, it can benefit both Apache™ Hadoop® > > YARN and the Kubernetes communities. > > Initial Goals > > > > Initial goals are: > > > > Move the existing codebase, documentation to Apache hosted repo > > Setup mailing lists, web-site, CI/CD pipeline under Apache > infrastructure > > Setup JIRA for issue tracking > > Incremental development and releases according to Apache guidelines > > Expand the community and bring more diversified contributors/users > > to the community > > > > Current Status > > Meritocracy > > > > Many of the initial developers of YuniKorn are already Apache > > committers and PMC members from other Apache projects, such as Apache > > Hadoop and Apache Submarine. Many of us have worked in the Apache > > Hadoop community for years and know the Apache way well. We believe > > strongly in meritocracy in electing committers and PMC members. We > > believe that contributions can come in forms other than just code: for > > example, one of our initial proposed committers has contributed solely > > in the area of project documentation. We will encourage contributions > > and participation of all types, and ensure that contributors are > > appropriately recognized. > > Community > > > > YuniKorn is a relatively new open source project, Cloudera is the > > original development sponsor for YuniKorn. From the beginning of the > > project itself, we had clearly aimed to have this as an open source > > project, so we started to build the community from the very early > > stages. We received a lot of feedback and valuable suggestions from > > other community members while the project was hosted as an open source > > project on github. This feedback has greatly influenced some of our > > designs. For e.g, developers from Alibaba had been involved in the > > very early stage of development, lots of effort related to > > performance/throughput enhancement were contributed by them. Lots of > > other organizations further showed their interest to join the > > community once we started talking about it in meetups, conferences > > etc. > > Core developers > > > > The project was initiated in Cloudera and so the core developers are > > heavily from this organization. Tao Yang from Alibaba joined the > > development at a very early stage. The core developers of YuniKorn are > > (listed in alphabetical order): > > > > Akhil PB (Cloudera) > > Sunil Govindan (Cloudera) > > Tao Yang (Alibaba) > > Vinod Vavilapalli (Cloudera) > > Wangda Tan (Cloudera) > > Weiwei Yang (Cloudera) > > Wilfred Spiegelenburg (Cloudera) > > > > > > Given the origin history, the core development team so far has not > > been very diverse, but we’ve been attempting to grow that diversity. > > We have every hope to continue building a diverse and sustainable > > community if the project gets accepted into Apache. > > Alignment > > > > The motivation of YuniKorn project is to resolve common resource > > scheduling problems for various workloads, on large scale distributed > > systems. Apache is home to one of these systems in the form of Apache > > Hadoop YARN. Many of thee workloads that we expect to leverage > > YuniKorn are computing engines like Apache Spark, Apache Flink whether > > they run on top of YARN or on Kubernetes. > > Known Risks > > Project Name > > > > We have done a search of the name "YuniKorn" on Github, and at the > > time of the search we found nothing related to resource scheduler or > > distributed system. We also did a search of the name YuniKorn as a > > trademark and there seem to be none. A generic web search also didn't > > return any relevant projects. Since the name seems to be unique, easy > > to remember, pronounce, and relevant to the project, we believe it is > > a suitable name even at the ASF. > > > > Cloudera does NOT have a trademark on the name YuniKorn, so there is > > no trademark assignment needed. Cloudera will commit to using Apache > > YuniKorn as the project name when/if it graduates and becomes an > > Apache project. > > Orphaned products > > > > The core developers of YuniKorn project from different companies plan > > to work full time on this project. Currently, the initial team intends > > to continue the investments on the YuniKorn project, it will be > > integrated into the solutions to the customers. Several other > > organizations (like Alibaba) have also started to evaluate the > > project, and plan to adopt it in their production environments. We > > anticipate the adoption will be further improved once it becomes an > > Apache project. > > > > We have also got support from core-platform developers and Apache > > committers who are interested in contributing to YuniKorn project from > > different companies like Microsoft, Nvidia, Tencent, etc. We’re > > expecting to see more contributions from these committers and usage by > > their internal platforms. So overall, the risk of YuniKorn being an > > orphaned project is low. > > Inexperience with Open source > > > > Most of the core developers in YuniKorn project are experienced open > > source veterans, several developers are Apache committers and PMC > > members of other projects, such as Apache™ Hadoop®. And the > > development style is already very likely the Apache way > > > > We have open community meetings to discuss designs, problems and > roadmaps > > We publish all patches and issue related discussions on github > > We enforce the code review and log all comments in github issues > > > > Length of Incubation > > > > We started the work 10 months ago, so far the groundwork for YuniKorn > > is done and the initial version can work with K8s seamlessly. Based on > > the initial contributers’ experience in ASF projects, we don’t expect > > that there will be huge gaps before YuniKorn can graduate with > > regarding to ASF’s policies on software and releases. The goal is to > > grow the community quickly and increase the user base within a few > > months while making releases that adhere to the ASF standards. When it > > reaches a reasonable size of adoption, and a strong community with a > > good number of committers/PMC members, we can prompt the graduation. > > We expect the length of incubation to be approximately 12 to 18 > > months. > > Homogenous Development > > > > The initial proposed list of committers and contributors includes > > developers from several institutions and industry participants. The > > developers are also from different regions like U.S, Australia, India, > > and the development team leverages slack, community mailing list, > > weekly community calls to collaborate efficiently. > > Reliance on Salaried Developers > > > > Clearly, Cloudera has contributed most of the initial development > > through salaried developers. But since the very beginning, YuniKorn is > > built as a community effort project. We have people from other > > organizations that are already collaborating with us on github. This > > includes both at the source code level, as well as participating in > > designs and providing feedback through community calls. We expect our > > reliance on salaried developers to decrease drastically during the > > incubation process itself. > > Relationship to Other Apache Products > > > > > > YuniKorn is very closely related to other Big-Data projects in Apache, > > such as Hadoop YARN, Spark, Hive, Flink, etc. > > > > YuniKorn’s core idea is to support both long-running and batch > > workloads like Spark, Hive, Flink etc, and provide a consistent, > > unified way to manage and schedule resources for Big Data workloads > > across resource managers like Apache™ Hadoop® YARN / Kubernetes and > > on-premise and cloud environments. > > > > Many of the core ideas for YuniKorn come from the experience of the > > initial team building Apache Hadoop YARN’s schedulers - Capacity > > Scheduler and Fair Scheduler. > > An Excessive Fascination with the Apache Brand > > > > Many of the initial developers in YuniKorn project are already > > experienced Apache committers, PMC members. We understand the value of > > the Apache way, and how to operate the project development on a day to > > day basis. The reason for proposing YuniKorn as an Apache project is > > to build a healthy community, increasing adoption & the size of the > > community and end users, because we believe the only way to build a > > highly valuable infrastructure layer software is to have wide adoption > > and cater to common use cases. > > Documentation > > > > Project summary: > > > > https://github.com/cloudera/yunikorn-core/blob/master/README.md > > > > User guides > > > > > https://github.com/cloudera/yunikorn-core/blob/master/docs/user-guide.md > > > > Developer guides > > > > > https://github.com/cloudera/yunikorn-core/blob/master/docs/developer-guide.md > > > > Roadmap: > > > > https://github.com/cloudera/yunikorn-core/blob/master/docs/roadmap.md > > > > Initial Source > > > > YuniKorn is written in Golang, and currently, the source code is > > hosted in several GitHub repositories > > > > Scheduler interface: > > https://github.com/cloudera/yunikorn-scheduler-interface > > Scheduler core: https://github.com/cloudera/yunikorn-core > > K8s Shim:https://github.com/cloudera/yunikorn-k8shim > > Scheduler Web UI: https://github.com/cloudera/yunikorn-web > > > > Source and Intellectual Property Submission Plan > > External Dependencies > > > > External dependencies are listed in below table > > > > Library > > > > > > Type > > > > > > License > > > > k8s.io/api > > > > > > K8s API > > > > > > Apache License 2.0 > > > > k8s.io/apimachinery > > > > > > K8s API > > > > > > Apache License 2.0 > > > > k8s.io/client-go > > > > > > K8s client library > > > > > > Apache License 2.0 > > > > github.com/looplab/fsm > > > > > > Go state machine library > > > > > > MIT License > > > > github.com/satori/go.uuid > > > > > > Go UUID library > > > > > > MIT License > > > > github.com/uber-go/zap > > > > > > Go logging library > > > > > > MIT License > > > > github.com/golang/protobuf > > > > > > Go protobuf library > > > > > > BSD 3-Clause License > > > > github.com/gorilla/mux > > > > > > Go network library > > > > > > BSD 3-Clause License > > > > google.golang.org/grpc > > > > > > Go RPC library > > > > > > Apache License 2.0 > > > > gopkg.in/yaml.v2 > > > > > > Go YAML library > > > > > > Apache License 2.0 > > > > github.com/prometheus/client_golang > > > > > > Prometheus Client Library > > > > > > Apache License 2.0 > > > > Angular v6.1.x > > > > > > Angular UI Framework Libraries > > > > > > MIT License > > > > TypeScript > > > > > > TypeScript Language Compiler > > > > > > Apache License 2.0 > > > > Chart.js > > > > > > JavaScript Charting Library > > > > > > MIT License > > > > Moment.js > > > > > > JavaScript Date & Time Library > > > > > > MIT License > > > > > > Build and test only: > > > > gotest.tools > > > > > > Test library > > > > > > Apache License 2.0 > > > > github.com/stretchr/testify > > > > > > Test library > > > > > > MIT License > > > > Karma > > > > > > Unit test library > > > > > > MIT License > > > > Protactor > > > > > > End2End test library > > > > > > MIT License > > > > Json-server > > > > > > Test server > > > > > > MIT License > > > > Yarn > > > > > > Dependency manager > > > > > > BSD 2-Clause License > > > > > > Cryptography > > > > YuniKorn does not currently include any cryptography-related code. > > Required Resources > > Mailing lists: > > > > priv...@yunikorn.incubator.apache.org (PMC list) > > comm...@yunikorn.incubator.apache.org (git push emails) > > iss...@yunikorn.incubator.apache.org (JIRA issue feed) > > d...@yunikorn.incubator.apache.org (Dev discussion) > > u...@yunikorn.incubator.apache.org (User questions) > > > > Git Repositories > > > > Git is the preferred source control system > > > > git://git.apache.org/yunikorn-* (We have multiple git repositories) > > > > Issue Tracking > > > > JIRA YuniKorn (YUNIKORN-) > > Other Resources > > > > None > > Initial Committers and Affinities > > > > Akhil PB (a...@cloudera.com) (Cloudera) > > Sunil Govindan (sun...@apache.org) (Cloudera) > > Vinod Kumar Vavilapalli (vino...@apache.org) (Cloudera) > > Wangda Tan (wan...@apache.org) (Cloudera) > > Weiwei Yang (w...@apache.org) (Cloudera) > > Wilfred Spiegelenburg (wspiegelenb...@cloudera.com) (Cloudera) > > Carlo Curino (cur...@apache.org) (Microsoft) > > Subramaniam Krishnan (su...@apache.org) (Microsoft) > > Arun Suresh (asur...@apache.org) (Microsoft) > > Konstantinos Karanasos (kkarana...@apache.org) (Microsoft) > > Jonathan Hung (jh...@apache.org) (LinkedIn) > > DB Tsai (dbt...@apache.org) (Apple) > > Junping Du (junping...@apache.org) (Tencent) > > Tao Yang (taoy...@apache.org) (Alibaba) > > Jason Lowe (jl...@apache.org) (Nvidia) > > > > Sponsors > > Champion > > > > Vinod Kumar Vavilapalli (vino...@apache.org) > > Nominated Mentors > > > > Junping Du (Tencent), (junping...@apache.org) > > > > Felix Cheung (Uber), (felixche...@apache.org) > > > > Jason Lowe (Nvidia), (jl...@apache.org) > > > > Holden Karau (Apple), (hol...@apache.org) > > Sponsoring Entity > > > > The Apache Incubator > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- Supun Kamburugamuve, PhD Digital Science Center, Indiana University Member, Apache Software Foundation; http://www.apache.org E-mail: supun@apache.o <supu...@gmail.com>rg; Mobile: +1 812 219 2563
