This will be a great addition. +1
On Wed, May 31, 2017 at 9:03 AM, Sean Busbey <bus...@apache.org> wrote: > Hi folks! > > I'm calling a vote to accept "Livy" into the Apache Incubator. > > The full proposal is available below, and is also available in the wiki: > > https://wiki.apache.org/incubator/LivyProposal > > For additional context, please see the discussion thread: > > https://s.apache.org/incubator-livy-proposal-thread > > Please cast your vote: > > [ ] +1, bring Livy into Incubator > [ ] -1, do not bring Livy into Incubator, because... > > The vote will open at least for 72 hours and only votes from the Incubator > PMC are binding. > > I start with my vote: > +1 > > ---- > > = Abstract = > > Livy is web service that exposes a REST interface for managing long running > Apache Spark contexts in your cluster. With Livy, new applications can be > built on top of Apache Spark that require fine grained interaction with > many > Spark contexts. > > = Proposal = > > Livy is an open-source REST service for Apache Spark. Livy enables > applications to submit Spark applications and retrieve results without a > co-location requirement on the Spark cluster. > > We propose to contribute the Livy codebase and associated artifacts (e.g. > documentation, web-site context etc) to the Apache Software Foundation. > > = Background = > > Apache Spark is a fast and general purpose distributed compute engine, with > a versatile API. It enables processing of large quantities of static data > distributed over a cluster of machines, as well as processing of continuous > streams of data. It is the preferred distributed data processing engine for > data engineering, stream processing and data science workloads. Each Spark > application uses a construct called the SparkContext, which is the > application’s connection or entry point to the Spark engine. Each Spark > application will have its own SparkContext. > > Livy enables clients to interact with one or more Spark sessions through > the > Livy Server, which acts as a proxy layer. Livy Clients have fine grained > control over the lifecycle of the Spark sessions, as well as the ability to > submit jobs and retrieve results, all over HTTP. Clients have two modes of > interaction: RPC Client API, available in Java and Python, which allows > results to be retrieved as Java or Python objects. The serialization and > deserialization of the results is handled by the Livy framework. HTTP based > API that allows submission of code snippets, and retrieval of the results > in > different formats. > > Multi-tenant resource allocation and security: Livy enables multiple > independent Spark sessions to be managed simultaneously. Multiple clients > can also interact simultaneously with the same Spark session and share the > resources of that Spark session. Livy can also enforce secure, > authenticated > communication between the clients and their respective Spark sessions. > > More information on Livy can be found at the existing open source website: > http://livy.io/ > > = Rationale = > > Users want to use Spark’s powerful processing engine and API as the data > processing backend for interactive applications. However, the job > submission > and application interaction mechanisms built into Apache Spark are > insufficient and cumbersome for multi-user interactive applications. > > The primary mechanism for applications to submit Spark jobs is via > spark-submit > (http://spark.apache.org/docs/latest/submitting-applications.html), which > is > available as a command line tool as well as a programmatic API. However, > spark-submit has the following limitations that make it difficult to build > interactive applications: It is slow: each invocation of spark-submit > involves a setup phase where cluster resources are acquired, new processes > are forked, etc. This setup phase runs for many seconds, or even minutes, > and hence is too slow for interactive applications. It is cumbersome and > lacks flexibility: application code and dependencies have to be > pre-compiled > and submitted as jars, and can not be submitted interactively. > > Apache Spark comes with an ODBC/JDBC server, which can be used to submit > SQL > queries to Spark. However, this solution is limited to SQL and does not > allow the client to leverage the rest of the Spark API, such as RDDs, MLlib > and Streaming. > > A third way of using Spark is via its command-line shell, which allows the > interactive submission of snippets of Spark code. However, the shell > entails > running Spark code on the client machine and hence is not a viable > mechanism > for remote clients to submit Spark jobs. > > Livy solves the limitations of the above three mechanisms, and provides the > full Spark API as a multi-tenant service to remote clients. > > Since the open source release of Livy in late 2015, we have seen tremendous > interest among a diverse set of application developers and ISVs that want > to > build applications with Apache Spark. To make Livy a robust and flexible > solution that will enable a broad and growing set of applications, it is > important to grow a large and varied community of contributors. > > = Initial Goals = > > * Move existing codebase, website, documentation and mailing lists to > Apache-hosted infrastructure > * Work with the infrastructure team to implement and approve our code > review, build, and testing workflows in the context of the ASF > * Incremental development and releases per Apache guidelines > > = Current Status = > > The Livy project began at Cloudera, as a part of the Hue project. Cloudera > soon realized the broad applicability of Livy, and separated it out into an > independent project in Nov 2015. > > == Releases == > > Livy has undergone two public releases, tagged here: > > * https://github.com/cloudera/livy/releases/tag/v0.2.0 > * https://github.com/cloudera/livy/releases/tag/v0.3.0 > > Tarballs and zip files were created for each release and hosted on github. > Upon joining the incubator, we will adopt a more typical ASF release > process. > > == Source == > > Livy’s source is currently hosted on Github at: > https://github.com/cloudera/livy > > This repository will be transitioned to Apache’s git hosting during > incubation. > > == Code review == > > Livy’s code reviews are currently public and hosted on github as pull > request reviews at: https://github.com/cloudera/livy/pulls > The Livy developer community so far is happy with github pull request > reviews and hopes to continue this after being admitted to the ASF. > > == Issue Tracking == > > Livy’s bug and feature tracking is hosted on JIRA at: > https://issues.cloudera.org/projects/LIVY/summary > This JIRA instance contains bugs and development discussion dating back 1 > year and will provide an initial seed for the ASF JIRA > > == Community Discussion == > > Livy has several public discussion forums: > > * https://groups.google.com/a/cloudera.org/forum/#!forum/livy-dev > * https://groups.google.com/a/cloudera.org/forum/#!forum/livy-user > > == Development Practices == > > The Livy project follows a review before commit philosophy. Every commit > automatically runs through the unit tests and generates coverage reports > presented as a pull request comment. Our experience with this process leads > us to believe that it helps ease new contributors into the project. They > get > feedback quickly on common mistakes, lowering the burden on reviewers. > Those > same reviewers get to lead by example, showing the new contributors that we > value feedback within our community even when changes are done by more > experienced folks. > > == Meritocracy == > > We believe strongly in meritocracy when electing committers and PMC > members. > In the past few months, the project has added two new committers from two > different organisations, in recognition of their significant contributions > to the project. We will encourage contributions and participation of all > types, and ensure that contributors are appropriately recognized. > > == Community == > > Though Livy is relatively new as a standalone open source project, it has > already seen promising growth in its community across several > organizations: > Cloudera is the original development sponsor for Livy > Microsoft pushed the development of the interpreter fixing high > availability > issues and adding additional features. > Hortonworks has contributed the security features to Livy allowing kerberos > and impersonation to work with Spark > IBM is starting to make contributions to the Livy project > A number of other patches contributed by community members > > Livy currently relies on Google Groups for mailing lists. These lists have > been active since the end of 2015/start of 2016. Currently, Livy’s user > mailing list has 173 subscribers and has hosted a total of 227 topic > threads. Livy’s developer list has 49 subscribers and has hosted 79 topic > threads. > > == Core Developers == > > The early contributions to Livy were made by Cloudera engineers. In 2016, > engineers from Microsoft and Hortonworks joined the core developer > community. > > == Alignment == > > Livy is built upon Apache Spark, and other Apache projects like Apache > Hadoop YARN. It’s used as a building block by Apache Zeppelin. These > community connections combined with our focus on development practices that > emphasize community engagement with a path to meritocratic recognition > naturally align us with the ASF. > > = Known Risks = > > == Orphaned Products == > > The risk of Livy being abandoned is low because it is supported by three > major big-data software vendors. Moreover, Livy is already used to power > multiple releases of services and products used in production. > > == Inexperience with Open Source == > > Several of the initial committers are experienced open source developers, > several being committers and/or PMC members on other ASF projects (Spark, > YARN). > > == Homogenous Developers == > > The project already has a diverse developer base. It has contributions from > 3 major organisations (Cloudera, Microsoft and Hortonworks), and is used in > diverse applications, in diverse settings (On-Prem and Cloud). > > == Reliance on salaried Developers == > > The contributions to the Livy project to date have been made by salaried > engineers from Cloudera, Microsoft and Hortonworks. One of the individuals > on the initial committer list has since left Microsoft and is currently > unaffiliated. The remaining contributors are from Cloudera and Hortonworks. > Since there are at least two major organizations involved, the risk of > reliance on a single group of salaried developers is mitigated. The Livy > user base is diverse, with users from across the globe, including users > from > academic settings. We aim to further diversify the Livy user and > contributor > base. > > == Relationships with other Apache projects == > > Livy is closely tied to the Apache Spark project and currently addresses > the > scenarios for a REST based batch and interactive gateway for Spark jobs on > YARN. Given the growing number of integrations with Livy, keeping it > outside > of Apache Spark aligns with the desire of the Apache Spark community to > reduce the number of external dependencies in the Spark project. > Specifically, the Apache Spark community has previously expressed a desire > to keep job servers independent from the project.<<FootNote(See, for > example, discussion of the Ooyala Spark Job Server in SPARK-818)>> > Furthermore, while Livy common usage is closely tied to Spark deployments > right now, its core building blocks can be reused elsewhere. Livy’s Remote > REPL could be used as a library for interactive scenarios in non-Spark > projects. In the future, integrations with cluster managers like Apache > Mesos and others could also be added. > > The features provided by Livy have already been integrated with existing > projects like Jupyter and Apache Zeppelin for their interactive Spark use > cases. This validates the need for a project like Livy and provides an > active downstream user base that the Livy community can interact with to > seed future interest in the project. > > Livy serves a similar purpose to Apache Toree (incubating) but differs in > making session management, security and impersonation a focal design point. > > == An Excessive Fascination with the Apache Brand == > > The primary motivation for submitting Livy to the ASF is to grow a diverse > and strong community. We wish to encourage diverse organisations, including > ISVs, to adopt Livy and contribute to Livy without any concerns about > ownership or licensing. > > = Documentation = > > Documentation can be found on the Livy website http://livy.io/ > > The Livy web site is version controlled on the ‘gh-pages’ branch of the > above repository. > Additional documentation is provided on the github wiki: > https://github.com/cloudera/livy/wiki > APis are documented within the source code as JavaDoc style documentation > comments. > > = Initial Source = > > The initial source code for Livy is hosted at > https://github.com/cloudera/livy > > = Source and Intellectual Property submission plan = > > The Livy codebase and web site is currently hosted on GitHub and will be > transitioned to the ASF repositories during incubation. Livy is already > licensed under the Apache 2.0 license. Cloudera has collected ICLAs and > CCLAs from all committers. There are, however, some contributions recently > from authors that have not signed the CCLA and ICLA. If necessary for a > successful SGA, we’ll seek the necessary documentation or replace the > contributions. > > The “Livy” name is not a registered trademark. We will need to do a > trademark search and make sure it is available for the Apache Foundation > prior to graduation. > > Cloudera currently owns the domain name: http://livy.io/. Once all the > documentation has moved over to ASF infrastructure, the main landing page > will become livy.incubator.apache.org and the old domain will just act as > a > redirect. > > = External Dependencies = > > The list below covers the non-Apache dependencies of the project and their > licenses. > > * Jetty: Apache 2.0 > * Dropwizard Metrics: Apache 2.0 > * FasterXML Jackson: Apache 2.0 > * Netty: Apache 2.0 > * Scala: BSD > * Py4J: BSD > * Scalatra: BSD > > Build/test-only dependencies: > > * Mockito: MIT > * JUnit: Eclipse > > = Required Resources = > > == Mailing Lists == > > * priv...@livy.incubator.apache.org (PPMC) > * d...@livy.incubator.apache.org (dev mailing list) > * u...@livy.incubator.apache.org (User questions) > * comm...@livy.incubator.apache.org (subscribers shouldn’t be able to > post) > * iss...@livy.incubator.apache.org (subscribers shouldn’t be able to > post) > > == Git Repository == > > git://git.apache.org/incubator-livy > > == Issue Tracking == > > We would like to import our current JIRA project into the ASF JIRA, such > that our historical commit message and code comments continue to reference > the appropriate bug numbers. > > = Initial Committers = > > * Marcelo Vanzin (van...@cloudera.com) > * Alex Man (alex@alexman.space) > * Jeff Zhang (zjf...@gmail.com) > * Saisai Shao (ss...@hortonworks.com) > * Kostas Sakellis (kos...@cloudera.com) > > = Affiliations = > > The initial set of committers includes people employed by Cloudera and > Hortonworks as well as one currently independent contributor. > > = Additional Interested Contributors = > > Those interested in getting involved with the project as we enter > incubation > are encouraged to list themselves here. > > * Ismaël Mejía (ieme...@apache.org) > > = Sponsors = > > == Champion == > > Sean Busbey (bus...@apache.org) > > == Nominated Mentors == > > * Bikas Saha (bi...@apache.org) > * Brock Noland (br...@phdata.io) > * Luciano Resende (lrese...@apache.org) > > == Sponsoring Entity == > > We ask that the Incubator PMC sponsor this proposal. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
