+1 (binding)
On Fri, Aug 8, 2014 at 10:40 PM, Byung-Gon Chun <bgc...@gmail.com> wrote: > Hi, > > Thanks for participating in the proposal discussion on REEF. The discussion > has calmed. I would like to call a vote for acceptance of REEF into the > Apache Incubator. > > The proposal is attached below, and it is also available at > https://wiki.apache.org/incubator/ReefProposal > > Let's keep this vote open for three business days, closing the voting on > August 11, 11:59PM (PDT). > > [] +1 Accept REEF into the Incubator > [] 0 Don't care > [] -1 Don't accept REEF because... > > Thanks! > -Gon > > -- > Byung-Gon Chun > > > # REEFProposal - Incubator > > > # Abstract > > REEF (Retainable Evaluator Execution Framework) is a scale-out > computing fabric that eases the development of Big Data applications > on top of resource managers such as Apache YARN and Mesos. > > > # Proposal > > REEF is a Big Data system that makes it easy to implement scalable, > fault-tolerant runtime environments for a range of data processing > models (e.g., graph processing and machine learning) on top of > resource managers such as Apache YARN and Mesos. REEF provides > capabilities to run multiple heterogeneous frameworks and workflows of > those efficiently. > > Additionally, REEF contains two libraries that are of independent > value: Wake is an event-based-programming framework inspired by Rx and > SEDA. Tang is a dependency injection framework inspired by Google > Guice, but designed specifically for configuring distributed systems. > > > # Background > > The resource management layer such as Apache YARN and Mesos has > emerged as a critical layer in the new scale-out data processing > stack; resource managers assume the responsibility of multiplexing a > cluster of shared-nothing machines across heterogeneous > applications. They operate behind an interface for leasing containers > - a slice of a machine’s resources - to computations in an elastic > fashion. However, building data processing frameworks directly on this > layer comes at a high cost: each framework must tackle the same > challenges (e.g., fault-tolerance, task scheduling and coordination) > and reimplement common mechanisms (e.g., caching, bulk transfers). > > REEF provides a reusable control-plane for scheduling and coordinating > task-level work on cluster resource managers. The REEF design enables > sophisticated optimizations, such as container re-use and data > caching, and facilitates workflows that span multiple > frameworks. Examples include pipelining data between different > operators in a relational system, retaining state across iterations in > iterative or recursive data flow, and passing the result of a > MapReduce job to a Machine Learning computation. > > > # Rationale > > Since REEF is a library that makes it easy to write distributed > applications on top of Apache YARN or Mesos, the Apache Software Foundation > is the perfect home for hosting REEF. > > > # Current Status > > REEF has been developed mostly by Microsoft, UCLA and the Seoul > National University. The REEF codebase is open-sourced under Apache > License 2.0 and is currently hosted in a public repository at > github.com. > > > # Meritocracy > > We plan to build a strong open community by following the Apache > meritocracy principles. We will work with those who contribute > significantly to the project and invite them to be its committers. > > > # Community > > REEF is currently being used internally at Microsoft. Also, SK > Telecom builds their data analytics infrastructure on top of REEF in > collaboration with Seoul National University. We hope to extend our > contributor base by becoming an Apache incubator project. REEF will > attract developers who are interested in creating common building > blocks for simplifying the development of large-scale big data > applications. > > > # Core Developers > > Core developers are engineers from Microsoft, Purestorage, UCB, UCLA, > UW and Seoul National University. > > > # Alignment > > REEF depends on many Apache projects and dependencies. REEF is built > on resource managers such as Apache YARN and Apache Mesos. REEF also > uses HDFS as a distributed storage layer. > > > # Known Risks > ## Orphaned Products > > The risk of REEF being orphaned is small because Microsoft products > are built on REEF. The core REEF developers continue to work on REEF > at Microsoft, UCLA, and Seoul National University. The REEF project is > gaining interest from other institutions to be used as their > infrastructure. > > ## Inexperience with Open Source > > Several core developers have experience with open source development. > REEF committers will be guided by the mentors with strong Apache open > source project backgrounds. > > ## Homogeneous Developers > > The initial committers include developers from several institutions > including Microsoft, Purestorage, UCB, UCLA, and Seoul National > University. > > ## Reliance on Salaried Developers > > Developers from Microsoft are paid to work on REEF. Since the work is > used internally at Microsoft, Microsoft will keep supporting the > developers to work on REEF. There are also engineers and graduate > students that contribute to REEF from UCLA, UCB, UW and Seoul National > University. We plan to attract active developers from other > institutions. > > ## Relationships with Other Apache Products > > Given REEF's position in the big data stack, there are three > relationships to consider: Projects that fit below, on top of, or > alongside REEF in the stack. > > ### Below REEF: Mesos and YARN > > REEF is designed to facilitate application development on top of > resource managers. Hence, its relationship with the aforementioned > resource managers is symbiotic by design. > > ### On Top of REEF > > Apache Spark, Giraph, MapReduce and Flink are only some of the > projects that logically belong at a higher layer of the big data stack > than REEF. Of course, none of these today actually are leveraging > REEF and had to each individually solve some of the issues REEF > addresses. It is our goal that REEF will help developers create > an even richer set of future big data frameworks. > > ### Alongside REEF > > Apache hosts several projects building intermediate, library layers on > top of a resource management platform. Twill, Slider, and Tez are > notable examples in the incubator. These projects share many > objectives with REEF (and each other). We expect these parallel > explorations to converge and differentiate within Apache, as the space > for distributed applications and deployment is too vast for a single > answer. > > Apache Twill and REEF both aim to simplify application development on > top of resource managers. However, REEF and Twill go about this in > different ways: Twill simplifies programming by exposing a programming > model, Java Threads. REEF on the other hand provides a set of common > building blocks (e.g., job coordination, state passing, cluster > membership) for building big data processing applications and > virtualizes underlying resources managers. None of this prescribes a > specific programming model. As such, REEF occupies a slot ever so > slightly below Twill in an architecture stack. > > Apache Slider is a framework to make it easy to deploy and manage > long-running static applications in a YARN cluster. The focus is to > adapt existing applications such as HBase and Accumulo to run on YARN > with little modification. Therefore, the goals of Slider and REEF are > different. > > Apache Tez is a project to develop a generic Directed Acyclic Graph (DAG) > processing framework with a reusable set of data processing primitives. > The initial focus is to provide improved data processing capabilities for > projects like Apache Hive, Apache Pig, and Cascading. Tez is still a single > framework for DAG processing. In contrast, REEF provides a generic > layer on which diverse computation models (DAG, ML, Graph processing, > and Interactive query processing) can be built. More importantly, > REEF provides a layer that facilitates inter-framework resource and > in-memory state use and virtualizes resource managers. Regarding > re-usable data processing primitives, Tez and REEF share the same > goal. We hope to collaborate on features which can be shared between > Tez and REEF. > > Apache Helix automates application-wide management operations which require > global knowledge and coordination, such as repartitioning of resources and > scheduling of maintenance tasks. Helix separates global coordination > concerns from the functional tasks of the application with a state machine > abstraction. REEF's generic layer makes it easy to program the functional > and management tasks, which may span small or large groups within the > application. Helix can work hand-in-hand with REEF, by providing the global > management component for REEF applications. > > ## An Excessive Fascination with the Apache Brand > > The Apache Software Foundation has a reputation of being the best place to > host open source projects. We believe that we will attract many developers > who want to contribute to innovating in the Big Data platform space by > joining the Apache Software Foundation. > > > # Documentation > > The current documentation for REEF is at > https://github.com/Microsoft-CISL/REEF as well as on > http://www.reef-project.org > > > # Initial Source > > The REEF codebase is currently hosted at > https://github.com/Microsoft-CISL/REEF. > > > # External Dependencies > > REEF makes extensive use of the vast array of Java libraries from the > Apache Software Foundation, namely: > > * avro (Apache 2.0) > * hadoop (Apache 2.0) > * hdfs (Apache 2.0) > * yarn (Apache 2.0) > * commons-cli (Apache 2.0) > * commons-configuration (Apache 2.0) > * commons-lang (Apache 2.0) > * commons-logging (Apache 2.0) > > To the best of our knowledge, the external dependencies of REEF are > distributed under Apache compatible licenses: > > * guava-libraries (Apache 2.0) > * protobuf (BSD) > * asm (BSD) > * netty (Apache 2.0) > * mockito (MIT) > * junit (EPL 1.0) > * slf4j (MIT) > > > # Cryptography > > REEF will depend on secure Hadoop, which can optionally use Kerberos. > > # Required Resources > > ## Mailing Lists > > * reef-private for private PMC discussions > * reef-dev for technical discussions among contributors and > notification about commits > > ## Subversion Directory > > The REEF team uses Git for source version control: > git://git.apache.org/reef > > ## Issue Tracking > > JIRA REEF (REEF) > > ## Other Resources > > Jenkins continuous integration testing > > # Initial Committers > > * Markus Weimer > * Sergiy Matusevych > * Julia Wang > * Shravan M Narayanamurthy > * Yingda Chen > * Tony Majestro > * Beysim Sezgin > * Boris Shulman > * Russell Sears > * Jung Ryong Lee > * You Sun Jung > * Dong Joon Hyun > * Josh Rosen > * Tyson Condie > * Brandon Myers > * Yunseong Lee > * Taegeon Um > * Youngseok Yang > * Brian Cho > * Byung-Gon Chun > > # Affiliations > > * Microsoft: > * Markus Weimer > * Sergiy Matusevych > * Julia Wang > * Shravan M Narayanamurthy > * Yingda Chen > * Tony Majestro > * Beysim Sezgin > * Boris Shulman > * Purestorage: > * Russell Sears > * SK Telecom: > * Jung Ryong Lee > * You Sun Jung > * Dong Joon Hyun > * University of California: > * Josh Rosen (Berkeley) > * Tyson Condie (LA) > * University of Washington: > * Brandon Myers > * Seoul National University: > * Yunseong Lee > * Taegeon Um > * Youngseok Yang > * Brian Cho > * Byung-Gon Chun > > > # Sponsors > > ## Champions > Chris Douglas <cdoug...@apache.org> > > ## Nominated Mentors > * Chris Mattmann <mattm...@apache.org> > * Ross Gardler <rgard...@apache.org> > * Owen O'Malley <omal...@apache.org> > > ## Sponsoring Entity > The Apache Incubator > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
