Thanks William! It will be great to work with you again. On Thu, Jun 15, 2017 at 10:12 AM, Supun Kamburugamuve <supu...@gmail.com> wrote:
> Thank you, William, for offering to help with the incubation process. It > will be really helpful. > > Supun.. > > On Wed, Jun 14, 2017 at 11:04 PM, William Markito Oliveira < > william.mark...@gmail.com> wrote: > > > Howdy! > > > > If Heron is looking for some help around incubation process, I'd love to > > help while Geode experience is still fresh in my mind and given that > it's a > > project/space that I do have interest. Since I'm not an ASF member, I > don't > > think I can offer to be a mentor, but can probably still help and > > participate on the process. > > > > Thanks! > > > > On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz <ptgo...@gmail.com> > > wrote: > > > > > Hi Bill/Supun, > > > > > > Sorry for not being a little more clear. I was asking more about how > the > > > Heron community would seek to engage with Storm community at the > > > *community* level as opposed to the technical level (i.e. “Community > over > > > Code”). > > > > > > I’ve been asked by many why this has never happened, and have always > > > struggled to answer. Maybe you could help answer that question as well > as > > > if and how that might change if Heron were to incubate. > > > > > > Another quick question: The proposal mentions Heron being used in > > > production at Google, but some Google employees I recently spoke to > > seemed > > > to contradict that. Could you explain? Note that’s nothing that would > > > preclude the project from incubating, I’m just curious. > > > > > > -Taylor > > > > > > > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve <supu...@gmail.com> > > > wrote: > > > > > > > > Hi Taylor, > > > > > > > > For me, one of the interesting differences between Heron and Storm is > > the > > > > execution model. Storm uses a shared memory model while Heron uses a > > > > process based model. It will be interesting to see how these two > > evolve. > > > > > > > > Thanks, > > > > Supun.. > > > > > > > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham <billgra...@gmail.com> > > > wrote: > > > > > > > >> Hi Taylor, > > > >> > > > >> Thanks for the mentor offer, we'd be glad to have your help. > > > >> > > > >> I think the best place for collaboration would be around the > evolution > > > of > > > >> the API. In addition we plan to look more into DSL solutions which > we > > > could > > > >> potentially collaborate on. This could be Trident, or Beam or > > something > > > >> else, but there could be synergies for future development here. > > > >> > > > >> thanks, > > > >> Bill > > > >> > > > >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz <ptgo...@gmail.com> > > > wrote: > > > >> > > > >>> Hi Bill, > > > >>> > > > >>> Could you comment on how/if the Heron community would be willing to > > > work > > > >>> with the Storm community? I've seen a number of new features in > Storm > > > >> being > > > >>> ported to Heron, but I have yet to see any attempt by the Heron > > > community > > > >>> to engage with the Apache Storm community. > > > >>> > > > >>> I don't think it would be too far off to say that the relationship > > > >> between > > > >>> Heron and Apache Storm has been somewhat adversarial. The pre- and > > > >>> post-open sourcing marketing around Heron seemed, at least to me, > > > >> somewhat > > > >>> aggressively negative toward Storm. > > > >>> > > > >>> As a peer to Apache Storm, how would the proposed "Apache Heron" > > > >> community > > > >>> work to collaborate with the Storm community? If Heron is adopting > > API > > > >>> changes in Storm, then it seems there is an opportunity for > > > >> collaboration. > > > >>> > > > >>> Don't take any of this as an objection to incubating the project. I > > > would > > > >>> support it. I would also be willing to be a mentor, if you would > > > consider > > > >>> taking on another. > > > >>> > > > >>> -Taylor > > > >>> > > > >>>> On Jun 8, 2017, at 1:23 PM, Bill Graham <billgra...@gmail.com> > > wrote: > > > >>>> > > > >>>> Dear Apache Incubator Community, > > > >>>> > > > >>>> We are excited to share our proposal for discussion and feedback > > > >>>> for entering Apache Incubation. Heron is a real-time, distributed, > > > >>>> fault-tolerant stream processing engine. > > > >>>> > > > >>>> Our proposal can be found at https://wiki.apache.org/ > > > >>> incubator/HeronProposal > > > >>>> and is included below. > > > >>>> > > > >>>> > > > >>>> Thank you, > > > >>>> > > > >>>> Bill Graham on behalf of the Heron developers > > > >>>> > > > >>>> > > > >>>> # Heron Proposal > > > >>>> > > > >>>> ## Abstract > > > >>>> Heron is a real-time, distributed, fault-tolerant stream > processing > > > >>> engine > > > >>>> initially developed by Twitter. > > > >>>> > > > >>>> ## Proposal > > > >>>> > > > >>>> Heron is a real-time stream processing engine built for high > > > >> performance, > > > >>>> ease of manageability, performance predictability and developer > > > >>>> productivity[1]. We wish to develop a community around Heron to > > > >> increase > > > >>>> contributions and see Heron thrive in an open forum. > > > >>>> > > > >>>> ## Background > > > >>>> > > > >>>> Heron provides the ability for developers to compose directed > > acyclic > > > >>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) > > and > > > >>>> submit the topology to execute on a pluggable job scheduling > system > > > >>> (e.g., > > > >>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the > > > native > > > >>>> Heron API or the Apache Storm API to develop the topology. Heron > > > >> supports > > > >>>> the Storm API for ease of migration, but beyond that Heron’s > > > >> architecture > > > >>>> differs considerably from Storm’s. > > > >>>> > > > >>>> Users submit a topology to the scheduler using the Heron client, > > which > > > >>> uses > > > >>>> the Heron binary libraries to deploy all daemons required to run > and > > > >>> manage > > > >>>> the topology. The topology therefore has no reliance on centrally > > > >> managed > > > >>>> Heron services, only on a generic job scheduling system, which > lends > > > >>> itself > > > >>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > > > >> (among > > > >>>> others). > > > >>>> > > > >>>> The scheduler runs each topology as a job consisting of multiple > > > >>>> containers. One of the containers runs the topology master, > > > responsible > > > >>> for > > > >>>> managing the topology. The remaining containers each runs a stream > > > >>> manager > > > >>>> responsible for data routing, a metrics manager that collects and > > > >> reports > > > >>>> various metrics and a number of processes called Heron instances > > which > > > >>> run > > > >>>> the user-defined logic on the stream of tuples. Parallelism is > > > achieved > > > >>> via > > > >>>> process-based isolation of Heron instances, which provides > > predictable > > > >>>> performance while simplifying debugging. The containers are > > allocated > > > >> and > > > >>>> managed by the scheduler framework based on resource availability > of > > > >>> nodes > > > >>>> in the cluster. The metadata for the topology, such as the > physical > > > >> plan > > > >>>> and execution details, are stored in the pluggable Heron State > > Manager > > > >>>> (e.g. Apache ZooKeeper). > > > >>>> > > > >>>> ## Rationale > > > >>>> > > > >>>> Heron is a general-purpose, modular and extensible platform that > can > > > be > > > >>>> leveraged to support common, real-time analytics use cases. There > is > > > an > > > >>>> increasing demand for open-source, scalable real-time analytics > > > >> systems. > > > >>> We > > > >>>> believe that Heron can be leveraged by other organizations to > build > > > >>>> streaming applications that can benefit from its robustness, high > > > >>>> performance, adaptability to cloud environments and ease of use. > > > >>> Moreover, > > > >>>> we hope that open-sourcing Heron will help to further evolve the > > > >>> technology > > > >>>> as the project attracts contributors with diverse backgrounds and > > > areas > > > >>> of > > > >>>> expertise. > > > >>>> > > > >>>> We believe the Apache foundation is a great fit as the long-term > > home > > > >> for > > > >>>> Heron, as it provides an established process for community-driven > > > >>>> development and decision making by consensus. This is exactly the > > > model > > > >>> we > > > >>>> want for future Heron development. > > > >>>> > > > >>>> ## Initial Goals > > > >>>> > > > >>>> * Move the existing codebase, website, documentation, and mailing > > > lists > > > >>> to > > > >>>> Apache-hosted infrastructure. > > > >>>> * Integrate with the Apache development process. > > > >>>> * Ensure all dependencies are compliant with Apache License > version > > > >> 2.0. > > > >>>> * Incrementally develop and release per Apache guidelines. > > > >>>> > > > >>>> ## Current Status > > > >>>> > > > >>>> Heron is a stable project used in production at Twitter since 2014 > > and > > > >>> open > > > >>>> sourced under the ASL v2 license in 2016. The Heron source code is > > > >>>> currently hosted at github.com (https://github.com/twitter/heron > ), > > > >> which > > > >>>> will seed the Apache git repository. > > > >>>> > > > >>>> ### Meritocracy > > > >>>> > > > >>>> By submitting this incubator proposal, we’re expressing our intent > > to > > > >>> build > > > >>>> a diverse developer community around Heron that will conduct > itself > > > >>>> according to The Apache Way and use a meritocratic means of > building > > > >> it's > > > >>>> committer base. Several companies and universities have already > > > >> expressed > > > >>>> interest in and contributed to Heron. Our goal is to grow the > Heron > > > >>>> community by encouraging open communication, contribution and > > > >>> participation > > > >>>> of all types, and ensuring that contributors are recognized > > > >>> appropriately. > > > >>>> > > > >>>> ### Community > > > >>>> > > > >>>> Heron is currently being used by Twitter, Google, Machine Zone and > > > >>>> ndustrial.io and has received significant contributions by > > Microsoft > > > >> and > > > >>>> Streamlio. By bringing Heron into the Apache ecosystem, we believe > > we > > > >> can > > > >>>> attract even more developers who are interested in creating > > real-time > > > >>>> systems to build the project's contributor base. > > > >>>> > > > >>>> ### Core Developers > > > >>>> > > > >>>> Current core developers are engineers from Twitter, Google, > > Microsoft > > > >> and > > > >>>> Streamlio. > > > >>>> > > > >>>> ### Alignment > > > >>>> > > > >>>> Heron utilizes a number of Apache technologies. Heron leverages > > Apache > > > >>>> ZooKeeper for coordination and has scheduler implementations to > > > >> integrate > > > >>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via > > Apache > > > >>> REEF) > > > >>>> as well as spout implementations to integrate with Apache Kafka > and > > > >>> metrics > > > >>>> implementations to integrate with Scribe. Heron also implements > the > > > >>> Apache > > > >>>> Storm user-level API, which allows topologies written against > Storm > > to > > > >>> run > > > >>>> in Heron. We believe that having Heron at Apache will help further > > the > > > >>>> growth of the streaming compute community, as well as encourage > > > >>> cooperation > > > >>>> and developer cross pollination with other Apache projects. > > > >>>> > > > >>>> ## Known Risks > > > >>>> > > > >>>> ### Orphaned Products > > > >>>> > > > >>>> The risk of the Heron project being abandoned is minimal. It is > used > > > in > > > >>>> production at Twitter and Google and other companies are > evaluating > > or > > > >>>> adopting it for production use. > > > >>>> > > > >>>> ### Inexperience with Open Source > > > >>>> > > > >>>> All of the core contributors to the project have considerable > > > >> experience > > > >>>> with open source software development. Bill Graham[2], Ashvin > > > >> Agrawal[3] > > > >>>> and Supun Kamburugamuve[4], committers on the project, are PMCs on > > > >> other > > > >>>> Apache projects and Bill and Ashvin have gone through the Apache > > > >>> incubator > > > >>>> process. Twitter has already donated numerous projects to the ASF > > > >> (e.g., > > > >>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > > > >> mentored > > > >>>> by experienced ASF members that can help with any roadblocks. > > > >>>> > > > >>>> ### Homogenous Developers > > > >>>> > > > >>>> Initial committers come from 5 separate organizations. Our > intention > > > is > > > >>>> increase the diversity of contributing developers and their > > > >> affiliations. > > > >>>> To date github contributions have come from approximately 50 > > > >> contributors > > > >>>> from outside the Twitter team. > > > >>>> > > > >>>> ### Reliance on Salaried Developers > > > >>>> > > > >>>> It is expected that Heron development will occur on both salaried > > time > > > >>> and > > > >>>> on volunteer time. The majority of initial committers are paid by > > > their > > > >>>> employers to contribute to this project. We are committed to > > > recruiting > > > >>>> additional committers from other organizations as well as > > non-salaried > > > >>>> committers to join project. > > > >>>> > > > >>>> ### Relationships with Other Apache Products > > > >>>> > > > >>>> As mentioned in the Alignment section, Heron implements the Apache > > > >> Storm > > > >>>> API and integrates with multiple Apache schedulers (Apache Mesos, > > > >> Apache > > > >>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and > > > Apache > > > >>>> Thrift. > > > >>>> > > > >>>> ### An Excessive Fascination with the Apache Brand > > > >>>> > > > >>>> Heron's popularity is growing in the streaming compute space and > we > > > are > > > >>>> long time supporters of the Apache brand. This proposal is not for > > the > > > >>>> purpose of generating publicity through. Rather, the primary > > benefits > > > >> to > > > >>>> joining Apache are those of community building and open decision > > > making > > > >>>> outlined in the Rationale section. > > > >>>> > > > >>>> ## Documentation > > > >>>> > > > >>>> This proposal exists online as http://wiki.apache.org/ > > > >>>> incubator/HeronProposal. Extensive documentation can be found on > > > github > > > >>> at > > > >>>> https://twitter.github.io/heron and the source code is well > > > >> documented. > > > >>>> > > > >>>> ## Source and Intellectual Property Submission Plan > > > >>>> > > > >>>> The Heron codebase is currently hosted on Github: > > > >>>> https://github.com/twitter/heron. During incubation, the codebase > > > will > > > >>> be > > > >>>> migrated to Apache infrastructure. The source code is already ASF > > 2.0 > > > >>>> licensed. > > > >>>> > > > >>>> ## External Dependencies > > > >>>> > > > >>>> All external libraries have ASF 2.0 compatible licenses except for > > > >>> pylint. > > > >>>> The pylint library is GPL licensed, but is only used for pre-build > > > >> Python > > > >>>> style checks and is neither bundled with, nor relied upon by, the > > > Heron > > > >>>> source or binary release artifacts. > > > >>>> > > > >>>> ## Cryptography > > > >>>> > > > >>>> Heron does not use any cryptography libraries. > > > >>>> > > > >>>> ## Required Resources > > > >>>> > > > >>>> ### Mailing lists > > > >>>> > > > >>>> priv...@heron.incubator.apache.org (with moderated subscriptions) > > > >>>> d...@heron.incubator.apache.org > > > >>>> comm...@heron.incubator.apache.org > > > >>>> u...@heron.incubator.apache.org > > > >>>> > > > >>>> ## Subversion Directory > > > >>>> > > > >>>> Git is the preferred source control system: git:// > > > git.apache.org/heron > > > >>>> > > > >>>> ## Issue Tracking > > > >>>> > > > >>>> JIRA: Heron (HERON) > > > >>>> > > > >>>> ## Initial Committers > > > >>>> > > > >>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > > >>>> * Ashvin Agrawal (ashvin at apache dot org)* > > > >>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > > >>>> * Bill Graham (billgraham at apache dot org)* > > > >>>> * Brian Hatfield (bmhatfield at gmail dot com) > > > >>>> * Chris Kellogg (cckellogg at gmail dot com) > > > >>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > > >>>> * Karthik Ramasamy (karthik at gmail dot com) > > > >>>> * Maosong Fu (maosongfu at gmail dot com) > > > >>>> * Neng Lu(freeneng at gmail dot com) > > > >>>> * Runhang Li (obj dot runhang at gmail dot com) > > > >>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > > >>>> * Supun Kamburugamuve (supun at apache dot org)* > > > >>>> * Thomas Sun (tom dot ssf at gmail dot com) > > > >>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > >>>> > > > >>>> ## Affiliations > > > >>>> > > > >>>> * Andrew Jorgensen (Google) > > > >>>> * Ashvin Agrawal (Microsoft) > > > >>>> * Avrilia Floratou (Microsoft) > > > >>>> * Bill Graham (Twitter) > > > >>>> * Brian Hatfield (Google) > > > >>>> * Chris Kellogg (Twitter) > > > >>>> * Huijun Wu (Twitter) > > > >>>> * Karthik Ramasamy (Streamlio) > > > >>>> * Maosong Fu (Twitter) > > > >>>> * Neng Lu (Twitter) > > > >>>> * Runhang Li (Twitter) > > > >>>> * Sanjeev Kulkarni (Streamlio) > > > >>>> * Supun Kamburugamuve (Indiana University) > > > >>>> * Thomas Sun (Twitter) > > > >>>> * Yaliang Wang (Twitter) > > > >>>> > > > >>>> ## Sponsors > > > >>>> > > > >>>> ### Champion > > > >>>> > > > >>>> * Julien Le Dem (julien at apache dot org) > > > >>>> > > > >>>> ### Nominated Mentors > > > >>>> > > > >>>> * Jake Farrell (jfarrell at apache dot org) > > > >>>> * Jacques Nadeau (jacques at apache dot org) > > > >>>> * Julien Le Dem (julien at apache dot org) > > > >>>> > > > >>>> ### Sponsoring Entity > > > >>>> > > > >>>> The Apache Incubator > > > >>>> > > > >>>> ### Footnotes > > > >>>> > > > >>>> 1 - Papers detailing Heron are available at > > > http://dl.acm.org/citation > > > >> . > > > >>>> cfm?id=2742788 and http://sites.computer.org/ > debull/A15dec/p15.pdf. > > > >>>> 2 - http://home.apache.org/phonebook.html?uid=billgraham > > > >>>> 3 - http://home.apache.org/phonebook.html?uid=ashvin > > > >>>> 4 - http://home.apache.org/phonebook.html?uid=supun > > > >>> > > > >>> ------------------------------------------------------------ > > --------- > > > >>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > >>> For additional commands, e-mail: general-h...@incubator.apache.org > > > >>> > > > >>> > > > >> > > > > > > > > > > > > > > > > -- > > > > Supun Kamburugamuve > > > > Member, Apache Software Foundation; http://www.apache.org > > > > E-mail: supun@apache.o <supu...@gmail.com>rg; Mobile: +1 812 219 > 2563 > > > > <(812)%20219-2563> > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > > > > > > > > -- > > ~/William > > > > > > -- > Supun Kamburugamuve > Member, Apache Software Foundation; http://www.apache.org > E-mail: supun@apache.o <supu...@gmail.com>rg; Mobile: +1 812 219 2563 >
