Re: [PROPOSAL] Heron
ween >>>>>>> Heron and Apache Storm has been somewhat adversarial. The pre- and >>>>>>> post-open sourcing marketing around Heron seemed, at least to me, >>>>>> somewhat >>>>>>> aggressively negative toward Storm. >>>>>>> >>>>>>> As a peer to Apache Storm, how would the proposed "Apache Heron" >>>>>> community >>>>>>> work to collaborate with the Storm community? If Heron is adopting >> API >>>>>>> changes in Storm, then it seems there is an opportunity for >>>>>> collaboration. >>>>>>> >>>>>>> Don't take any of this as an objection to incubating the project. I >>>> would >>>>>>> support it. I would also be willing to be a mentor, if you would >>>> consider >>>>>>> taking on another. >>>>>>> >>>>>>> -Taylor >>>>>>> >>>>>>>> On Jun 8, 2017, at 1:23 PM, Bill Graham >> wrote: >>>>>>>> >>>>>>>> Dear Apache Incubator Community, >>>>>>>> >>>>>>>> We are excited to share our proposal for discussion and feedback >>>>>>>> for entering Apache Incubation. Heron is a real-time, distributed, >>>>>>>> fault-tolerant stream processing engine. >>>>>>>> >>>>>>>> Our proposal can be found at https://wiki.apache.org/ >>>>>>> incubator/HeronProposal >>>>>>>> and is included below. >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Bill Graham on behalf of the Heron developers >>>>>>>> >>>>>>>> >>>>>>>> # Heron Proposal >>>>>>>> >>>>>>>> ## Abstract >>>>>>>> Heron is a real-time, distributed, fault-tolerant stream processing >>>>>>> engine >>>>>>>> initially developed by Twitter. >>>>>>>> >>>>>>>> ## Proposal >>>>>>>> >>>>>>>> Heron is a real-time stream processing engine built for high >>>>>> performance, >>>>>>>> ease of manageability, performance predictability and developer >>>>>>>> productivity[1]. We wish to develop a community around Heron to >>>>>> increase >>>>>>>> contributions and see Heron thrive in an open forum. >>>>>>>> >>>>>>>> ## Background >>>>>>>> >>>>>>>> Heron provides the ability for developers to compose directed >> acyclic >>>>>>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) >> and >>>>>>>> submit the topology to execute on a pluggable job scheduling system >>>>>>> (e.g., >>>>>>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the >>>> native >>>>>>>> Heron API or the Apache Storm API to develop the topology. Heron >>>>>> supports >>>>>>>> the Storm API for ease of migration, but beyond that Heron’s >>>>>> architecture >>>>>>>> differs considerably from Storm’s. >>>>>>>> >>>>>>>> Users submit a topology to the scheduler using the Heron client, >> which >>>>>>> uses >>>>>>>> the Heron binary libraries to deploy all daemons required to run and >>>>>>> manage >>>>>>>> the topology. The topology therefore has no reliance on centrally >>>>>> managed >>>>>>>> Heron services, only on a generic job scheduling system, which lends >>>>>>> itself >>>>>>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN >>>>>> (among >>>>>>>> others). >>>>>>>> >>>>>>>> The scheduler runs each topology as a job consisting of multiple >>>>>>>> containers. One of the containers runs the topology master, >>>> responsible >>>>>>> f
Re: [PROPOSAL] Heron
t; >>>>> > > >>>>> As a peer to Apache Storm, how would the proposed "Apache Heron" > > >>>> community > > >>>>> work to collaborate with the Storm community? If Heron is adopting > > API > > >>>>> changes in Storm, then it seems there is an opportunity for > > >>>> collaboration. > > >>>>> > > >>>>> Don't take any of this as an objection to incubating the project. I > > >> would > > >>>>> support it. I would also be willing to be a mentor, if you would > > >> consider > > >>>>> taking on another. > > >>>>> > > >>>>> -Taylor > > >>>>> > > >>>>>> On Jun 8, 2017, at 1:23 PM, Bill Graham > > wrote: > > >>>>>> > > >>>>>> Dear Apache Incubator Community, > > >>>>>> > > >>>>>> We are excited to share our proposal for discussion and feedback > > >>>>>> for entering Apache Incubation. Heron is a real-time, distributed, > > >>>>>> fault-tolerant stream processing engine. > > >>>>>> > > >>>>>> Our proposal can be found at https://wiki.apache.org/ > > >>>>> incubator/HeronProposal > > >>>>>> and is included below. > > >>>>>> > > >>>>>> > > >>>>>> Thank you, > > >>>>>> > > >>>>>> Bill Graham on behalf of the Heron developers > > >>>>>> > > >>>>>> > > >>>>>> # Heron Proposal > > >>>>>> > > >>>>>> ## Abstract > > >>>>>> Heron is a real-time, distributed, fault-tolerant stream > processing > > >>>>> engine > > >>>>>> initially developed by Twitter. > > >>>>>> > > >>>>>> ## Proposal > > >>>>>> > > >>>>>> Heron is a real-time stream processing engine built for high > > >>>> performance, > > >>>>>> ease of manageability, performance predictability and developer > > >>>>>> productivity[1]. We wish to develop a community around Heron to > > >>>> increase > > >>>>>> contributions and see Heron thrive in an open forum. > > >>>>>> > > >>>>>> ## Background > > >>>>>> > > >>>>>> Heron provides the ability for developers to compose directed > > acyclic > > >>>>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) > > and > > >>>>>> submit the topology to execute on a pluggable job scheduling > system > > >>>>> (e.g., > > >>>>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the > > >> native > > >>>>>> Heron API or the Apache Storm API to develop the topology. Heron > > >>>> supports > > >>>>>> the Storm API for ease of migration, but beyond that Heron’s > > >>>> architecture > > >>>>>> differs considerably from Storm’s. > > >>>>>> > > >>>>>> Users submit a topology to the scheduler using the Heron client, > > which > > >>>>> uses > > >>>>>> the Heron binary libraries to deploy all daemons required to run > and > > >>>>> manage > > >>>>>> the topology. The topology therefore has no reliance on centrally > > >>>> managed > > >>>>>> Heron services, only on a generic job scheduling system, which > lends > > >>>>> itself > > >>>>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > > >>>> (among > > >>>>>> others). > > >>>>>> > > >>>>>> The scheduler runs each topology as a job consisting of multiple > > >>>>>> containers. One of the containers runs the topology master, > > >> responsible > > >>>>> for > > >>>>>> managing the topology. The remaining containers each runs a stream > > >>>>
Re: [PROPOSAL] Heron
Thanks William! It will be great to work with you again. On Thu, Jun 15, 2017 at 10:12 AM, Supun Kamburugamuve wrote: > Thank you, William, for offering to help with the incubation process. It > will be really helpful. > > Supun.. > > On Wed, Jun 14, 2017 at 11:04 PM, William Markito Oliveira < > william.mark...@gmail.com> wrote: > > > Howdy! > > > > If Heron is looking for some help around incubation process, I'd love to > > help while Geode experience is still fresh in my mind and given that > it's a > > project/space that I do have interest. Since I'm not an ASF member, I > don't > > think I can offer to be a mentor, but can probably still help and > > participate on the process. > > > > Thanks! > > > > On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz > > wrote: > > > > > Hi Bill/Supun, > > > > > > Sorry for not being a little more clear. I was asking more about how > the > > > Heron community would seek to engage with Storm community at the > > > *community* level as opposed to the technical level (i.e. “Community > over > > > Code”). > > > > > > I’ve been asked by many why this has never happened, and have always > > > struggled to answer. Maybe you could help answer that question as well > as > > > if and how that might change if Heron were to incubate. > > > > > > Another quick question: The proposal mentions Heron being used in > > > production at Google, but some Google employees I recently spoke to > > seemed > > > to contradict that. Could you explain? Note that’s nothing that would > > > preclude the project from incubating, I’m just curious. > > > > > > -Taylor > > > > > > > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve > > > wrote: > > > > > > > > Hi Taylor, > > > > > > > > For me, one of the interesting differences between Heron and Storm is > > the > > > > execution model. Storm uses a shared memory model while Heron uses a > > > > process based model. It will be interesting to see how these two > > evolve. > > > > > > > > Thanks, > > > > Supun.. > > > > > > > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham > > > wrote: > > > > > > > >> Hi Taylor, > > > >> > > > >> Thanks for the mentor offer, we'd be glad to have your help. > > > >> > > > >> I think the best place for collaboration would be around the > evolution > > > of > > > >> the API. In addition we plan to look more into DSL solutions which > we > > > could > > > >> potentially collaborate on. This could be Trident, or Beam or > > something > > > >> else, but there could be synergies for future development here. > > > >> > > > >> thanks, > > > >> Bill > > > >> > > > >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz > > > wrote: > > > >> > > > >>> Hi Bill, > > > >>> > > > >>> Could you comment on how/if the Heron community would be willing to > > > work > > > >>> with the Storm community? I've seen a number of new features in > Storm > > > >> being > > > >>> ported to Heron, but I have yet to see any attempt by the Heron > > > community > > > >>> to engage with the Apache Storm community. > > > >>> > > > >>> I don't think it would be too far off to say that the relationship > > > >> between > > > >>> Heron and Apache Storm has been somewhat adversarial. The pre- and > > > >>> post-open sourcing marketing around Heron seemed, at least to me, > > > >> somewhat > > > >>> aggressively negative toward Storm. > > > >>> > > > >>> As a peer to Apache Storm, how would the proposed "Apache Heron" > > > >> community > > > >>> work to collaborate with the Storm community? If Heron is adopting > > API > > > >>> changes in Storm, then it seems there is an opportunity for > > > >> collaboration. > > > >>> > > > >>> Don't take any of this as an objection to incubating the project. I > > > would > > > >>> support it. I would also be willing to b
Re: [PROPOSAL] Heron
API > >>>>> changes in Storm, then it seems there is an opportunity for > >>>> collaboration. > >>>>> > >>>>> Don't take any of this as an objection to incubating the project. I > >> would > >>>>> support it. I would also be willing to be a mentor, if you would > >> consider > >>>>> taking on another. > >>>>> > >>>>> -Taylor > >>>>> > >>>>>> On Jun 8, 2017, at 1:23 PM, Bill Graham > wrote: > >>>>>> > >>>>>> Dear Apache Incubator Community, > >>>>>> > >>>>>> We are excited to share our proposal for discussion and feedback > >>>>>> for entering Apache Incubation. Heron is a real-time, distributed, > >>>>>> fault-tolerant stream processing engine. > >>>>>> > >>>>>> Our proposal can be found at https://wiki.apache.org/ > >>>>> incubator/HeronProposal > >>>>>> and is included below. > >>>>>> > >>>>>> > >>>>>> Thank you, > >>>>>> > >>>>>> Bill Graham on behalf of the Heron developers > >>>>>> > >>>>>> > >>>>>> # Heron Proposal > >>>>>> > >>>>>> ## Abstract > >>>>>> Heron is a real-time, distributed, fault-tolerant stream processing > >>>>> engine > >>>>>> initially developed by Twitter. > >>>>>> > >>>>>> ## Proposal > >>>>>> > >>>>>> Heron is a real-time stream processing engine built for high > >>>> performance, > >>>>>> ease of manageability, performance predictability and developer > >>>>>> productivity[1]. We wish to develop a community around Heron to > >>>> increase > >>>>>> contributions and see Heron thrive in an open forum. > >>>>>> > >>>>>> ## Background > >>>>>> > >>>>>> Heron provides the ability for developers to compose directed > acyclic > >>>>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) > and > >>>>>> submit the topology to execute on a pluggable job scheduling system > >>>>> (e.g., > >>>>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the > >> native > >>>>>> Heron API or the Apache Storm API to develop the topology. Heron > >>>> supports > >>>>>> the Storm API for ease of migration, but beyond that Heron’s > >>>> architecture > >>>>>> differs considerably from Storm’s. > >>>>>> > >>>>>> Users submit a topology to the scheduler using the Heron client, > which > >>>>> uses > >>>>>> the Heron binary libraries to deploy all daemons required to run and > >>>>> manage > >>>>>> the topology. The topology therefore has no reliance on centrally > >>>> managed > >>>>>> Heron services, only on a generic job scheduling system, which lends > >>>>> itself > >>>>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > >>>> (among > >>>>>> others). > >>>>>> > >>>>>> The scheduler runs each topology as a job consisting of multiple > >>>>>> containers. One of the containers runs the topology master, > >> responsible > >>>>> for > >>>>>> managing the topology. The remaining containers each runs a stream > >>>>> manager > >>>>>> responsible for data routing, a metrics manager that collects and > >>>> reports > >>>>>> various metrics and a number of processes called Heron instances > which > >>>>> run > >>>>>> the user-defined logic on the stream of tuples. Parallelism is > >> achieved > >>>>> via > >>>>>> process-based isolation of Heron instances, which provides > predictable > >>>>>> performance while simplifying debugging. The containers are > allocated > >>>> and > >>>>>> m
Re: [PROPOSAL] Heron
Am happy to help too! Thx Debo Sent from my iPhone > On Jun 14, 2017, at 8:05 PM, William Markito Oliveira > wrote: > > Howdy! > > If Heron is looking for some help around incubation process, I'd love to > help while Geode experience is still fresh in my mind and given that it's a > project/space that I do have interest. Since I'm not an ASF member, I don't > think I can offer to be a mentor, but can probably still help and > participate on the process. > > Thanks! > >> On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz wrote: >> >> Hi Bill/Supun, >> >> Sorry for not being a little more clear. I was asking more about how the >> Heron community would seek to engage with Storm community at the >> *community* level as opposed to the technical level (i.e. “Community over >> Code”). >> >> I’ve been asked by many why this has never happened, and have always >> struggled to answer. Maybe you could help answer that question as well as >> if and how that might change if Heron were to incubate. >> >> Another quick question: The proposal mentions Heron being used in >> production at Google, but some Google employees I recently spoke to seemed >> to contradict that. Could you explain? Note that’s nothing that would >> preclude the project from incubating, I’m just curious. >> >> -Taylor >> >>> On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve >> wrote: >>> >>> Hi Taylor, >>> >>> For me, one of the interesting differences between Heron and Storm is the >>> execution model. Storm uses a shared memory model while Heron uses a >>> process based model. It will be interesting to see how these two evolve. >>> >>> Thanks, >>> Supun.. >>> >>> On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham >> wrote: >>> >>>> Hi Taylor, >>>> >>>> Thanks for the mentor offer, we'd be glad to have your help. >>>> >>>> I think the best place for collaboration would be around the evolution >> of >>>> the API. In addition we plan to look more into DSL solutions which we >> could >>>> potentially collaborate on. This could be Trident, or Beam or something >>>> else, but there could be synergies for future development here. >>>> >>>> thanks, >>>> Bill >>>> >>>> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz >> wrote: >>>> >>>>> Hi Bill, >>>>> >>>>> Could you comment on how/if the Heron community would be willing to >> work >>>>> with the Storm community? I've seen a number of new features in Storm >>>> being >>>>> ported to Heron, but I have yet to see any attempt by the Heron >> community >>>>> to engage with the Apache Storm community. >>>>> >>>>> I don't think it would be too far off to say that the relationship >>>> between >>>>> Heron and Apache Storm has been somewhat adversarial. The pre- and >>>>> post-open sourcing marketing around Heron seemed, at least to me, >>>> somewhat >>>>> aggressively negative toward Storm. >>>>> >>>>> As a peer to Apache Storm, how would the proposed "Apache Heron" >>>> community >>>>> work to collaborate with the Storm community? If Heron is adopting API >>>>> changes in Storm, then it seems there is an opportunity for >>>> collaboration. >>>>> >>>>> Don't take any of this as an objection to incubating the project. I >> would >>>>> support it. I would also be willing to be a mentor, if you would >> consider >>>>> taking on another. >>>>> >>>>> -Taylor >>>>> >>>>>> On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: >>>>>> >>>>>> Dear Apache Incubator Community, >>>>>> >>>>>> We are excited to share our proposal for discussion and feedback >>>>>> for entering Apache Incubation. Heron is a real-time, distributed, >>>>>> fault-tolerant stream processing engine. >>>>>> >>>>>> Our proposal can be found at https://wiki.apache.org/ >>>>> incubator/HeronProposal >>>>>> and is included below. >>>>>> >>>>>> >>>>>> Thank you, &
Re: [PROPOSAL] Heron
Thank you, William, for offering to help with the incubation process. It will be really helpful. Supun.. On Wed, Jun 14, 2017 at 11:04 PM, William Markito Oliveira < william.mark...@gmail.com> wrote: > Howdy! > > If Heron is looking for some help around incubation process, I'd love to > help while Geode experience is still fresh in my mind and given that it's a > project/space that I do have interest. Since I'm not an ASF member, I don't > think I can offer to be a mentor, but can probably still help and > participate on the process. > > Thanks! > > On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz > wrote: > > > Hi Bill/Supun, > > > > Sorry for not being a little more clear. I was asking more about how the > > Heron community would seek to engage with Storm community at the > > *community* level as opposed to the technical level (i.e. “Community over > > Code”). > > > > I’ve been asked by many why this has never happened, and have always > > struggled to answer. Maybe you could help answer that question as well as > > if and how that might change if Heron were to incubate. > > > > Another quick question: The proposal mentions Heron being used in > > production at Google, but some Google employees I recently spoke to > seemed > > to contradict that. Could you explain? Note that’s nothing that would > > preclude the project from incubating, I’m just curious. > > > > -Taylor > > > > > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve > > wrote: > > > > > > Hi Taylor, > > > > > > For me, one of the interesting differences between Heron and Storm is > the > > > execution model. Storm uses a shared memory model while Heron uses a > > > process based model. It will be interesting to see how these two > evolve. > > > > > > Thanks, > > > Supun.. > > > > > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham > > wrote: > > > > > >> Hi Taylor, > > >> > > >> Thanks for the mentor offer, we'd be glad to have your help. > > >> > > >> I think the best place for collaboration would be around the evolution > > of > > >> the API. In addition we plan to look more into DSL solutions which we > > could > > >> potentially collaborate on. This could be Trident, or Beam or > something > > >> else, but there could be synergies for future development here. > > >> > > >> thanks, > > >> Bill > > >> > > >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz > > wrote: > > >> > > >>> Hi Bill, > > >>> > > >>> Could you comment on how/if the Heron community would be willing to > > work > > >>> with the Storm community? I've seen a number of new features in Storm > > >> being > > >>> ported to Heron, but I have yet to see any attempt by the Heron > > community > > >>> to engage with the Apache Storm community. > > >>> > > >>> I don't think it would be too far off to say that the relationship > > >> between > > >>> Heron and Apache Storm has been somewhat adversarial. The pre- and > > >>> post-open sourcing marketing around Heron seemed, at least to me, > > >> somewhat > > >>> aggressively negative toward Storm. > > >>> > > >>> As a peer to Apache Storm, how would the proposed "Apache Heron" > > >> community > > >>> work to collaborate with the Storm community? If Heron is adopting > API > > >>> changes in Storm, then it seems there is an opportunity for > > >> collaboration. > > >>> > > >>> Don't take any of this as an objection to incubating the project. I > > would > > >>> support it. I would also be willing to be a mentor, if you would > > consider > > >>> taking on another. > > >>> > > >>> -Taylor > > >>> > > >>>> On Jun 8, 2017, at 1:23 PM, Bill Graham > wrote: > > >>>> > > >>>> Dear Apache Incubator Community, > > >>>> > > >>>> We are excited to share our proposal for discussion and feedback > > >>>> for entering Apache Incubation. Heron is a real-time, distributed, > > >>>> fault-tolerant stream processing engine. > > >>>> > > >>>> Our pro
Re: [PROPOSAL] Heron
Howdy! If Heron is looking for some help around incubation process, I'd love to help while Geode experience is still fresh in my mind and given that it's a project/space that I do have interest. Since I'm not an ASF member, I don't think I can offer to be a mentor, but can probably still help and participate on the process. Thanks! On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz wrote: > Hi Bill/Supun, > > Sorry for not being a little more clear. I was asking more about how the > Heron community would seek to engage with Storm community at the > *community* level as opposed to the technical level (i.e. “Community over > Code”). > > I’ve been asked by many why this has never happened, and have always > struggled to answer. Maybe you could help answer that question as well as > if and how that might change if Heron were to incubate. > > Another quick question: The proposal mentions Heron being used in > production at Google, but some Google employees I recently spoke to seemed > to contradict that. Could you explain? Note that’s nothing that would > preclude the project from incubating, I’m just curious. > > -Taylor > > > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve > wrote: > > > > Hi Taylor, > > > > For me, one of the interesting differences between Heron and Storm is the > > execution model. Storm uses a shared memory model while Heron uses a > > process based model. It will be interesting to see how these two evolve. > > > > Thanks, > > Supun.. > > > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham > wrote: > > > >> Hi Taylor, > >> > >> Thanks for the mentor offer, we'd be glad to have your help. > >> > >> I think the best place for collaboration would be around the evolution > of > >> the API. In addition we plan to look more into DSL solutions which we > could > >> potentially collaborate on. This could be Trident, or Beam or something > >> else, but there could be synergies for future development here. > >> > >> thanks, > >> Bill > >> > >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz > wrote: > >> > >>> Hi Bill, > >>> > >>> Could you comment on how/if the Heron community would be willing to > work > >>> with the Storm community? I've seen a number of new features in Storm > >> being > >>> ported to Heron, but I have yet to see any attempt by the Heron > community > >>> to engage with the Apache Storm community. > >>> > >>> I don't think it would be too far off to say that the relationship > >> between > >>> Heron and Apache Storm has been somewhat adversarial. The pre- and > >>> post-open sourcing marketing around Heron seemed, at least to me, > >> somewhat > >>> aggressively negative toward Storm. > >>> > >>> As a peer to Apache Storm, how would the proposed "Apache Heron" > >> community > >>> work to collaborate with the Storm community? If Heron is adopting API > >>> changes in Storm, then it seems there is an opportunity for > >> collaboration. > >>> > >>> Don't take any of this as an objection to incubating the project. I > would > >>> support it. I would also be willing to be a mentor, if you would > consider > >>> taking on another. > >>> > >>> -Taylor > >>> > >>>> On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: > >>>> > >>>> Dear Apache Incubator Community, > >>>> > >>>> We are excited to share our proposal for discussion and feedback > >>>> for entering Apache Incubation. Heron is a real-time, distributed, > >>>> fault-tolerant stream processing engine. > >>>> > >>>> Our proposal can be found at https://wiki.apache.org/ > >>> incubator/HeronProposal > >>>> and is included below. > >>>> > >>>> > >>>> Thank you, > >>>> > >>>> Bill Graham on behalf of the Heron developers > >>>> > >>>> > >>>> # Heron Proposal > >>>> > >>>> ## Abstract > >>>> Heron is a real-time, distributed, fault-tolerant stream processing > >>> engine > >>>> initially developed by Twitter. > >>>> > >>>> ## Proposal > >>>> > >>>> Heron is a real-time stream processing engine
Re: [PROPOSAL] Heron
Hi Bill/Supun, Sorry for not being a little more clear. I was asking more about how the Heron community would seek to engage with Storm community at the *community* level as opposed to the technical level (i.e. “Community over Code”). I’ve been asked by many why this has never happened, and have always struggled to answer. Maybe you could help answer that question as well as if and how that might change if Heron were to incubate. Another quick question: The proposal mentions Heron being used in production at Google, but some Google employees I recently spoke to seemed to contradict that. Could you explain? Note that’s nothing that would preclude the project from incubating, I’m just curious. -Taylor > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve wrote: > > Hi Taylor, > > For me, one of the interesting differences between Heron and Storm is the > execution model. Storm uses a shared memory model while Heron uses a > process based model. It will be interesting to see how these two evolve. > > Thanks, > Supun.. > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham wrote: > >> Hi Taylor, >> >> Thanks for the mentor offer, we'd be glad to have your help. >> >> I think the best place for collaboration would be around the evolution of >> the API. In addition we plan to look more into DSL solutions which we could >> potentially collaborate on. This could be Trident, or Beam or something >> else, but there could be synergies for future development here. >> >> thanks, >> Bill >> >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz wrote: >> >>> Hi Bill, >>> >>> Could you comment on how/if the Heron community would be willing to work >>> with the Storm community? I've seen a number of new features in Storm >> being >>> ported to Heron, but I have yet to see any attempt by the Heron community >>> to engage with the Apache Storm community. >>> >>> I don't think it would be too far off to say that the relationship >> between >>> Heron and Apache Storm has been somewhat adversarial. The pre- and >>> post-open sourcing marketing around Heron seemed, at least to me, >> somewhat >>> aggressively negative toward Storm. >>> >>> As a peer to Apache Storm, how would the proposed "Apache Heron" >> community >>> work to collaborate with the Storm community? If Heron is adopting API >>> changes in Storm, then it seems there is an opportunity for >> collaboration. >>> >>> Don't take any of this as an objection to incubating the project. I would >>> support it. I would also be willing to be a mentor, if you would consider >>> taking on another. >>> >>> -Taylor >>> >>>> On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: >>>> >>>> Dear Apache Incubator Community, >>>> >>>> We are excited to share our proposal for discussion and feedback >>>> for entering Apache Incubation. Heron is a real-time, distributed, >>>> fault-tolerant stream processing engine. >>>> >>>> Our proposal can be found at https://wiki.apache.org/ >>> incubator/HeronProposal >>>> and is included below. >>>> >>>> >>>> Thank you, >>>> >>>> Bill Graham on behalf of the Heron developers >>>> >>>> >>>> # Heron Proposal >>>> >>>> ## Abstract >>>> Heron is a real-time, distributed, fault-tolerant stream processing >>> engine >>>> initially developed by Twitter. >>>> >>>> ## Proposal >>>> >>>> Heron is a real-time stream processing engine built for high >> performance, >>>> ease of manageability, performance predictability and developer >>>> productivity[1]. We wish to develop a community around Heron to >> increase >>>> contributions and see Heron thrive in an open forum. >>>> >>>> ## Background >>>> >>>> Heron provides the ability for developers to compose directed acyclic >>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and >>>> submit the topology to execute on a pluggable job scheduling system >>> (e.g., >>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native >>>> Heron API or the Apache Storm API to develop the topology. Heron >> supports >>>> the Storm API for ease of migration, but beyond that Heron’s >> archit
Re: [PROPOSAL] Heron
Hi Taylor, For me, one of the interesting differences between Heron and Storm is the execution model. Storm uses a shared memory model while Heron uses a process based model. It will be interesting to see how these two evolve. Thanks, Supun.. On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham wrote: > Hi Taylor, > > Thanks for the mentor offer, we'd be glad to have your help. > > I think the best place for collaboration would be around the evolution of > the API. In addition we plan to look more into DSL solutions which we could > potentially collaborate on. This could be Trident, or Beam or something > else, but there could be synergies for future development here. > > thanks, > Bill > > On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz wrote: > > > Hi Bill, > > > > Could you comment on how/if the Heron community would be willing to work > > with the Storm community? I've seen a number of new features in Storm > being > > ported to Heron, but I have yet to see any attempt by the Heron community > > to engage with the Apache Storm community. > > > > I don't think it would be too far off to say that the relationship > between > > Heron and Apache Storm has been somewhat adversarial. The pre- and > > post-open sourcing marketing around Heron seemed, at least to me, > somewhat > > aggressively negative toward Storm. > > > > As a peer to Apache Storm, how would the proposed "Apache Heron" > community > > work to collaborate with the Storm community? If Heron is adopting API > > changes in Storm, then it seems there is an opportunity for > collaboration. > > > > Don't take any of this as an objection to incubating the project. I would > > support it. I would also be willing to be a mentor, if you would consider > > taking on another. > > > > -Taylor > > > > > On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: > > > > > > Dear Apache Incubator Community, > > > > > > We are excited to share our proposal for discussion and feedback > > > for entering Apache Incubation. Heron is a real-time, distributed, > > > fault-tolerant stream processing engine. > > > > > > Our proposal can be found at https://wiki.apache.org/ > > incubator/HeronProposal > > > and is included below. > > > > > > > > > Thank you, > > > > > > Bill Graham on behalf of the Heron developers > > > > > > > > > # Heron Proposal > > > > > > ## Abstract > > > Heron is a real-time, distributed, fault-tolerant stream processing > > engine > > > initially developed by Twitter. > > > > > > ## Proposal > > > > > > Heron is a real-time stream processing engine built for high > performance, > > > ease of manageability, performance predictability and developer > > > productivity[1]. We wish to develop a community around Heron to > increase > > > contributions and see Heron thrive in an open forum. > > > > > > ## Background > > > > > > Heron provides the ability for developers to compose directed acyclic > > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > > > submit the topology to execute on a pluggable job scheduling system > > (e.g., > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > > > Heron API or the Apache Storm API to develop the topology. Heron > supports > > > the Storm API for ease of migration, but beyond that Heron’s > architecture > > > differs considerably from Storm’s. > > > > > > Users submit a topology to the scheduler using the Heron client, which > > uses > > > the Heron binary libraries to deploy all daemons required to run and > > manage > > > the topology. The topology therefore has no reliance on centrally > managed > > > Heron services, only on a generic job scheduling system, which lends > > itself > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > (among > > > others). > > > > > > The scheduler runs each topology as a job consisting of multiple > > > containers. One of the containers runs the topology master, responsible > > for > > > managing the topology. The remaining containers each runs a stream > > manager > > > responsible for data routing, a metrics manager that collects and > reports > > > various metrics and a number of processes called Heron instances which > > run > > > the user-defined
Re: [PROPOSAL] Heron
Hi Taylor, Thanks for the mentor offer, we'd be glad to have your help. I think the best place for collaboration would be around the evolution of the API. In addition we plan to look more into DSL solutions which we could potentially collaborate on. This could be Trident, or Beam or something else, but there could be synergies for future development here. thanks, Bill On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz wrote: > Hi Bill, > > Could you comment on how/if the Heron community would be willing to work > with the Storm community? I've seen a number of new features in Storm being > ported to Heron, but I have yet to see any attempt by the Heron community > to engage with the Apache Storm community. > > I don't think it would be too far off to say that the relationship between > Heron and Apache Storm has been somewhat adversarial. The pre- and > post-open sourcing marketing around Heron seemed, at least to me, somewhat > aggressively negative toward Storm. > > As a peer to Apache Storm, how would the proposed "Apache Heron" community > work to collaborate with the Storm community? If Heron is adopting API > changes in Storm, then it seems there is an opportunity for collaboration. > > Don't take any of this as an objection to incubating the project. I would > support it. I would also be willing to be a mentor, if you would consider > taking on another. > > -Taylor > > > On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: > > > > Dear Apache Incubator Community, > > > > We are excited to share our proposal for discussion and feedback > > for entering Apache Incubation. Heron is a real-time, distributed, > > fault-tolerant stream processing engine. > > > > Our proposal can be found at https://wiki.apache.org/ > incubator/HeronProposal > > and is included below. > > > > > > Thank you, > > > > Bill Graham on behalf of the Heron developers > > > > > > # Heron Proposal > > > > ## Abstract > > Heron is a real-time, distributed, fault-tolerant stream processing > engine > > initially developed by Twitter. > > > > ## Proposal > > > > Heron is a real-time stream processing engine built for high performance, > > ease of manageability, performance predictability and developer > > productivity[1]. We wish to develop a community around Heron to increase > > contributions and see Heron thrive in an open forum. > > > > ## Background > > > > Heron provides the ability for developers to compose directed acyclic > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > > submit the topology to execute on a pluggable job scheduling system > (e.g., > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > > Heron API or the Apache Storm API to develop the topology. Heron supports > > the Storm API for ease of migration, but beyond that Heron’s architecture > > differs considerably from Storm’s. > > > > Users submit a topology to the scheduler using the Heron client, which > uses > > the Heron binary libraries to deploy all daemons required to run and > manage > > the topology. The topology therefore has no reliance on centrally managed > > Heron services, only on a generic job scheduling system, which lends > itself > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > > others). > > > > The scheduler runs each topology as a job consisting of multiple > > containers. One of the containers runs the topology master, responsible > for > > managing the topology. The remaining containers each runs a stream > manager > > responsible for data routing, a metrics manager that collects and reports > > various metrics and a number of processes called Heron instances which > run > > the user-defined logic on the stream of tuples. Parallelism is achieved > via > > process-based isolation of Heron instances, which provides predictable > > performance while simplifying debugging. The containers are allocated and > > managed by the scheduler framework based on resource availability of > nodes > > in the cluster. The metadata for the topology, such as the physical plan > > and execution details, are stored in the pluggable Heron State Manager > > (e.g. Apache ZooKeeper). > > > > ## Rationale > > > > Heron is a general-purpose, modular and extensible platform that can be > > leveraged to support common, real-time analytics use cases. There is an > > increasing demand for open-source, scalable real-time analytics systems. > We > > believe that Heron c
Re: [PROPOSAL] Heron
Hi Bill, Could you comment on how/if the Heron community would be willing to work with the Storm community? I've seen a number of new features in Storm being ported to Heron, but I have yet to see any attempt by the Heron community to engage with the Apache Storm community. I don't think it would be too far off to say that the relationship between Heron and Apache Storm has been somewhat adversarial. The pre- and post-open sourcing marketing around Heron seemed, at least to me, somewhat aggressively negative toward Storm. As a peer to Apache Storm, how would the proposed "Apache Heron" community work to collaborate with the Storm community? If Heron is adopting API changes in Storm, then it seems there is an opportunity for collaboration. Don't take any of this as an objection to incubating the project. I would support it. I would also be willing to be a mentor, if you would consider taking on another. -Taylor > On Jun 8, 2017, at 1:23 PM, Bill Graham wrote: > > Dear Apache Incubator Community, > > We are excited to share our proposal for discussion and feedback > for entering Apache Incubation. Heron is a real-time, distributed, > fault-tolerant stream processing engine. > > Our proposal can be found at https://wiki.apache.org/incubator/HeronProposal > and is included below. > > > Thank you, > > Bill Graham on behalf of the Heron developers > > > # Heron Proposal > > ## Abstract > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > ## Proposal > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > ## Background > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > ## Rationale > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > ## Initial Goals > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with
Re: [PROPOSAL] Heron
Thanks Supun. The main commonality with Storm is that Heron implements the Storm user-facing API for developing topologies. Besides that the architecture is completely different (no central services, full isolation at the process, task and topology levels, pluggable scheduler, etc). The key unique elements of the Heron architecture are described in paragraphs 2 and 3 of Background section. On Fri, Jun 9, 2017 at 12:17 PM, Supun Kamburugamuva wrote: > Thanks Bill for bringing up the proposal. Should there be a sentence or > two describing how Heron compares with Storm? > > Regards, > Supun.. > > On Thu, Jun 8, 2017 at 1:23 PM, Bill Graham wrote: > >> Dear Apache Incubator Community, >> >> We are excited to share our proposal for discussion and feedback >> for entering Apache Incubation. Heron is a real-time, distributed, >> fault-tolerant stream processing engine. >> >> Our proposal can be found at https://wiki.apache.org/incuba >> tor/HeronProposal >> and is included below. >> >> >> Thank you, >> >> Bill Graham on behalf of the Heron developers >> >> >> # Heron Proposal >> >> ## Abstract >> Heron is a real-time, distributed, fault-tolerant stream processing engine >> initially developed by Twitter. >> >> ## Proposal >> >> Heron is a real-time stream processing engine built for high performance, >> ease of manageability, performance predictability and developer >> productivity[1]. We wish to develop a community around Heron to increase >> contributions and see Heron thrive in an open forum. >> >> ## Background >> >> Heron provides the ability for developers to compose directed acyclic >> graphs (DAGs) of real-time query execution logic (i.e. a topology) and >> submit the topology to execute on a pluggable job scheduling system (e.g., >> Apache Aurora, YARN, Marathon, etc). Users can employ either the native >> Heron API or the Apache Storm API to develop the topology. Heron supports >> the Storm API for ease of migration, but beyond that Heron’s architecture >> differs considerably from Storm’s. >> >> Users submit a topology to the scheduler using the Heron client, which >> uses >> the Heron binary libraries to deploy all daemons required to run and >> manage >> the topology. The topology therefore has no reliance on centrally managed >> Heron services, only on a generic job scheduling system, which lends >> itself >> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among >> others). >> >> The scheduler runs each topology as a job consisting of multiple >> containers. One of the containers runs the topology master, responsible >> for >> managing the topology. The remaining containers each runs a stream manager >> responsible for data routing, a metrics manager that collects and reports >> various metrics and a number of processes called Heron instances which run >> the user-defined logic on the stream of tuples. Parallelism is achieved >> via >> process-based isolation of Heron instances, which provides predictable >> performance while simplifying debugging. The containers are allocated and >> managed by the scheduler framework based on resource availability of nodes >> in the cluster. The metadata for the topology, such as the physical plan >> and execution details, are stored in the pluggable Heron State Manager >> (e.g. Apache ZooKeeper). >> >> ## Rationale >> >> Heron is a general-purpose, modular and extensible platform that can be >> leveraged to support common, real-time analytics use cases. There is an >> increasing demand for open-source, scalable real-time analytics systems. >> We >> believe that Heron can be leveraged by other organizations to build >> streaming applications that can benefit from its robustness, high >> performance, adaptability to cloud environments and ease of use. Moreover, >> we hope that open-sourcing Heron will help to further evolve the >> technology >> as the project attracts contributors with diverse backgrounds and areas of >> expertise. >> >> We believe the Apache foundation is a great fit as the long-term home for >> Heron, as it provides an established process for community-driven >> development and decision making by consensus. This is exactly the model we >> want for future Heron development. >> >> ## Initial Goals >> >> * Move the existing codebase, website, documentation, and mailing lists to >> Apache-hosted infrastructure. >> * Integrate with the Apache development process. >> * Ensure al
Re: [PROPOSAL] Heron
Thanks Bill for bringing up the proposal. Should there be a sentence or two describing how Heron compares with Storm? Regards, Supun.. On Thu, Jun 8, 2017 at 1:23 PM, Bill Graham wrote: > Dear Apache Incubator Community, > > We are excited to share our proposal for discussion and feedback > for entering Apache Incubation. Heron is a real-time, distributed, > fault-tolerant stream processing engine. > > Our proposal can be found at https://wiki.apache.org/ > incubator/HeronProposal > and is included below. > > > Thank you, > > Bill Graham on behalf of the Heron developers > > > # Heron Proposal > > ## Abstract > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > ## Proposal > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > ## Background > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > ## Rationale > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > ## Initial Goals > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > ## Current Status > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > ### Meritocracy > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging o
[PROPOSAL] Heron
Dear Apache Incubator Community, We are excited to share our proposal for discussion and feedback for entering Apache Incubation. Heron is a real-time, distributed, fault-tolerant stream processing engine. Our proposal can be found at https://wiki.apache.org/incubator/HeronProposal and is included below. Thank you, Bill Graham on behalf of the Heron developers # Heron Proposal ## Abstract Heron is a real-time, distributed, fault-tolerant stream processing engine initially developed by Twitter. ## Proposal Heron is a real-time stream processing engine built for high performance, ease of manageability, performance predictability and developer productivity[1]. We wish to develop a community around Heron to increase contributions and see Heron thrive in an open forum. ## Background Heron provides the ability for developers to compose directed acyclic graphs (DAGs) of real-time query execution logic (i.e. a topology) and submit the topology to execute on a pluggable job scheduling system (e.g., Apache Aurora, YARN, Marathon, etc). Users can employ either the native Heron API or the Apache Storm API to develop the topology. Heron supports the Storm API for ease of migration, but beyond that Heron’s architecture differs considerably from Storm’s. Users submit a topology to the scheduler using the Heron client, which uses the Heron binary libraries to deploy all daemons required to run and manage the topology. The topology therefore has no reliance on centrally managed Heron services, only on a generic job scheduling system, which lends itself well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among others). The scheduler runs each topology as a job consisting of multiple containers. One of the containers runs the topology master, responsible for managing the topology. The remaining containers each runs a stream manager responsible for data routing, a metrics manager that collects and reports various metrics and a number of processes called Heron instances which run the user-defined logic on the stream of tuples. Parallelism is achieved via process-based isolation of Heron instances, which provides predictable performance while simplifying debugging. The containers are allocated and managed by the scheduler framework based on resource availability of nodes in the cluster. The metadata for the topology, such as the physical plan and execution details, are stored in the pluggable Heron State Manager (e.g. Apache ZooKeeper). ## Rationale Heron is a general-purpose, modular and extensible platform that can be leveraged to support common, real-time analytics use cases. There is an increasing demand for open-source, scalable real-time analytics systems. We believe that Heron can be leveraged by other organizations to build streaming applications that can benefit from its robustness, high performance, adaptability to cloud environments and ease of use. Moreover, we hope that open-sourcing Heron will help to further evolve the technology as the project attracts contributors with diverse backgrounds and areas of expertise. We believe the Apache foundation is a great fit as the long-term home for Heron, as it provides an established process for community-driven development and decision making by consensus. This is exactly the model we want for future Heron development. ## Initial Goals * Move the existing codebase, website, documentation, and mailing lists to Apache-hosted infrastructure. * Integrate with the Apache development process. * Ensure all dependencies are compliant with Apache License version 2.0. * Incrementally develop and release per Apache guidelines. ## Current Status Heron is a stable project used in production at Twitter since 2014 and open sourced under the ASL v2 license in 2016. The Heron source code is currently hosted at github.com (https://github.com/twitter/heron), which will seed the Apache git repository. ### Meritocracy By submitting this incubator proposal, we’re expressing our intent to build a diverse developer community around Heron that will conduct itself according to The Apache Way and use a meritocratic means of building it's committer base. Several companies and universities have already expressed interest in and contributed to Heron. Our goal is to grow the Heron community by encouraging open communication, contribution and participation of all types, and ensuring that contributors are recognized appropriately. ### Community Heron is currently being used by Twitter, Google, Machine Zone and ndustrial.io and has received significant contributions by Microsoft and Streamlio. By bringing Heron into the Apache ecosystem, we believe we can attract even more developers who are interested in creating real-time systems to build the project's contributor base. ### Core Developers Current core developers are engineers from Twitter, Google, Microsoft and Streamlio. ### Alignment Heron utilizes a number of Apache technologies. Heron levera
