Hi - > On Jan 15, 2019, at 11:35 PM, Ning Wang <[email protected]> wrote: > > Hi, all, > > A few of us (Spencer, Saikat, Siming, Karthik, Josh, Sree) discussed today > in our general slack channel that we should have spouts code somewhere so > that people can reuse them (spouts are highly reusable in general) and > contribute improvements. This is just a recap of the idea and some updates. > > We have two options: > 1. add a spouts/ dir in heron project. > 2. create a new project in github. > > For option 1, it is easy to start. But the iteration and release will be > coupled with Heron project itself. It is likely there will be quite some > activities around spouts time by time when new spouts are added. Also, > Heron itself is basically the engine itself plus APIs and tooling, while > there could be quite some spouts in future with many new dependencies like > Kafka, pubsub, neo4j and neptune, etc. It is debatable to have spout > implementations in Heron project, and these extra dependencies could add > some unnecessary complexity. > > For option 2, there will be some work up front. but it will be much easier > to manage and evolve. And here will be less concerns about new spouts (in > different languages) and dependencies because spouts are relatively > independent to each other and we may generate artifacts per spout. > > Overall most people prefer option 2 for its cleanness. > > I talked with Twitter OSS team. They are happy to support the initiative > and suggest us to check with Apache team and see what is the best process. > First question is that should this new side project be under Apache or not? > This might be a question to mentors. What do you think/suggest?
The Heron PPMC is an Apache Team. The choice should be this communities. (1) It is ok for an Apache Project to have multiple products and repos. Examples that quickly come to mind include: - Apache Lucene with both Lucene and Solr. - Apache POI which recently took control of the Atticed XMLBean project’s XMLBean product. - Apache Sling has tens of repositories since it releases its features as a large set of OSGi bundles. Have a look at Sling which seems to have a model that would be similar to that contemplated for “Spouts”. (2) It is ok to host a new repos in Twitter’s or Ning's Github accounts. If you wanted to then bring these back to the Heron (or other project) there would then need to be an IP Clearance process. If this is a new project that you wanted to Incubate then the group could reenter the Incubator. That said if you aspire to be an Apache project with “Spouts” then you are already here. (3) It is ok for the project to change direction and decide to be about “Spouts”, “Bolts” and “IO Connectors”. This seems like an important discussion for the future of Heron and I am glad that you brought it to the list. I’d like to read what the community thinks about the choices. Regards, Dave > > Another topic being discussed is the build tool in case we decide to create > a new side project. Maven is more mature for sure, but we will likely need > multi language support so currently Bazel seems to be the winner (I > personally vote for Bazel 1.0 because the backward compatibility has been > bad so far). > > Any ideas or suggestions, please feel free to reply. > > Regards, > --ning
