This is a good question. Each version spout must maintain a compatibility
matrix {Spout Version, external SDK version, Heron API version}. It’s more
of a documentation effort so that user haves enough information to
determine which one to pick, isn’t it?On Thu, Jan 17, 2019 at 7:48 AM Josh Fischer <[email protected]> wrote: > If we were to go with a separate repo for the spouts how would we version > it? Would it be consistent with the Heron repo? How would people know > what version spout to use with the Heron version they are running? > > > On Thu, Jan 17, 2019 at 1:26 AM Ning Wang <[email protected]> wrote: > >> This is an option. I have a few concerns about it: >> - There will be a lot of repos and it will be messy to manage and it might >> be harder for users to find it. I am expecting at least more than ten >> (different services times different languages). >> - There will be some duplicated code such as build/release configs, >> scripts. etc. >> >> I think we should be able to achieve the first reason with a single repo. >> Different spouts should likely be in different folders and they can evolve >> separately. >> The second reason is valid, but duplicated code is a side effect. >> The third reason depends on building tool I feel. Bazel is powerful, but >> it >> is just changing time by time. :( >> >> Just my two cents. >> >> >> >> >> >> On Wed, Jan 16, 2019 at 8:09 PM Simon Weng <[email protected]> wrote: >> >> > Hi, all: >> > >> > Can it also be one of the options to even have separate repo for each >> type >> > of spouts? The reasons it is worth considering are: >> > >> > 1. Allow each spout to evolve and release in different pace because each >> > is technically driven by external source software. For example, the >> > community may need different versions of the Kafka Spout to be >> compatible >> > with their deployed Kafka cluster in production >> > 2. Allow each spout project to use the de facto build tool that suits >> the >> > external SDK best. This will help to minimize the learning curve for >> > constributors who specialize in different source software stack >> > 3. Simply the maintainence of the build and CI >> > >> > I’m not familiar with the capability of Bazel, so certainly I’m not >> > against it. If it can help to achieve some of the above, I guess one >> single >> > repo will also work then. >> > >> > SiMing >> > >> > On Wed, Jan 16, 2019 at 5:34 PM Ning Wang <[email protected]> wrote: >> > >> >> +Siming >> >> >> >> On Tue, Jan 15, 2019 at 11:35 PM Ning Wang <[email protected]> >> wrote: >> >> >> >>> Hi, all, >> >>> >> >>> A few of us (Spencer, Saikat, Siming, Karthik, Josh, Sree) discussed >> >>> today in our general slack channel that we should have spouts code >> >>> somewhere so that people can reuse them (spouts are highly reusable in >> >>> general) and contribute improvements. This is just a recap of the >> idea and >> >>> some updates. >> >>> >> >>> We have two options: >> >>> 1. add a spouts/ dir in heron project. >> >>> 2. create a new project in github. >> >>> >> >>> For option 1, it is easy to start. But the iteration and release will >> be >> >>> coupled with Heron project itself. It is likely there will be quite >> some >> >>> activities around spouts time by time when new spouts are added. Also, >> >>> Heron itself is basically the engine itself plus APIs and tooling, >> while >> >>> there could be quite some spouts in future with many new dependencies >> like >> >>> Kafka, pubsub, neo4j and neptune, etc. It is debatable to have spout >> >>> implementations in Heron project, and these extra dependencies could >> add >> >>> some unnecessary complexity. >> >>> >> >>> For option 2, there will be some work up front. but it will be much >> >>> easier to manage and evolve. And here will be less concerns about new >> >>> spouts (in different languages) and dependencies because spouts are >> >>> relatively independent to each other and we may generate artifacts per >> >>> spout. >> >>> >> >>> Overall most people prefer option 2 for its cleanness. >> >>> >> >>> I talked with Twitter OSS team. They are happy to support the >> initiative >> >>> and suggest us to check with Apache team and see what is the best >> process. >> >>> First question is that should this new side project be under Apache >> or not? >> >>> This might be a question to mentors. What do you think/suggest? >> >>> >> >>> Another topic being discussed is the build tool in case we decide to >> >>> create a new side project. Maven is more mature for sure, but we will >> >>> likely need multi language support so currently Bazel seems to be the >> >>> winner (I personally vote for Bazel 1.0 because the backward >> compatibility >> >>> has been bad so far). >> >>> >> >>> Any ideas or suggestions, please feel free to reply. >> >>> >> >>> Regards, >> >>> --ning >> >>> >> >> -- >> > Sent from Gmail Mobile >> > >> > -- > Sent from A Mobile Device > -- Sent from Gmail Mobile
