Hugo, My idea is basically came from Apache bahir. (http://bahir.apache.org/) It was for Apache Spark, but Flink decided to migrate their connectors to bahir so it is also for Apache Flink. I admit we may want to keep some connectors in core even we split out connectors, since moving to another repo. makes connectors hard to be in sync with core version, and even has a chance to being forgotten. Kafka connector is first class connector, so maybe storm-kafka-client would not take this way even we have similar, but we could incubate it and bring to core once it's stabilized (like storm-kafka-client in 1.2.0).
I don't think storm-kafka-client has been technically beta. It might be considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months ago. storm-kafka-client is introduced over a year being included as Storm 1.0.0 implicitly announcing we are no longer beta. Explicitly marking as beta is very important in practice and that's why InterfaceStability annotation came in and widely used for big project. (We have started to apply it for streams API in 2.0.0: they're marked as @Unstable.) Thanks, Jungtaek Lim (HeartSaVioR) 2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 작성: > I am in favor of not incubating storm-kafka-client but rather keep it as > part of the storm project. We can consider supporting a separate branch, > but before we agree to go that route I would like to hear lessons learned > from community members that have been part of similar transitions in other > projects. > > As for not back-porting all the storm-kafka-client changes onto 1.0.x and > 1.1.x branches, I expressed my opinion in the JIRA when the this discussion > first came up. Basically, I don’t think it is feasible to back-port > everything. Furthermore, 1.2.0 will be a minor version for which it is > reasonable to expect users of earlier versions to upgrade to without major > hassle. In any software it is expected that there will be divergence across > minor versions. New versions become available because they have > improvements and and sometimes new, small, features in it. If the user > wants to benefit from those improvements, she/he will have to upgrade to > the most recent version. Compatibility should be accounted for across > versions, but as Stig mentioned, for the most part we believe it has been. > > Although it is possible to copy the 1.x storm-kafka-client changes to > 1.1.x and 1.0.x, that same argument could be made for every other connector > or storm component, and I am not sure it’s a good principle. I just think > that it’s natural and beneficial to the community and the product that > users keep upgrading (especially across backwards compatible versions), > rather that keeping on patching things up. > > A few more opinions on storm-kafka-client > - Can we identify the pros and cons of keeping Kafka 0.9 dependencies, and > unless there are very strong reasons not to do so, simply upgrade to a more > recent version. > - Although storm-kafka-client was not labeled as beta, technically it was > beta, and in practice it was used by users as beta. As far as I know, users > that used it from the beginning were still exploring it, and I don’t think > that until a lot of the improvements came in it made it to production. So > Beta is a nice label to have behind a component that can cover for some > necessary non backwards compatible changes. However, I don’t think it would > have made a world of difference in practice for storm-kafka-client. > - The Kaka community has at times made non backward compatible changes. If > Kafka itself makes such changes, is it that concerning if > storm-kafka-client has to make or or two such non backward compatible > changes? > > Thanks, > Hugo > > On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <ptgo...@gmail.com<mailto: > ptgo...@gmail.com>> wrote: > > > > On Jan 25, 2018, at 12:32 PM, Bobby Evans <ev...@oath.com.INVALID<mailto: > ev...@oath.com.INVALID>> wrote: > > To make this happen though someone is going to need to step up and take > lead on this. > > We will need a plan on what to do (is it becoming a separate repo with a > separate release cycle but still a part of the storm project?) > Do we plan to spin it off to be an incubator project on its own? > > I don’t think it’s necessary to spin it off as a separate incubator > project, that seems like overkill and would be weird from a community > perspective. > > The path of least resistance might be a separate git repo, or just keeping > it in the current Storm repo and decoupling it from the main build/release > process so it can be independently released. > > -Taylor > > >
