I don't think storm-kafka-client needs its own project, but I like the idea of decoupling storm-kafka-client from the release cadence of Storm. Primarily because it would allow us to release more frequently, but also because I think most of the time there's no tight coupling between the Storm minor/patch version and storm-kafka-client. As far as I know storm-kafka-client 1.2.0 runs fine on other Storm 1.x versions. If storm-kafka-client released independently, we could probably get away with maintaining only 1.x and 2.x compatible versions. I also see some value in being able to do breaking changes independently of Storm's major version, without breaking semantic versioning. People clearly expect us to not introduce breaking changes in minor/patch releases (e.g. Alexandre when storm-kafka-client 1.2.0 wouldn't run with his 1.1.1 topology a while back), but we've had to do that a few times to avoid postponing fixes until Storm 2.0.0. Releasing independently would also address Erik's concern that we're releasing a "new" storm-kafka-client 1.1.2 that has known issues.
If we want to decouple storm-kafka-client's release cycle from Storm, I don't think we can keep it in the Storm repo. It would get confusing with branches and release tags. We might try to decouple it in the same way Maven have decoupled their plugins from the main Maven repo, where the code for a given plugin is in a separate repository, but the plugin is still part of the Maven project and uses the Maven mailing list for discussion and release announcements. My main concern for moving storm-kafka-client to another repository would be how we build against unreleased Storm versions, e.g. 2.0.0. I'm wondering if anyone has ideas for this? It looks to me like both the Maven plugins and Bahir are building against only released versions. @Hugo I think at least a few of your points about storm-kafka-client's implied beta status would have been easier to handle if storm-kafka-client were not coupled to the Storm release version. It was weird to have a new connector's initial release be version 1.0.0, and following Storm's release cycle has prevented us from releasing fixes as often as a new connector will likely need. I think we could have also saved ourselves a bit of work by releasing 0.x versions first, since we then wouldn't have had to worry about backward compatibility when doing major API changes like https://issues.apache.org/jira/browse/STORM-2548. Regarding whether it is a concern when we make breaking changes in storm-kafka-client if it is due to Kafka breaking something: Kafka was free to include breaking changes, because they were still in the pre-1.0.0 version range which tells the user to expect breaking changes from time to time. When we release breaking changes in a 1.0.y version, it defeats the purpose of semantic versioning, which is to tell the user upgrading whether an upgrade can be done freely, or whether they should expect to have to recompile and maybe reserve some time to upgrade. 2018-01-25 23:44 GMT+01:00 Jungtaek Lim <[email protected]>: > Hugo, > > My idea is basically came from Apache bahir. (http://bahir.apache.org/) It > was for Apache Spark, but Flink decided to migrate their connectors to > bahir so it is also for Apache Flink. > I admit we may want to keep some connectors in core even we split out > connectors, since moving to another repo. makes connectors hard to be in > sync with core version, and even has a chance to being forgotten. Kafka > connector is first class connector, so maybe storm-kafka-client would not > take this way even we have similar, but we could incubate it and bring to > core once it's stabilized (like storm-kafka-client in 1.2.0). > > I don't think storm-kafka-client has been technically beta. It might be > considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months ago. > storm-kafka-client is introduced over a year being included as Storm 1.0.0 > implicitly announcing we are no longer beta. Explicitly marking as beta is > very important in practice and that's why InterfaceStability annotation > came in and widely used for big project. (We have started to apply it for > streams API in 2.0.0: they're marked as @Unstable.) > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <[email protected]>님이 > 작성: > > > I am in favor of not incubating storm-kafka-client but rather keep it as > > part of the storm project. We can consider supporting a separate branch, > > but before we agree to go that route I would like to hear lessons learned > > from community members that have been part of similar transitions in > other > > projects. > > > > As for not back-porting all the storm-kafka-client changes onto 1.0.x and > > 1.1.x branches, I expressed my opinion in the JIRA when the this > discussion > > first came up. Basically, I don’t think it is feasible to back-port > > everything. Furthermore, 1.2.0 will be a minor version for which it is > > reasonable to expect users of earlier versions to upgrade to without > major > > hassle. In any software it is expected that there will be divergence > across > > minor versions. New versions become available because they have > > improvements and and sometimes new, small, features in it. If the user > > wants to benefit from those improvements, she/he will have to upgrade to > > the most recent version. Compatibility should be accounted for across > > versions, but as Stig mentioned, for the most part we believe it has > been. > > > > Although it is possible to copy the 1.x storm-kafka-client changes to > > 1.1.x and 1.0.x, that same argument could be made for every other > connector > > or storm component, and I am not sure it’s a good principle. I just think > > that it’s natural and beneficial to the community and the product that > > users keep upgrading (especially across backwards compatible versions), > > rather that keeping on patching things up. > > > > A few more opinions on storm-kafka-client > > - Can we identify the pros and cons of keeping Kafka 0.9 dependencies, > and > > unless there are very strong reasons not to do so, simply upgrade to a > more > > recent version. > > - Although storm-kafka-client was not labeled as beta, technically it was > > beta, and in practice it was used by users as beta. As far as I know, > users > > that used it from the beginning were still exploring it, and I don’t > think > > that until a lot of the improvements came in it made it to production. So > > Beta is a nice label to have behind a component that can cover for some > > necessary non backwards compatible changes. However, I don’t think it > would > > have made a world of difference in practice for storm-kafka-client. > > - The Kaka community has at times made non backward compatible changes. > If > > Kafka itself makes such changes, is it that concerning if > > storm-kafka-client has to make or or two such non backward compatible > > changes? > > > > Thanks, > > Hugo > > > > On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <[email protected]<mailto: > > [email protected]>> wrote: > > > > > > > > On Jan 25, 2018, at 12:32 PM, Bobby Evans <[email protected]< > mailto: > > [email protected]>> wrote: > > > > To make this happen though someone is going to need to step up and take > > lead on this. > > > > We will need a plan on what to do (is it becoming a separate repo with a > > separate release cycle but still a part of the storm project?) > > Do we plan to spin it off to be an incubator project on its own? > > > > I don’t think it’s necessary to spin it off as a separate incubator > > project, that seems like overkill and would be weird from a community > > perspective. > > > > The path of least resistance might be a separate git repo, or just > keeping > > it in the current Storm repo and decoupling it from the main > build/release > > process so it can be independently released. > > > > -Taylor > > > > > > >
