To give some background information, the Spark PMC decided to remove a number of connectors from their repo (and thus releases). Some members of the community wanted to see some sort of official community support for those connectors, thus the Apache Bahir project was created. Flink also decided to follow that model.
I don’t feel we’ve reached the same conclusion (to remove connectors from our distribution), or, perhaps, not yet. I think where we are right now is wanting to decouple storm-kafka-client from Storm’s release cycle so updates can be released more often. As I mentioned in the vote thread, there are two approaches we could take: 1. Move storm-kafka-client to a new git repo under the purview of the Storm PMC. 2. Leave storm-kafka-client in place, but decouple it from the main build/release process so it can be released independently of storm proper. I lean toward option 2. One downside as Stig pointed out is that it would make tagging awkward, but I don’t see that as much of a problem — we could simply keep the top-level tag convention “vX.X.X” and add another along the lines of “storm-kafka-client-vX.X.X”. Yes, it adds another tag/tag convention, but even if we moved all the connectors to another repo and versioned them all independently we would have to do the same in that repo. If we start with option 2, it would make it easier to rollback if for some reason we later decided it was’t a good idea. It could also represent a “try before you buy” option toward moving to option 1. The argument I see for option 1 would be a better IDE experience — i.e. you would have two separate IDE projects that might make development/testing with various versions easier. I’m open to either approach and would volunteer to do the work to make it happen. -Taylor > On Jan 29, 2018, at 12:45 AM, Jungtaek Lim <kabh...@gmail.com> wrote: > > My idea is basically came from Apache Bahir. (http://bahir.apache.org/) It > was for Apache Spark, but Flink decided to migrate their connectors to > Bahir so it is also for Apache Flink. They're also maintaining some > connectors (I'd say first class support) in their repositories, but not > all. I think we could select some of connectors to support as first class, > and move out others to Bahir or another storm repository (storm-connectors? > storm-externals?). > > - Jungtaek Lim (HeartSavioR) > > 2018년 1월 29일 (월) 오후 2:30, Jungtaek Lim <kabh...@gmail.com>님이 작성: > >> Hi devs, >> >> This is initial post to separate out discussion topic from vote thread, >> and continue discussing. >> >> Background of the topic: >> 1. Releasing Storm requires huge bootstrapping, and normally takes several >> months to release bugfix version. Note that it is not minor version... >> Minor version is released per near a year. Connectors are maintained with >> same release cadence, which makes connectors also long period to release, >> whether it is (implicitly) beta or not. >> 2. Most of the change for connectors are not related to Storm core. It >> tends to be compatible with all release versions with same major version. >> 3. (IMHO) We have too many connectors which we even can't maintain >> actively. For example, ES connector couldn't support ES higher than 1.x. >> 4. Connectors are having same release version for Storm core, hence newly >> added connector will have at least 1.x version which no one would think it >> is beta. >> >> Downside: >> 1. Detached connectors can be easy to be forgotten. (easier than current) >> 2. Connectors may have hard time if we bring backward incompatible change >> to Storm core. We may remedy this with having supported version range for >> specific connector version. >> >> Please put your opinion regarding topic. You're encouraged to copy your >> previous post in vote thread which helps to centralize opinions in current >> thread. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >>
signature.asc
Description: Message signed with OpenPGP
