Hello Stig, +1 for your proposal (non-binding).
No problem to drop Java 7 support from my perspective (next Java Long Term Support is Java 11, Java 8 end of support is in Jan 2019, so who cares about Java 7?) And kudos for your proposed item: "Update the storm-kafka-client docs with a compatibility matrix showing which versions of Storm the connector is compatible with" - I'm still blocked with Kafka libs 0.10.2.0 for consuming our Kafka 1.0.1 Brokers using Storm Kafka Client 1.2.0, I had very unstable behavior when trying to use Kafka 1.0.0 libs for consuming our brokers (which were at Kafka 1.0.0 version). I hope to find time to resume tests and to provide feedbacks to clarify this. Best regards, Alexandre Vermeerbergen 2018-04-28 15:06 GMT+02:00 Stig Rohde Døssing <stigdoess...@gmail.com>: > Sorry about the necro, but I think this is still relevant. > > Would everyone be okay with trying out the following for storm-kafka-client > (to start)? > * Detach storm-kafka-client's release process from Storm's release process, > so we can release it separately, but keep the code in the Storm repo. We > can probably do this by making some changes to the poms, e.g. by skipping > release for storm-kafka-client unless a specific profile is set. > * Make the code on the master branch the new storm-kafka-client baseline. > This would drop support for Java 7. I doubt this will be an issue since > Java 7 has been out of date for a while now, but if we're not sure about > this we could open a poll on the mailing list to see if there's a need to > stay on Java 7. > * Update the storm-kafka-client docs with a compatibility matrix showing > which versions of Storm the connector is compatible with. > * Bump the storm-kafka-client version to e.g. 2.0.0(?) for the first > release using this method. > > I'm not sure if we should create a branch just for storm-kafka-client, or > if we can get away with just having development for it happen on master. > > As Taylor mentioned this should allow us to get some experience with > releasing an external component separately, without making it too hard to > roll back if we decide that releasing separately doesn't work well. > > If no one objects, I'll open an issue to make these changes. > > > 2018-02-09 16:51 GMT+01:00 Xin Wang <data.xinw...@gmail.com>: > >> I agree with Jungtaek. The same case has happened again on RocketMQ.( >> https://github.com/apache/storm/pull/2518) >> The following is my advice. >> >> 1. Now storm has too many connectors, we can separate the first class >> connectors from others. >> The following is a possible list including all existing connectors. >> >> First class: >> >> - Kafka, >> - HDFS, >> - HBase, >> - Hive, >> - Redis, >> - JDBC, >> - JMS >> >> >> >> Others: >> >> - Solr, >> - Cassandra, >> - Elasticsearch, >> - Event Hubs >> - RocketMQ >> - MongoDB >> - OpenTSDB >> - Kinesis >> - Druid >> - MQTT, >> - PMML >> >> >> 2. For first class connectors we can leave the code as it is, but release >> them independently; >> for other connectors, I prefer to move them to Bahir like the way of >> Spark/Flink. >> We can have a communication with the Bahir community, and request to create >> a https://github.com/apache/bahir-storm.git repo. >> >> >> >> 2018-02-01 9:10 GMT+08:00 P. Taylor Goetz <ptgo...@gmail.com>: >> >> > I’d start with Storm-Kafka-client as an experiment, and if that goes >> well, >> > move all connectors to the same model. >> > >> > Some connectors are bound to a stable protocol (e.g. JMS, MQTT), some are >> > bound to frequently changing APIs (e.g. Apache Kafka, cassandra, ES, >> etc.). >> > The former tend to be stable in terms of usage patterns and use cases, >> the >> > latter case case not so much. For example, consider hdfs integration. >> It’s >> > changed a lot in response to different usage patterns. Kafka due to >> > new/changing APIs. JMS hasn’t changed much at all since it’s tied to a >> > stable API. >> > >> > There’s also the fact that a high percentage of connectors integrate with >> > the most stable Storm APIs (spout, bolt, trident). The volatile (using >> the >> > term loosely) parts of our API affect projects like Mesos and >> streamparse, >> > but not the connectors we sponsor. >> > >> > -Taylor >> > >> > > On Jan 31, 2018, at 7:07 PM, Roshan Naik <ros...@hortonworks.com> >> wrote: >> > > >> > > I was thinking if the any connector is released more frequently, their >> > quality would be more mature and typically have lower impact on a Storm >> > release (compared to now) … if we decide to bundle them in Storm as well. >> > > -roshan >> > > >> > > >> > > On 1/31/18, 4:02 PM, "P. Taylor Goetz" <ptgo...@gmail.com> wrote: >> > > >> > > I think we all agree that releasing connectors as part of a Storm >> > release hinders the frequency of the release cycle for both Storm proper, >> > as well as connectors. >> > > >> > > If that’s the case, then the question is how to proceed. >> > > >> > > -Taylor >> > > >> > >> On Jan 31, 2018, at 6:46 PM, Roshan Naik <ros...@hortonworks.com> >> > wrote: >> > >> >> > >> One thought is to … >> > >> - do a frequent separate release >> > >> - *and also* include the latest stuff along with each Storm release. >> > >> >> > >> -roshan >> > >> >> > >> >> > >> On 1/31/18, 10:43 AM, "generalbas....@gmail.com on behalf of Stig >> > Rohde Døssing" <generalbas....@gmail.com on behalf of >> > stigdoess...@gmail.com> wrote: >> > >> >> > >> Hugo, >> > >> It's not my impression that anyone is complaining that >> > storm-kafka-client >> > >> has been exceptionally buggy, or that we haven't been fixing the >> > issues as >> > >> they crop up. The problem is that we're sitting on the fixes for way >> > longer >> > >> than is reasonable, and even if we release Storm more often, users >> > have to >> > >> go out of their way to know that they should really be using the >> > latest >> > >> storm-kafka-client rather than the one that ships with their Storm >> > >> installation, because the version number of storm-kafka-client >> > happens to >> > >> not mean anything regarding compatibility with Storm. >> > >> >> > >> Everyone, >> > >> >> > >> Most of what I've written here has already been said, but I've >> already >> > >> written it so... >> > >> >> > >> I really don't see the point in going through the effort of >> separating >> > >> connectors out to another repository if we're just going to make the >> > other >> > >> repository the second class citizen connector graveyard. >> > >> >> > >> The point to separating storm-kafka-client out is so it can get a >> > release >> > >> cycle different from Storm, so we can avoid the situation we're in >> > now in >> > >> the future. There's obviously a flaw in our process when we have to >> > choose >> > >> between breaking semantic versioning and releasing broken software. >> > >> >> > >> I agree that it would be good to release Storm a little more often, >> > but I >> > >> don't think that fully addresses my concerns. Are we willing to >> > increment >> > >> Storm's major version number if a connector needs to break its API >> > (e.g. as >> > >> I want to do in https://github.com/apache/storm/pull/2300)? >> > >> >> > >> I think a key observation is that Storm's core API is extremely >> > stable. >> > >> Storm and the connectors aren't usually tightly coupled in the sense >> > that >> > >> e.g. version 1.0.2 of storm-kafka-client would only work with Storm >> > 1.0.2 >> > >> and not 1.0.0, so in many cases there's no reason you wouldn't use >> the >> > >> latest connector version instead of the one that happens to ship >> with >> > the >> > >> version of Storm you're using. I think it would be attractive if we >> > could >> > >> reduce the number of branches of connectors we need to maintain, and >> > >> instead keep a compatibility matrix between Storm and the connector >> > in each >> > >> README, for the rare occasions when the Storm core API changes. >> > >> >> > >> +1 for trying out storm-kafka-client with its own release cycle and >> > >> branches/subrepo/whichever way we want to separate the code, but >> > still part >> > >> of the main Storm project JIRA and mailing list. Worst case we merge >> > it >> > >> back in after a while. We may want to think about how to do that >> > before we >> > >> separate out, just so we don't release e.g. storm-kafka-client 2.3.1 >> > and >> > >> then have to merge back to Storm which is still on 2.0.0. >> > >> >> > >> 2018-01-31 3:36 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>: >> > >> >> > >>> Agreed for this topic: this is not related to current release >> > candidate and >> > >>> verifying release candidate is higher priority. >> > >>> For me I didn't start verifying 1.1.2 / 1.0.6 RC2 because the other >> > topic I >> > >>> initiated could affect the current release. I'll post a short notice >> in >> > >>> that discussion thread. >> > >>> >> > >>> -Jungtaek Lim (HeartSaVioR) >> > >>> >> > >>> 2018년 1월 31일 (수) 오전 10:58, P. Taylor Goetz <ptgo...@gmail.com>님이 작성: >> > >>> >> > >>>> Hit send on that too soon... >> > >>>> >> > >>>> This is an important discussion topic, but has no effect on the >> > current >> > >>>> RCs. Id recommend focusing on the current releases and come back to >> > this >> > >>>> after getting releases out. >> > >>>> >> > >>>> -Taylor >> > >>>> >> > >>>>> On Jan 30, 2018, at 8:51 PM, P. Taylor Goetz <ptgo...@gmail.com> >> > >>> wrote: >> > >>>>> >> > >>>>> Also, in the interest of getting releases out, we have 3 open RC >> > cycles >> > >>>> in flight. >> > >>>>> >> > >>>>> Discussion energy might be better focused on that. >> > >>>>> >> > >>>>> -Taylor >> > >>>>> >> > >>>>>> On Jan 30, 2018, at 7:52 PM, P. Taylor Goetz <ptgo...@gmail.com> >> > >>> wrote: >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>>> On Jan 30, 2018, at 7:31 PM, Harsha <st...@harsha.io> wrote: >> > >>>>>>> >> > >>>>>>> Hi, >> > >>>>>>> In general connectors are independent of Storm run-time >> for >> > >>>> most parts. I.e if the APIs are not changed (storm-core or trident >> > >>> haven't >> > >>>> changed in years except the package re-name). You can take the >> latest >> > >>>> connector and run in storm 1.0 or higher. So the users doesn't need >> to >> > >>>> upgrade their storm cluster just to get a latest connector upgrade. >> > Which >> > >>>> they might be doing it but by making the release separate and >> stating >> > the >> > >>>> minimum supported storm version for the connectors will help the >> > users. >> > >>>>>>> This makes it easier for the connectors to be released >> > independently >> > >>>> of the core/run-time and makes it easy for them to be fixed and >> > released >> > >>>> more often. But moving them to Bahir or other external project will >> > make >> > >>> it >> > >>>> detached from Storm itself that it might not see any co-ordination >> as >> > >>>> reviewers from storm will need to be aware of an external project. >> > >>>>>>> My proposal would be >> > >>>>>>> 1. Can we create a sub-project in git under Storm so we can move >> > the >> > >>>> connectors there and everything else related Storm applies there. >> > >>>>>>> 2. Can we keep maintaining storm connectors within same repo but >> > >>>> different release module for it . >> > >>>>>> >> > >>>>>> +1 That’s exactly my point. Just jettisoning connectors to Bahir >> > >>>> without commitments from the Storm community would be a mistake. >> > >>>>>> >> > >>>>>> Releasing connectors independently can be handled easily at the >> > Maven >> > >>>> level. No need for a separate repo initiaially. >> > >>>>>> >> > >>>>>> >> > >>>>>>> >> > >>>>>>> This is a separate topic but can improve the release timelines if >> > we >> > >>>> have multiple release managers that are handling the maint release >> and >> > >>> also >> > >>>> main release versions. Its good to have rotation of release managers >> > from >> > >>>> PMC so that everyone will understand the process and can spread the >> > >>>> responsibilities. There are threads started before but don't think >> > they >> > >>> are >> > >>>> addressed or any action item is taken. We should start another >> thread >> > to >> > >>>> discuss this process as well. >> > >>>>>> >> > >>>>>> Breaking up external modules into separately released versions >> would >> > >>> be >> > >>>> a great way to indoctrinate those new to the license grooming and >> > release >> > >>>> process. Everyone could participate. >> > >>>>>> >> > >>>>>> >> > >>>>>>> >> > >>>>>>> Thanks, >> > >>>>>>> Harsha >> > >>>>>> >> > >>>>>> -Taylor >> > >>>>>> >> > >>>>>>> >> > >>>>>>>> On Tue, Jan 30, 2018, at 9:49 AM, Hugo Da Cruz Louro wrote: >> > >>>>>>>> I think that the bahir approach makes sense for connectors that >> > >>> don’t >> > >>>>>>>> fall into the "first class support” category. I am in favor of >> > >>> moving >> > >>>>>>>> such lower adoption connectors and have the interested >> communities >> > >>>>>>>> support them with the most suitable release cycle. Connectors >> that >> > >>> are >> > >>>>>>>> idle, such as some examples that Jungtaek gave, we should >> consider >> > >>>>>>>> removing them altogether, especially if they are so outdated >> that >> > >>> they >> > >>>>>>>> may not even work. >> > >>>>>>>> >> > >>>>>>>> Mainstream connectors such as storm-kafka-client should be kept >> in >> > >>> the >> > >>>>>>>> Storm repo. For example, Flink keeps flink-connector-kafka-0.x >> in >> > >>> the >> > >>>>>>>> Flink repo. >> > >>>>>>>> >> > >>>>>>>> I am in agreement with Jungtaek when he says: "fixing critical >> > bugs >> > >>> in >> > >>>>>>>> storm-kafka-client should trigger release, instead of waiting >> for >> > >>>> Storm >> > >>>>>>>> core to have some fixes to be worth to release”. Storm’s release >> > >>>> cadence >> > >>>>>>>> is currently not very high and one can argue that Storm entirely >> > >>> could >> > >>>>>>>> benefit from more frequent releases. If it is sto >> rm-kafka-client >> > >>>>>>>> triggering those releases, so be it. Moving forward I do not >> > expect >> > >>>> the >> > >>>>>>>> storm-kafka-client connector to be subject to so many changes >> that >> > >>> it >> > >>>>>>>> would warrant its own release cycle. >> > >>>>>>>> >> > >>>>>>>> I also would like to highlight that although storm-kafka-client >> > has >> > >>>> been >> > >>>>>>>> the center of this discussion, as it was mentioned in this >> > >>>>>>>> thread<https://goo.gl/VY7QTG>, storm-kafka-client has had a >> much >> > >>> less >> > >>>>>>>> rocky road to stability compared to for example storm-kafka. >> > >>> Therefore >> > >>>>>>>> it’s worth evaluating if the challenges that we have faced with >> > >>> storm- >> > >>>>>>>> kafka-client have been out of norm for such an important and >> > complex >> > >>>>>>>> feature, and if they warrant significant changes in how we do >> > >>> things. >> > >>>>>>>> >> > >>>>>>>> Thanks, >> > >>>>>>>> Hugo >> > >>>>>>>> >> > >>>>>>>> On Jan 29, 2018, at 9:18 PM, Jungtaek Lim >> > >>>>>>>> <kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote: >> > >>>>>>>> >> > >>>>>>>> Let me add a proof of my opinion: major patch of storm-eventhubs >> > >>>> hasn't >> > >>>>>>>> been getting even a comment over 4 months. >> > >>>>>>>> https://github.com/apache/storm/pull/2322 >> > >>>>>>>> >> > >>>>>>>> I'd rather want to discuss regarding discontinue supporting >> > >>>> officially if >> > >>>>>>>> we no longer interest of, or we don't have resource to support, >> or >> > >>> any >> > >>>>>>>> valid reasons. If we agree on discontinue supporting officially, >> > we >> > >>>> can >> > >>>>>>>> move out to other repo. and let it self maintained. It may be >> able >> > >>> to >> > >>>> get >> > >>>>>>>> attention and have enough contributors so that we feel better to >> > get >> > >>>> to >> > >>>>>>>> Storm core Repository again, or it can be silently forgotten. It >> > >>>> shouldn't >> > >>>>>>>> affect Storm core repository at any case. >> > >>>>>>>> >> > >>>>>>>> 2018년 1월 30일 (화) 오후 2:03, Jungtaek Lim <kabh...@gmail.com>님이 >> 작성: >> > >>>>>>>> >> > >>>>>>>> If we worry about breaking somethings along with our >> > >>>>>>>> users/consumers/distributors, picking one of less used/updated >> > >>>> connector as >> > >>>>>>>> experiment makes more sense to me. It's OK if we want to pick >> one >> > of >> > >>>> most >> > >>>>>>>> active and widely used connector intentionally to accelerate >> > >>>> experiment. >> > >>>>>>>> >> > >>>>>>>> Decoupling connectors and moving to other repo. like Bahir will >> > make >> > >>>> it >> > >>>>>>>> clear who are having interest of which connectors. >> storm-eventhubs >> > >>> for >> > >>>>>>>> example, major code contributions were done from MS developers. >> > Now >> > >>>> they >> > >>>>>>>> are gone, and I don't know even storm-eventhubs are compatible >> > with >> > >>>> recent >> > >>>>>>>> Azure Eventhub. That's just a one of them. I've seen many >> > connectors >> > >>>> in >> > >>>>>>>> same, or similar, or possible (say truck number 1) situation. >> > >>>>>>>> >> > >>>>>>>> -Jungtaek Lim (HeartSaVioR) >> > >>>>>>>> >> > >>>>>>>> 2018년 1월 30일 (화) 오후 1:30, P. Taylor Goetz <ptgo...@gmail.com>님이 >> > 작성: >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> On Jan 29, 2018, at 8:03 PM, Jungtaek Lim <kabh...@gmail.com> >> > >>> wrote: >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> - Do we ensure they're all maintained? >> > >>>>>>>> -- Did we exclude inactive committers/PMCs for connector's >> > committer >> > >>>>>>>> >> > >>>>>>>> sponsors, and do they have enough committer sponsors after that? >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> Good point. We’ve had some sponsors go silent recently. Maybe >> ping >> > >>>>>>>> sponsors and ask if they wish to maintain sponsorship? >> > >>>>>>>> >> > >>>>>>>> As a sponsor for a number of connectors, I’ll check on the ones >> > I’ve >> > >>>>>>>> sponsored. >> > >>>>>>>> >> > >>>>>>>> - Do they all worth to keep maintaining in Storm main >> repository? >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> Again, that’s a question of whether there is user/dev interest. >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> -- Should we trigger release if we find and resolve >> > critical/blocker >> > >>>> issue >> > >>>>>>>> from them? If not, why we allow to leave the thing which is in >> > main >> > >>>>>>>> repository as inconsistent state? >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> Some are tied to fairly well established protocols, some target >> > >>> really >> > >>>>>>>> volatile APIs. Bug reports and mailing list activity may not be >> a >> > >>> good >> > >>>>>>>> status indicator. >> > >>>>>>>> >> > >>>>>>>> Storm’s Kafka integration was the initial model for the >> “batteries >> > >>>>>>>> included” impetus behind `external`. If we want to evolve how >> that >> > >>>> works, >> > >>>>>>>> why not start there, see what works/doesn’t work, and adapt. >> > >>>>>>>> >> > >>>>>>>> I don’t want to shock our users/consumers/distributors. >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> -Taylor >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>> >> > >>> >> > >> >> > >> >> > > >> > > >> > > >> > >> >> >> >> -- >> Thanks, >> Xin >>
