Re: [DISCUSS] Decouple Storm core and connectors

Put up a PR for discussion, I think it's helpful to see some code before we decide whether we want to try releasing storm-kafka-client independently or not https://github.com/apache/storm/pull/2653. 2018-04-28 18:49 GMT+02:00 Alexandre Vermeerbergen : > Hello Stig, > >

Re: [DISCUSS] Decouple Storm core and connectors

Hello Stig, +1 for your proposal (non-binding). No problem to drop Java 7 support from my perspective (next Java Long Term Support is Java 11, Java 8 end of support is in Jan 2019, so who cares about Java 7?) And kudos for your proposed item: "Update the storm-kafka-client docs with a

Re: [DISCUSS] Decouple Storm core and connectors

Sorry about the necro, but I think this is still relevant. Would everyone be okay with trying out the following for storm-kafka-client (to start)? * Detach storm-kafka-client's release process from Storm's release process, so we can release it separately, but keep the code in the Storm repo. We

Re: [DISCUSS] Decouple Storm core and connectors

I agree with Jungtaek. The same case has happened again on RocketMQ.( https://github.com/apache/storm/pull/2518) The following is my advice. 1. Now storm has too many connectors, we can separate the first class connectors from others. The following is a possible list including all existing

Re: [DISCUSS] Decouple Storm core and connectors

I’d start with Storm-Kafka-client as an experiment, and if that goes well, move all connectors to the same model. Some connectors are bound to a stable protocol (e.g. JMS, MQTT), some are bound to frequently changing APIs (e.g. Apache Kafka, cassandra, ES, etc.). The former tend to be stable

Re: [DISCUSS] Decouple Storm core and connectors

I was thinking if the any connector is released more frequently, their quality would be more mature and typically have lower impact on a Storm release (compared to now) … if we decide to bundle them in Storm as well. -roshan On 1/31/18, 4:02 PM, "P. Taylor Goetz" wrote:

Re: [DISCUSS] Decouple Storm core and connectors

I think we all agree that releasing connectors as part of a Storm release hinders the frequency of the release cycle for both Storm proper, as well as connectors. If that’s the case, then the question is how to proceed. -Taylor > On Jan 31, 2018, at 6:46 PM, Roshan Naik

Re: [DISCUSS] Decouple Storm core and connectors

One thought is to … - do a frequent separate release - *and also* include the latest stuff along with each Storm release. -roshan On 1/31/18, 10:43 AM, "generalbas@gmail.com on behalf of Stig Rohde Døssing" wrote: Hugo,

Re: [DISCUSS] Decouple Storm core and connectors

Hugo, It's not my impression that anyone is complaining that storm-kafka-client has been exceptionally buggy, or that we haven't been fixing the issues as they crop up. The problem is that we're sitting on the fixes for way longer than is reasonable, and even if we release Storm more often, users

Re: [DISCUSS] Decouple Storm core and connectors

Agreed for this topic: this is not related to current release candidate and verifying release candidate is higher priority. For me I didn't start verifying 1.1.2 / 1.0.6 RC2 because the other topic I initiated could affect the current release. I'll post a short notice in that discussion thread.

Re: [DISCUSS] Decouple Storm core and connectors

Hit send on that too soon... This is an important discussion topic, but has no effect on the current RCs. Id recommend focusing on the current releases and come back to this after getting releases out. -Taylor > On Jan 30, 2018, at 8:51 PM, P. Taylor Goetz wrote: > >

Re: [DISCUSS] Decouple Storm core and connectors

Also, in the interest of getting releases out, we have 3 open RC cycles in flight. Discussion energy might be better focused on that. -Taylor > On Jan 30, 2018, at 7:52 PM, P. Taylor Goetz wrote: > > > >> On Jan 30, 2018, at 7:31 PM, Harsha wrote: >>

Re: [DISCUSS] Decouple Storm core and connectors

> On Jan 30, 2018, at 7:31 PM, Harsha wrote: > > Hi, >In general connectors are independent of Storm run-time for most > parts. I.e if the APIs are not changed (storm-core or trident haven't changed > in years except the package re-name). You can take the latest

Re: [DISCUSS] Decouple Storm core and connectors

Hi, In general connectors are independent of Storm run-time for most parts. I.e if the APIs are not changed (storm-core or trident haven't changed in years except the package re-name). You can take the latest connector and run in storm 1.0 or higher. So the users doesn't need to

Re: [DISCUSS] Decouple Storm core and connectors

I think that the bahir approach makes sense for connectors that don’t fall into the "first class support” category. I am in favor of moving such lower adoption connectors and have the interested communities support them with the most suitable release cycle. Connectors that are idle, such as

Re: [DISCUSS] Decouple Storm core and connectors

Let me add a proof of my opinion: major patch of storm-eventhubs hasn't been getting even a comment over 4 months. https://github.com/apache/storm/pull/2322 I'd rather want to discuss regarding discontinue supporting officially if we no longer interest of, or we don't have resource to support, or

Re: [DISCUSS] Decouple Storm core and connectors

If we worry about breaking somethings along with our users/consumers/distributors, picking one of less used/updated connector as experiment makes more sense to me. It's OK if we want to pick one of most active and widely used connector intentionally to accelerate experiment. Decoupling connectors

Re: [DISCUSS] Decouple Storm core and connectors

> On Jan 29, 2018, at 8:03 PM, Jungtaek Lim wrote: > > - Do we ensure they're all maintained? > -- Did we exclude inactive committers/PMCs for connector's committer > sponsors, and do they have enough committer sponsors after that? Good point. We’ve had some sponsors go

Re: [DISCUSS] Decouple Storm core and connectors

If we are only considering about storm-kafka-client, I'd rather not decouple and consider shorter release cycle for Storm itself, since storm-kafka-client 1.2.0 looks pretty stabilized. (That's more alike retrospect, like if we were doing it in 1.0.0 or even before 1.1.0.) Now storm-kafka-client

Re: [DISCUSS] Decouple Storm core and connectors

> On Jan 29, 2018, at 3:55 PM, Arun Mahadevan wrote: > > When you say storm-kafka-client-vX.X.X, is it going to be completely > independent of the Storm version and will work across Storm 1.0.x, 1.x and > 2.0 (future) storm releases? It would be completely independent in

Re: [DISCUSS] Decouple Storm core and connectors

If the storm-kafka-client is fairly stable with the changes we made in the 1.2 release, then would we just want to continue the current process ? If we want to decouple, I think option 2 may be better than option 1 to start with. When you say storm-kafka-client-vX.X.X, is it going to be

Re: [DISCUSS] Decouple Storm core and connectors

To give some background information, the Spark PMC decided to remove a number of connectors from their repo (and thus releases). Some members of the community wanted to see some sort of official community support for those connectors, thus the Apache Bahir project was created. Flink also

Re: [DISCUSS] Decouple Storm core and connectors

My idea is basically came from Apache Bahir. (http://bahir.apache.org/) It was for Apache Spark, but Flink decided to migrate their connectors to Bahir so it is also for Apache Flink. They're also maintaining some connectors (I'd say first class support) in their repositories, but not all. I think