Regarding storm-kafka-client, what Jungtaek is saying lines up with why I (and likely others) haven't been backporting all fixes to 1.0.x and 1.1.x. We've been trying to follow semantic versioning, so new features and deprecations have been going in 1.2.0 rather than patch releases. This has caused 1.1.x and 1.2.0 to diverge a bunch, so backporting fixes has become difficult. Several fixes also required deprecations, so shouldn't have gone in patch releases. I didn't bother backporting because I reasoned that people could just use the 1.2.0 spout if they encounter problems, since it is compatible with 1.0.x clusters. I also wasn't sure at the time whether there would be any more 1.0.x releases.
I don't believe 1.2.0 has any breaking changes from 1.1.x (not sure about 1.0.x), so I'd be fine with just overwriting the storm-kafka-client in those branches with the one from 1.2.0. I haven't tried out storm-kafka-client 1.2.0 with an 0.9 cluster, but the Kafka documentation for pre-0.10 versions recommends upgrading the broker before the clients, so it may not work. Due to API changes to KafkaConsumer, I think we can either support Kafka 0.9 or Kafka >= 0.10, so I'd rather drop 0.9 support if we need to. It's been a little annoying that we can only release storm-kafka-client alongside Storm, but I'm hoping it becomes less of an issue as storm-kafka-client stabilizes. I think the 1.2.0 release has also been delayed unusually long (I think we started talking about releasing it sometime around August), so maybe it won't be a problem going forward? 2018-01-25 7:14 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>: > Btw, I found blocker issue based on the contributor's report in 1.2.0 RC1, > and found the cause. > > Filed https://issues.apache.org/jira/browse/STORM-2912. Will craft patches > for branches shortly. > > -1 (binding) > > - Jungtaek Lim (HeartSaVioR) > > 2018년 1월 25일 (목) 오후 12:43, Jungtaek Lim <kabh...@gmail.com>님이 작성: > > > IMHO, I have been wondering that it is still a good strategy to couple > > Storm core and connectors with one version. That has been making painful > > situations in storm-kafka-client, where it could be marked as beta and > > release often with free to break backward compatibility until it is > fairly > > stable, but it can't be released often because it requires Storm itself > to > > be released which requires huge bootstrap. > > > > 2018년 1월 25일 (목) 오후 12:37, Jungtaek Lim <kabh...@gmail.com>님이 작성: > > > >> Hi Erik, > >> > >> Thanks for bring up issue. Yes you're right that all the patches are > >> going through 1.x-branch (1.2.0), but not 1.1.x-branch because of huge > >> divergence. We have limited resource so I'm afraid to say folks to port > >> back issues with much pain of conflicts. > >> > >> Maybe we could make somewhat crazy but easiest decision: break backward > >> compatibility once again for storm-kafka-client (sad), and just copy the > >> code on 1.x-branch to 1.1.x-branch and also 1.0.x-branch. 1.0.x-branch > was > >> support for Kafka 0.9 (odd strategy btw), but once Kafka published > >> 1.0.0, I'd rather not thinking about Kafka 0.9. > >> > >> To. PMCs > >> Makes sense? Need to initiate another discussion thread? > >> > >> Thanks, > >> Jungtaek Lim (HeartSaVioR) > >> > >> 2018년 1월 25일 (목) 오전 10:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성: > >> > >>> Erik, > >>> > >>> Open and honest opinions are always welcome. Even if they may come > >>> across as harsh (and yours did not). We value your opinion, and that > of the > >>> mesos community. > >>> > >>> I’d like to get these issues resolved and would ask others to help > where > >>> they can. > >>> > >>> You’ve been very forthcoming with information to help resolve this. > >>> Please don’t stop. The more information the better. > >>> > >>> Thanks again, > >>> > >>> -Taylor > >>> > >>> > >>> > >>> > >>> > >>> > On Jan 24, 2018, at 8:12 PM, Erik Weathers > >>> <eweath...@groupon.com.INVALID> wrote: > >>> > > >>> > Hi Taylor, > >>> > > >>> > I apologize that this objection is a bit long, incomplete, and late > for > >>> > this release vote. Also, I by no means intend this as an attack on > the > >>> > good folks that develop and maintain storm-kafka-client. That being > >>> said, > >>> > I'm uncomfortable with the situation of storm-kafka-client in the > >>> > 1.0.x-branch (and possibly 1.1.x-branch too) and I wonder if everyone > >>> is > >>> > aware of the situation. There seem to be a number of important > >>> > storm-kafka-client changes that haven't been backported to the > >>> > 1.0.x-branch. e.g., > >>> > > >>> > (1) We discovered in storm-1.0.3 that a spout can get stuck forever > if > >>> its > >>> > stored offsets fall behind the earliest available and > >>> > FirstPollOffsetStrategy is set to UNCOMMITTED_EARLIEST or > >>> > UNCOMMITTED_LATEST. We believe this behavior to be fixed in newer > >>> > branches, but not in 1.0.x-branch. The issue is here (note that the > >>> > fetchOffset doesn't get updated in the case that it was out of bounds > >>> for > >>> > the seek): > >>> > * https://github.com/apache/storm/blob/v1.0.6/external/ > >>> > storm-kafka-client/src/main/java/org/apache/storm/kafka/ > >>> > spout/KafkaSpout.java#L188-L192 > >>> > > >>> > (2) storm-1.0.x is using kafka-clients-0.9.0.1, which isn't > acceptable > >>> when > >>> > using Kafka 0.10 due to the performance impact on the Kafka > cluster. I > >>> > believe, perhaps naively, that we could use kafka-clients-0.10.x just > >>> fine > >>> > in storm-kafka-client-1.0.x, even when speaking to a Kafka 0.9 > cluster. > >>> > (Obviously we'd have to fix any issues with the Kafka API usage from > >>> > storm-kafka-clients.) > >>> > > >>> > > >>> > Notably, this situation is especially problematic for storm-on-mesos, > >>> since > >>> > we *cannot* run anything newer than storm-1.0.x for the daemons > >>> (Nimbus, > >>> > Supervisor, Worker), because of a fundamental change that was made to > >>> the > >>> > storm-core scheduling logic. > >>> > > >>> > I haven't yet spent the time to fully analyze the set of changes made > >>> to > >>> > storm-kafka-client in the various active branches, so I apologize > for a > >>> > lack of links to existing PRs and Jira tickets. I also need to file > >>> > tickets for some of this stuff, as I believe issue (1) above was > fixed > >>> in > >>> > newer branches as part of a large patch for another issue. > >>> > > >>> > I also suppose that this isn't super actionable, since it might not > be > >>> fair > >>> > to hold back the entire storm release just for these issues. > However, > >>> if > >>> > we can get these issues fixed in 1.0.x-branch I hope we can cut the > >>> 1.0.7 > >>> > release soon thereafter. > >>> > > >>> > Thanks! > >>> > > >>> > - Erik > >>> > > >>> >> On Wed, Jan 24, 2018 at 10:41 AM, P. Taylor Goetz < > ptgo...@gmail.com> > >>> wrote: > >>> >> > >>> >> This is a call to vote on releasing Apache Storm 1.0.6 (rc1) > >>> >> > >>> >> Full list of changes in this release: > >>> >> > >>> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1. > >>> >> 0.6-rc1/RELEASE_NOTES.html > >>> >> > >>> >> The tag/commit to be voted upon is v1.0.6: > >>> >> > >>> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h > >>> >> =24a421e34a71353dc6c750b1f026d06df8ead3f2;hb=bce45993f8622e4 > >>> >> d3e9ccba96cc78e4ef76e48ae > >>> >> > >>> >> The source archive being voted upon can be found here: > >>> >> > >>> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1. > >>> >> 0.6-rc1/apache-storm-1.0.6-src.tar.gz > >>> >> > >>> >> Other release files, signatures and digests can be found here: > >>> >> > >>> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1. > 0.6-rc1/ > >>> >> > >>> >> The release artifacts are signed with the following key: > >>> >> > >>> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_p > >>> >> lain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd > >>> >> > >>> >> The Nexus staging repository for this release is: > >>> >> > >>> >> > >>> https://repository.apache.org/content/repositories/orgapachestorm-1054 > >>> >> > >>> >> Please vote on releasing this package as Apache Storm 1.0.6. > >>> >> > >>> >> When voting, please list the actions taken to verify the release. > >>> >> > >>> >> This vote will be open for at least 72 hours. > >>> >> > >>> >> [ ] +1 Release this package as Apache Storm 1.0.6 > >>> >> [ ] 0 No opinion > >>> >> [ ] -1 Do not release this package because... > >>> >> > >>> >> Thanks to everyone who contributed to this release. > >>> >> > >>> >> -Taylor > >>> >> > >>> > >> >
