Hello, +1 (non binding)
Details: I almost got nuts when applying storm-kafka-client-1.2.0 RC4 to our preproduction Storm cluster whose performances dropped quite a lot (at least x2 less throughput) ... but then I realized that I made the classic mistake of mixing two changes : I had also upgraded our Kafka dependencies from Kafka Client 0.10.20.0 to Kafka Client 1.0.0.0. Then I rollbacked our Kafka dependencies to Kafka 0.10.2.0 but I kept Storm 1.2.0 RC4 : since then (~10 hours ago) it's working smoothly. In particular, performances of Storm Kafka Client 1.2.0 RC4 having autocommit=true setting are equivalent to the ones of Storm Kafka Client 1.2.0 with .setProcessingGuarantee(ProcessingGuarantee.NO_GUARANTEE). Note : our preproduction Storm cluster consumes & produces data in a Kafka Cluster based on Kafka 1.0.0. I guess there's some details we missed when using Kafka Client 1.0.0 libraries instead of Kafka Client 0.10.2.0, but I think this is something we can study outside this 1.2.0 voting process, because consuming & producing Kafka messages against a Kafka cluster based on Kafka 1.0.0 Brokers is considered as compatible by Kakfa documentation. Since we want to move to Kafka Client 1.0.0 at some time (in particular, to take advantage of official JRE 9 support of this Kafka version), we will keep on testing our system with this newer Kafka version and we will feeback to Storm dev list whatever the outcome is (I'm guessing some property changes from 0.10.2 to 1.0.0) Thank you very much to all Storm developers for bringing this great Storm 1.2.0 to us, and for your taking into account my feedbacks: I have no more bad news for 1.2.0 :) Best regards, Alexandre Vermeerbergen PS: just one question: is there a way to access 1.2.0 RC4 Storm documentation before it's switched on public Storm site? PPS: I saw Stig's feedback about our deprecated usage of auto.commit.internal.ms, but I haven't yet tried the "new way", meaning that it has no impact on our big scale test. But I will try it later. 2018-02-09 9:18 GMT+01:00 Alexandre Vermeerbergen <avermeerber...@gmail.com>: > Hello, > > For your information, yesterday I upgraded our preproduction cluster > from Storm 1.2.0 SNAPSHOT of December 2017 to 1.2.0 Release Candidate > 4. > Our alerting system based on Storm behaved quite bad with 1.2.0 RC4, > and this morning, one of our team member noticed this message in our > topologie's startup logs: > > 1279 [main] WARN o.a.s.k.s.KafkaSpoutConfig - The KafkaConsumer > enable.auto.commit setting is not supported. You can configure similar > behavior through KafkaSpoutConfig.Builder.setProcessingGuarantee.This > will be treated as an error in the next major release. > > 1279 [main] INFO o.a.s.k.s.KafkaSpoutConfig - Setting Kafka consumer > property 'enable.auto.commit' to 'false', because the spout does not > support auto-commit > > > So I will resume our tests with this new setting and I will feedback > once I have enought uptime on our "large preproduction cluster" with > 1.2.0 RC4. > > Note: may I suggest to make this breaking change visible in Storm > 1.2.0 releases notes? this is quite impacting. Or event better: make > the topologies unable to start when they use such a removed property, > so that at least people aren't fooled until they wonder why their > Kafka spouts aren't anymore behaving like before and check logs? > > More to come when my preproduction tests will have been completed (1or > 2 days needed). > > Best regards, > Alexandre > > 2018-02-07 23:24 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>: >> This is a call to vote on releasing Apache Storm 1.2.0 (rc4) >> >> Note that the only difference between rc4 and rc3 is the fix for >> https://issues.apache.org/jira/browse/STORM-2942. >> >> Full list of changes in this release: >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/RELEASE_NOTES.html >> >> The tag/commit to be voted upon is v1.2.0: >> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=cef4d49e222e53656f38c40d754d4f41799cd9a9;hb=2a0097f9a20b9df494caadb87c87d4e4db01a7ed >> >> The source archive being voted upon can be found here: >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/apache-storm-1.2.0-src.tar.gz >> >> Other release files, signatures and digests can be found here: >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/ >> >> The release artifacts are signed with the following key: >> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd >> >> The Nexus staging repository for this release is: >> >> https://repository.apache.org/content/repositories/orgapachestorm-1058 >> >> Please vote on releasing this package as Apache Storm 1.2.0. >> >> When voting, please list the actions taken to verify the release. >> >> This vote will be open for at least 72 hours. >> >> [ ] +1 Release this package as Apache Storm 1.2.0 >> [ ] 0 No opinion >> [ ] -1 Do not release this package because... >> >> Thanks to everyone who contributed to this release. >> >> -Taylor
