Looks like now we are only waiting on below Kafka spout issues : https://github.com/apache/storm/pull/2428 https://github.com/apache/storm/pull/2438
Maybe we should include the metrics changes as well? https://github.com/apache/storm/pull/2203 Can we try to get the above merged ASAP and start the 1.2.0 release process ? Thanks, Arun On 11/21/17, 3:18 AM, "generalbas....@gmail.com on behalf of Stig Rohde Døssing" <generalbas....@gmail.com on behalf of stigdoess...@gmail.com> wrote: >Alexandre, > >It's a bug in the way I tried to fix the NPE you had a few days ago in >https://github.com/apache/storm/pull/2428. I missed that using >setKey/setValue actually builds a new KafkaSpoutConfig.Builder instead of >just setting a field, and the change I made to the copy constructor means >that if the value deserializer is set in kafkaProps (which it is when using >KafkaSpoutConfig.builder), using setKey/Value is ignored. > >I've amended the fix to STORM-2826 and added a few more tests. The new jar >is at >https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_v9fd/view?usp=sharing >. > >There is not as far as I know a way for you to get the subscribed topics >from the subscription. > >2017-11-21 11:04 GMT+01:00 Alexandre Vermeerbergen <avermeerber...@gmail.com >>: > >> Hello Stig, >> >> Here's an update of my tests with storm 1.2.0 preview: >> - I accept the limitation on the stability of the string format returned by >> getTopicsString(), as I have adapted our code to detect both 1.1.0-style & >> 1.2.0-style. Isn't there a clean way to get the list of topics other than >> our fragile parsing? >> - My ~15 topologies have been running for 24 hours with storm 1.2.0 preview >> + our own Kafka spout deriving from Storm kafka client 1.2.0 preview >> setting, I have seen no stability nor performance issue (but that's not yet >> a large scale test). >> - When I tried to switch one of our topologies to your storm-kafka-client, >> I was surprised to get no stats on the topology. >> Then I noticed exceptions for all messages read by the spout: >> >> java.lang.String cannot be cast to >> com.dassault_systemes.infra.monitoring.model.Event >> java.lang.ClassCast*Excep >> <http://ows-171-33-121-83.eu-west-2.compute.outscale.com:8000/log?file= >> statefulAlerting_ows-171-33-69-118-eu-west-2-compute-outscale-com_ >> defaultStormTopic-165-1511258026%2F6706%2Fworker. >> log&start=19293&length=51200>*tion: >> java.lang.String cannot be cast to >> com.acme_systemes.infra.monitoring.model.Event >> at com.dassault_systemes.storm.eval >> >> >> And also: >> >> 2017-11-21 08:28:40.958 o.a.s.k.s.KafkaSpout >> Thread-5-eventFromAdminTopic-executor[12 12] [INFO] Kafka Spout opened >> with >> the following configuration: >> KafkaSpoutConfig{kafkaProps={key.deserializer=class >> org.apache.kafka.common.serialization.StringDeserializer, >> value.deserializer=class >> org.apache.kafka.common.serialization.StringDeserializer, >> enable.auto.commit=false, request.timeout.ms=1200000, >> group.id=Storm_RealTimeSupervision_9XkvRUExS2GFNAZNcBjQug_ >> defaultStormTopic_alerting_administration, >> bootstrap.servers=ows-171-33-69-118.eu-west-2.compute.outscale.com:9092, >> auto.commit.interval.ms=60000, session.timeout.ms=120000, >> auto.offset.reset=earliest}, >> key=org.apache.kafka.common.serialization.StringDeserializer@61dc4a48, >> value=com.acme_systemes.storm.evaluator.spout.EventKafkaDeserializer@ >> 5d6e1916, >> pollTimeoutMs=200, offsetCommitPeriodMs=30000, >> maxUncommittedOffsets=10000000, firstPollOffsetStrategy=LATEST, >> subscription=org.apache.storm.kafka.spout.ManualPartitionSubscription@ >> 4ff512c9, >> translator=com.acme_systemes.storm.evaluator.spout. >> EventKafkaRecordTranslator@593c16f5, >> retryService=KafkaSpoutRetryExponentialBackoff{delay=TimeInterval{length= >> 0, >> timeUnit=SECONDS}, ratio=TimeInterval{length=2, timeUnit=MILLISECONDS}, >> maxRetries=2147483647, maxRetryDelay=TimeInterval{length=10, >> timeUnit=SECONDS}}, tupleListener=EmptyKafkaTupleListener} >> >> This later stack is very strange: it shows that our custom deserializer was >> indeed taken into account in a field called "value=..." but not as the >> key.deserializer which remained set to StringDeserializer. >> >> Our Kafka spout initialization code is the following one: >> >> KafkaSpoutConfig<String, Event> spoutConfigForMainTopic = >> KafkaSpoutConfig >> .builder(elasticKafkaBrokers, KafkaTopics.MAIN) >> .setValue(EventKafkaDeserializer.class) >> .setGroupId(consumerId + "_" + KafkaTopics.MAIN) >> .setFirstPollOffsetStrategy(strategy) >> .setProp(kafkaConsumerProp) >> .setRecordTranslator( >> new EventKafkaRecordTranslator(true)) >> .build(); >> >> We then noticed the following discussion: >> >> http://mail-archives.apache.org/mod_mbox/storm-user/201709.mbox/% >> 3ccag09er3yzsxw84u6xvqevdojy-j2hf-jrnnxhk845bzo5d4...@mail.gmail.com%3e >> >> so if I understand well, there's a breaking change between Storm 1.1.0 and >> 1.2.0 in the way of registering Kafka deserializers. >> >> Do you confirm ? >> >> I can change our code, but I would like to keep the ability to run it with >> either 1.1.0 or 1.2.0 for a while : is there a way I can register Kafka >> deserializers for storm-kafka-client that will work with both 1.1.0 and >> 1.2.0 ? >> >> Best regards, >> Alexandre Vermeerbergen >> >> >> >> >> >> >> >> 2017-11-20 18:40 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: >> >> > Happy to hear that it's at least running for you now :) Eager to hear >> your >> > results. >> > >> > Regarding the getTopicsString format, the Subscription interface doesn't >> > specify the string format (and it was never intended to be parsed in this >> > way). In my opinion the format is an implementation detail, so we should >> be >> > free to change it at any time. >> > >> > 2017-11-20 15:01 GMT+01:00 Alexandre Vermeerbergen < >> > avermeerber...@gmail.com >> > >: >> > >> > > Hello All, >> > > >> > > Good news today: I found & solved what was preventing my topologies >> from >> > > consuming since I had upgraded from storm-kafka-client 1.1.0 to >> > > storm-kafka-client.1.2.0-lastestsnapopfromstig. >> > > >> > > The reasons from our own BasicKafkaSpout class, which is our homebrewed >> > > Kafka spout based on same configuration settings as the official >> > > storm-kafka-client Kafka spout. >> > > >> > > The issue came from the fact that, at least storm-kafka-client 1.1.0 >> > > doesn't exposes a way to get the list of consumed topics, so we had to >> > > parse the value returned by KafkaSpoutConfig<K, >> > > V>.getSubscription().getTopicsString() in order to extract the list of >> > > topics. >> > > >> > > Here's where the issue lies: >> > > >> > > public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) { >> > > this.kafkaBrokers = (String) config.getKafkaProps().get( >> > > ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG); >> > > this.consumerId = config.getConsumerGroupId(); >> > > logger.debug("topics are: {}", config.getSubscription() >> > > .getTopicsString()); >> > > String topicsStr = config.getSubscription().getTopicsString(); >> > > this.topics = topicsStr.substring(1, topicsStr.length() - >> > > 1).split(","); >> > > >> > > => Bug here with storm-kafka-client 1.2.0 snapshot, because if >> > > there is only one topic, >> > > the value of topicsStr which was "[mytopic]" with 1.1.0 is >> now >> > > "mytopic" >> > > >> > > and here's a fixed (and ugly) version which works with both 1.1.0 & >> 1.2.0 >> > > snapshot of the same code snippet: >> > > >> > > public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) { >> > > this.kafkaBrokers = (String) config.getKafkaProps().get( >> > > ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG); >> > > this.consumerId = config.getConsumerGroupId(); >> > > logger.debug("topics are: {}", config.getSubscription() >> > > .getTopicsString()); >> > > String topicsStr = config.getSubscription().getTopicsString(); >> > > if (topicsStr.startsWith("[") && topicsStr.endsWith("]")) { >> > > this.topics = topicsStr.substring(1, topicsStr.length() - >> > > 1).split(","); >> > > } >> > > else { >> > > this.topics = topicsStr.split(","); >> > > } >> > > >> > > Maybe it would be good to either restore this string representation as >> it >> > > was in 1.1.0, or if not your intention, they document the change. >> > > >> > > Now I guess your question is : "why on earth did you implement your own >> > > Kafka Spout?" >> > > >> > > The answer is simple: we were waiting for 1.2.0 which contains a fix to >> > > storm-kafka-client to that Spout statistics are visible in Storm UI >> when >> > > this spout is used in autocommit mode. Our own spout was design on >> > purpose >> > > to have same config as torm-kafka-client so as to make switch from our >> > > implementation to official storm-kafka-client-1.2.0 as simple as simple >> > (as >> > > a matter of fact, it's a simple property to change on our side). We >> > > couldn't wait for 1.2.0 because we had to switch to Kafka 0.10.2.0 as >> > soon >> > > as possible. >> > > >> > > Next steps for me related to Storm 1.2.0 preview tests: >> > > * Try our topologies with the official storm-kafka-client-1.2.0 in >> order >> > to >> > > test non-regression >> > > * Try our topologies with the official storm-kafka-client-1.2.0 on our >> > > pre-production in order to test non-regression of performances. >> > > >> > > Hope it helps, stay tuned :) >> > > >> > > Best regards, >> > > Alexandre Vermeerbergen >> > > >> > > >> > > 2017-11-20 8:49 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: >> > > >> > > > Alexandre, >> > > > >> > > > Could you also post the BasicKafkaSpout source file? I'm curious what >> > > it's >> > > > doing. >> > > > >> > > > 2017-11-20 7:50 GMT+01:00 Alexandre Vermeerbergen < >> > > > avermeerber...@gmail.com> >> > > > : >> > > > >> > > > > Hello Jungtaek, >> > > > > >> > > > > OK I will activate these traces, but since we need to capture the >> > > Spouts' >> > > > > initialization traces, how should I activate these traces? >> > > > > >> > > > > Indeed, if I use one of the techniques shown here >> > > > > https://community.hortonworks.com/articles/36151/debugging- >> > > > > an-apache-storm-topology.html >> > > > > then I'm afraid I have to wait until the topology is deployed >> before >> > > > > setting its trace level. >> > > > > >> > > > > Would you please clarify how to activate traces enabling to get >> > spout's >> > > > > initialization. >> > > > > (please be as specific as possible for our current case - I can >> even >> > > > modify >> > > > > some code if that's the only to get this "early" activation, but >> then >> > > > > please specify which lines of code I need to add) >> > > > > >> > > > > Best regards, >> > > > > Alexandre Vermeerbergen >> > > > > >> > > > > >> > > > > 2017-11-20 1:20 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>: >> > > > > >> > > > > > It would be much appreciated if you could change topology log >> level >> > > to >> > > > > the >> > > > > > following: level: DEBUG, logger name: 'ROOT' or 'org.apache', >> > > timeout: >> > > > > long >> > > > > > enough (say 1800 or 3600), and kill worker which contains Spout >> in >> > UI >> > > > or >> > > > > > console. >> > > > > > >> > > > > > Above instruction enables logging with DEBUG level, and Kafka >> spout >> > > > being >> > > > > > restarted, so we could see initialization phase of Kafka spout. >> > > > > > >> > > > > > Thanks, >> > > > > > Jungtaek Lim (HeartSaVioR) >> > > > > > >> > > > > > 2017년 11월 20일 (월) 오전 8:09, Alexandre Vermeerbergen < >> > > > > > avermeerber...@gmail.com>님이 >> > > > > > 작성: >> > > > > > >> > > > > > > Hello Stig, >> > > > > > > >> > > > > > > Thanks again for your latest fix. >> > > > > > > >> > > > > > > I have no longer any exception when submitting my topologies, >> but >> > > > then >> > > > > > they >> > > > > > > read nothing from my Kafka topics. >> > > > > > > >> > > > > > > So I made another test: I reverted from your latest >> > > > storm-kafka-client >> > > > > > > 1.2.0 snapshot jar to storm-kafka-client-1.1.0.jar in my >> > > topologies, >> > > > > but >> > > > > > I >> > > > > > > kept an installation based on Storm 1.2.0 snapshot => Kafka >> > > > consumption >> > > > > > is >> > > > > > > OK >> > > > > > > >> > > > > > > If I revert again to your latest storm-kafka-client 1.2.0 >> > snapshot >> > > > jar, >> > > > > > > then Kafka consumption never starts. >> > > > > > > >> > > > > > > I have no exceptions in logs worker's artifact logs, I'm unsure >> > > what >> > > > to >> > > > > > try >> > > > > > > now... >> > > > > > > >> > > > > > > Any trace which I could activate? >> > > > > > > >> > > > > > > Best regards, >> > > > > > > Alexandre Vermeerbergen >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > 2017-11-19 16:48 GMT+01:00 Stig Rohde Døssing < >> > > > stigdoess...@gmail.com >> > > > > >: >> > > > > > > >> > > > > > > > Thanks, I've addressed the issue here >> > > > > > > > https://github.com/apache/storm/pull/2428 and uploaded a new >> > jar >> > > > at >> > > > > > the >> > > > > > > > same link here https://drive.google.com/file/d/ >> > > > > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_ >> > > > > > > > v9fd/view?usp=sharing. I went over the PR that made these >> > > changes, >> > > > > and >> > > > > > I >> > > > > > > > don't believe anything else breaks backward compatibility, >> but >> > > > we'll >> > > > > > see. >> > > > > > > > >> > > > > > > > 2017-11-19 15:48 GMT+01:00 Alexandre Vermeerbergen < >> > > > > > > > avermeerber...@gmail.com >> > > > > > > > >: >> > > > > > > > >> > > > > > > > > Hi Stig, >> > > > > > > > > >> > > > > > > > > Here's the source of our "BasicKafkaSpout" class's >> > constructor >> > > (I >> > > > > can >> > > > > > > > send >> > > > > > > > > the full source if needed), >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) >> { >> > > > > > > > > this.kafkaBrokers = (String) >> > > config.getKafkaProps().get( >> > > > > > > > > ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG); >> > > > > > > > > this.consumerId = config.getConsumerGroupId(); >> > > > > > > > > logger.debug("topics are: {}", >> > config.getSubscription() >> > > > > > > > > .getTopicsString()); >> > > > > > > > > String topicsStr = config.getSubscription(). >> > > > > > getTopicsString(); >> > > > > > > > > this.topics = topicsStr.substring(1, >> > > topicsStr.length() - >> > > > > > > > > 1).split(","); >> > > > > > > > > switch (config.getFirstPollOffsetStrategy()) { >> > > > > > > > > case UNCOMMITTED_EARLIEST: >> > > > > > > > > case EARLIEST: >> > > > > > > > > this.strategy = "earliest"; >> > > > > > > > > break; >> > > > > > > > > case UNCOMMITTED_LATEST: >> > > > > > > > > case LATEST: >> > > > > > > > > default: >> > > > > > > > > this.strategy = "latest"; >> > > > > > > > > } >> > > > > > > > > this.keyDeserializer = config.getKeyDeserializer(). >> > > > > > getClass(); >> > > > > > > > > this.valueDeserializer = >> > config.getValueDeserializer(). >> > > > > > > > getClass(); >> > > > > > > > > this.translator = config.getTranslator(); >> > > > > > > > > this.consumerProps = new Properties(); >> > > > > > > > > if (config.getKafkaProps() != null) { >> > > > > > > > > consumerProps.putAll(config.getKafkaProps()); >> > > > > > > > > } >> > > > > > > > > } >> > > > > > > > > >> > > > > > > > > line 72 is the following one: >> > > > > > > > > >> > > > > > > > > this.keyDeserializer = config.getKeyDeserializer(). >> > > > > > getClass(); >> > > > > > > > > >> > > > > > > > > Hope it helps, >> > > > > > > > > >> > > > > > > > > Alexandre Vermeerbergen >> > > > > > > > > >> > > > > > > > > 2017-11-19 15:37 GMT+01:00 Stig Rohde Døssing < >> > > > > > stigdoess...@gmail.com >> > > > > > > >: >> > > > > > > > > >> > > > > > > > > > Alexandre, >> > > > > > > > > > >> > > > > > > > > > I'm sorry this is giving you so much trouble. Looking at >> > the >> > > > > stack >> > > > > > > > trace >> > > > > > > > > > you posted, it seems like the NPE is coming from >> > > > BasicKafkaSpout >> > > > > > line >> > > > > > > > 72. >> > > > > > > > > > Can you post that line (and maybe some of the surrounding >> > > > code)? >> > > > > > > > > > >> > > > > > > > > > 2017-11-19 15:13 GMT+01:00 Alexandre Vermeerbergen < >> > > > > > > > > > avermeerber...@gmail.com >> > > > > > > > > > >: >> > > > > > > > > > >> > > > > > > > > > > Hi Stig, >> > > > > > > > > > > >> > > > > > > > > > > After having included commons-lang-2.5.jar in my >> > > topologies, >> > > > I >> > > > > > > still >> > > > > > > > > have >> > > > > > > > > > > 11 topologies failing with this new message, this time >> a >> > > > > > > > > > > NullPointerException: >> > > > > > > > > > > >> > > > > > > > > > > Running: /usr/local/jdk/bin/java -client -Ddaemon.name= >> > > > > > > > > -Dstorm.options= >> > > > > > > > > > > -Dstorm.home=/usr/local/Storm/storm-stable >> > > > > > > > > > > -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs >> > > > > > > > > > > -Djava.library.path=/usr/ >> local/lib:/opt/local/lib:/usr/ >> > lib >> > > > > > > > > > > -Dstorm.conf.file= -cp >> > > > > > > > > > > /usr/local/Storm/storm-stable/ >> *:/usr/local/Storm/storm- >> > > > > > > > > > > stable/lib/*:/usr/local/Storm/ >> > storm-stable/extlib/*:/usr/ >> > > > > > > > > > > local/Storm/StormTopologiesBJ. >> > jar:/usr/local/Storm/storm- >> > > > > > > > > > > stable/conf:/usr/local/Storm/storm-stable/bin >> > > > > > > > > > > -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar >> > > > > > > > > > > -Dstorm.dependency.jars= >> > > > > > > > > > > -Dstorm.dependency.artifacts={} >> > > > > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology >> > > > > > > > > > slaEventsInterceptor >> > > > > > > > > > > 1939 [main] WARN c.a.s.u.StormUtils - Couldn't read >> > > > > > > > > > > /usr/local/Storm/flux/slaEventsInterceptor_ >> > > > > > kafkaConsumer.properties >> > > > > > > > > > file: >> > > > > > > > > > > /usr/local/Storm/flux/slaEventsInterceptor_ >> > > > > > kafkaConsumer.properties >> > > > > > > > > (No >> > > > > > > > > > > such file or directory) >> > > > > > > > > > > 1978 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not >> set >> > > > > > > > > > > enable.auto.commit manually. Instead use >> > > > > > > > > > > KafkaSpoutConfig.Builder.setProcessingGuarantee. This >> > will >> > > > be >> > > > > > > > treated >> > > > > > > > > as >> > > > > > > > > > > an >> > > > > > > > > > > error in the next major release. For now the spout will >> > be >> > > > > > > configured >> > > > > > > > > to >> > > > > > > > > > > behave like it would have in pre-1.2.0 releases. >> > > > > > > > > > > 1982 [main] INFO c.a.s.u.StormUtils - Use Kafka Spout: >> > > > > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout >> > > > > > > > > > > Exception in thread "main" >> java.lang.NullPointerException >> > > > > > > > > > > at >> > > > > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>( >> > > > > > > > > > > BasicKafkaSpout.java:72) >> > > > > > > > > > > at com.acme.storm.util. >> StormUtils.getKafkaSpout( >> > > > > > > > > > > StormUtils.java:888) >> > > > > > > > > > > at >> > > > > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology. >> > > main( >> > > > > > > > > > > SLAEventsInterceptorTopology.java:120) >> > > > > > > > > > > >> > > > > > > > > > > and here's the line from our source from which this >> > > exception >> > > > > was >> > > > > > > > > > > triggered: >> > > > > > > > > > > >> > > > > > > > > > > IRichSpout kafkaSpoutForMainTopic = StormUtils >> > > > > > > > > > > .getKafkaSpout( >> spoutConfigForMainTopic, >> > > > > > > > > > kafkaSpoutClass); >> > > > > > > > > > > >> > > > > > > > > > > Looks like upgrade from 1.1.0 (our current production >> > > setup) >> > > > to >> > > > > > > 1.2.0 >> > > > > > > > > is >> > > > > > > > > > a >> > > > > > > > > > > bit more difficult than I expected. >> > > > > > > > > > > But anyway, I greatly appreciate Storm, so I hope my >> > > > feedbacks >> > > > > > will >> > > > > > > > > help >> > > > > > > > > > > keeping it as good as possible :) >> > > > > > > > > > > >> > > > > > > > > > > Please let me know if further details are required to >> > solve >> > > > > this >> > > > > > > > issue >> > > > > > > > > > with >> > > > > > > > > > > storm-kafka-client 1.2.0 snapshot. >> > > > > > > > > > > >> > > > > > > > > > > Best regards, >> > > > > > > > > > > Alexandre >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > 2017-11-19 13:26 GMT+01:00 Stig Rohde Døssing < >> > > > > > > > stigdoess...@gmail.com >> > > > > > > > > >: >> > > > > > > > > > > >> > > > > > > > > > > > Using Maven and controlling dependencies aren't >> > mutually >> > > > > > > exclusive. >> > > > > > > > > > Tools >> > > > > > > > > > > > like Nexus (https://help.sonatype.com/ >> > > > > > display/NXRM2/Procurement+ >> > > > > > > > > Suite) >> > > > > > > > > > > > allow you to control access to dependencies without >> > > > manually >> > > > > > > > putting >> > > > > > > > > > jars >> > > > > > > > > > > > in an SCM. >> > > > > > > > > > > > >> > > > > > > > > > > > I realize this doesn't help you right now, just >> thought >> > > I'd >> > > > > > > mention >> > > > > > > > > it >> > > > > > > > > > so >> > > > > > > > > > > > you don't get the idea that I'm advocating a build >> > > process >> > > > > that >> > > > > > > > > blindly >> > > > > > > > > > > > includes new dependencies. >> > > > > > > > > > > > >> > > > > > > > > > > > 2017-11-19 13:05 GMT+01:00 Alexandre Vermeerbergen < >> > > > > > > > > > > > avermeerber...@gmail.com >> > > > > > > > > > > > >: >> > > > > > > > > > > > >> > > > > > > > > > > > > Believe it or not, we don't internally use Maven : >> we >> > > > > > strictly >> > > > > > > > > > control >> > > > > > > > > > > > our >> > > > > > > > > > > > > dependencies. >> > > > > > > > > > > > > >> > > > > > > > > > > > > So okay, I have delivered a >> "storm-commons-lang2.jar" >> > > in >> > > > > our >> > > > > > > SCM >> > > > > > > > > > system >> > > > > > > > > > > > in >> > > > > > > > > > > > > order to specifically add it to our topologies big >> > jars >> > > > > > > depending >> > > > > > > > > on >> > > > > > > > > > > > > storm-kafka-client (feeling sad). >> > > > > > > > > > > > > >> > > > > > > > > > > > > I'll tell you the outcome ASAP. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Best regard, >> > > > > > > > > > > > > Alexandre Vermeerbergen >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > 2017-11-19 12:57 GMT+01:00 Stig Rohde Døssing < >> > > > > > > > > > stigdoess...@gmail.com >> > > > > > > > > > > >: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > I also don't think that adding dependencies >> > > > constitutes a >> > > > > > > > > breaking >> > > > > > > > > > > > > change, >> > > > > > > > > > > > > > nor should it. Dependency management tooling like >> > > Maven >> > > > > > will >> > > > > > > > > handle >> > > > > > > > > > > > > pulling >> > > > > > > > > > > > > > the right dependencies automatically. >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > 2017-11-19 12:34 GMT+01:00 Jungtaek Lim < >> > > > > kabh...@gmail.com >> > > > > > >: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Alexandre, >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > There's so much pain on using relocated >> artifact >> > in >> > > > > other >> > > > > > > > > module >> > > > > > > > > > > in a >> > > > > > > > > > > > > > > project. You need to refer the class as >> relocated >> > > > name, >> > > > > > and >> > > > > > > > > your >> > > > > > > > > > > IDE >> > > > > > > > > > > > > (at >> > > > > > > > > > > > > > > least IntelliJ) complains and the compilation >> in >> > > IDE >> > > > > will >> > > > > > > not >> > > > > > > > > > > > succeed. >> > > > > > > > > > > > > > (The >> > > > > > > > > > > > > > > build via Maven will succeed though.) >> > > > > > > > > > > > > > > There was same issue on 2.0.0 as well, and for >> > now >> > > we >> > > > > > don't >> > > > > > > > > shade >> > > > > > > > > > > > > > anything >> > > > > > > > > > > > > > > in 2.0.0. We need to discuss which things to >> > shade >> > > > > before >> > > > > > > > > > > discussing >> > > > > > > > > > > > > > about >> > > > > > > > > > > > > > > releasing, indeed. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > And normally we have been allowing adding >> > > dependency >> > > > as >> > > > > > > minor >> > > > > > > > > > > > release. >> > > > > > > > > > > > > I >> > > > > > > > > > > > > > > understand someone could say that it is >> breaking >> > > > > > > dependency, >> > > > > > > > > and >> > > > > > > > > > > > should >> > > > > > > > > > > > > > > lead another relocation as well. But my 2 cents >> > the >> > > > > > > > requirement >> > > > > > > > > > is >> > > > > > > > > > > > too >> > > > > > > > > > > > > > > restrictive, and may end up with Guava-like >> > > > versioning. >> > > > > > > > > > (releasing >> > > > > > > > > > > > > major >> > > > > > > > > > > > > > > versions sooner...) >> > > > > > > > > > > > > > > I think relocating is not an ideal solution, >> and >> > > > > > eventually >> > > > > > > > we >> > > > > > > > > > may >> > > > > > > > > > > > have >> > > > > > > > > > > > > > > nice solution (like classloader isolation) and >> > then >> > > > we >> > > > > > > could >> > > > > > > > > sort >> > > > > > > > > > > out >> > > > > > > > > > > > > all >> > > > > > > > > > > > > > > the related issues. Before that, I could not >> > block >> > > > > adding >> > > > > > > > > > > > dependencies >> > > > > > > > > > > > > in >> > > > > > > > > > > > > > > minor release if it is necessary. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > > > Jungtaek Lim (HeartSaVioR) >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 2017년 11월 19일 (일) 오후 8:02, Stig Rohde Døssing < >> > > > > > > > > > > > stigdoess...@gmail.com >> > > > > > > > > > > > > >님이 >> > > > > > > > > > > > > > > 작성: >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Alexandre, >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Thanks for trying. I think this time it's >> not a >> > > > > problem >> > > > > > > > with >> > > > > > > > > > > > > > > > storm-kafka-client. It has a dependency on >> > > > > commons-lang >> > > > > > > > 2.5, >> > > > > > > > > > > which >> > > > > > > > > > > > is >> > > > > > > > > > > > > > > > declared in the storm-kafka-client pom. We >> > > usually >> > > > > > > > recommend >> > > > > > > > > > that >> > > > > > > > > > > > > > people >> > > > > > > > > > > > > > > > use Maven/Gradle/Ivy or similar systems to >> > ensure >> > > > > that >> > > > > > > they >> > > > > > > > > get >> > > > > > > > > > > all >> > > > > > > > > > > > > > > > dependencies when they build their >> topologies. >> > In >> > > > > such >> > > > > > a >> > > > > > > > case >> > > > > > > > > > > Maven >> > > > > > > > > > > > > (or >> > > > > > > > > > > > > > > > similar) would see that your topology depends >> > on >> > > > > > > > > > > > storm-kafka-client, >> > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > also download and include commons-lang in the >> > > > > topology >> > > > > > > jar. >> > > > > > > > > > When >> > > > > > > > > > > > you >> > > > > > > > > > > > > > put >> > > > > > > > > > > > > > > > together dependencies manually you need to >> > > include >> > > > > > > > > commons-lang >> > > > > > > > > > > > > > manually >> > > > > > > > > > > > > > > as >> > > > > > > > > > > > > > > > well. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > You can get the commons-lang 2.5 jar here >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > http://central.maven.org/ >> > > > > maven2/commons-lang/commons- >> > > > > > > > > > > > > > > lang/2.5/commons-lang-2.5.jar >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > 2017-11-19 11:49 GMT+01:00 Alexandre >> > > Vermeerbergen >> > > > < >> > > > > > > > > > > > > > > > avermeerber...@gmail.com >> > > > > > > > > > > > > > > > >: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Hello Stig, >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Than you very much for you quick answer & >> > fix. >> > > > > > > > > > > > > > > > > Unfortunaly, I still have topologies >> failing >> > to >> > > > > start >> > > > > > > but >> > > > > > > > > > with >> > > > > > > > > > > a >> > > > > > > > > > > > > > > > different >> > > > > > > > > > > > > > > > > exception this time: >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Exception in thread "main" java.lang. >> > > > > > > > NoClassDefFoundError: >> > > > > > > > > > > > > > > > > org/apache/commons/lang/StringUtils >> > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > org.apache.storm.kafka.spout. >> > NamedTopicFilter. >> > > > > > > > > > getTopicsString( >> > > > > > > > > > > > > > > > > NamedTopicFilter.java:66) >> > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > org.apache.storm.kafka.spout. >> > > > > > > > ManualPartitionSubscription. >> > > > > > > > > > > > > > > getTopicsString( >> > > > > > > > > > > > > > > > > ManualPartitionSubscription.java:70) >> > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > com.acme.storm.evaluator. >> > > > > > spout.BasicKafkaSpout.<init>( >> > > > > > > > > > > > > > > > > BasicKafkaSpout.java:59) >> > > > > > > > > > > > > > > > > at >> > > > > > > com.acme.storm.util.StormUtils.getKafkaSpout( >> > > > > > > > > > > > > > > > > StormUtils.java:888) >> > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > com.acme.storm.evaluator. >> > > > > > SLAEventsInterceptorTopology. >> > > > > > > > > main( >> > > > > > > > > > > > > > > > > SLAEventsInterceptorTopology.java:120) >> > > > > > > > > > > > > > > > > Caused by: java.lang. >> ClassNotFoundException: >> > > > > > > > > > > > > > > > > org.apache.commons.lang.StringUtils >> > > > > > > > > > > > > > > > > at java.net.URLClassLoader. >> > > > > > > > > > > > findClass(URLClassLoader.java: >> > > > > > > > > > > > > > 381) >> > > > > > > > > > > > > > > > > at java.lang.ClassLoader. >> > > > > > > > > loadClass(ClassLoader.java: >> > > > > > > > > > > 424) >> > > > > > > > > > > > > > > > > at sun.misc.Launcher$ >> > > > > > AppClassLoader.loadClass( >> > > > > > > > > > > > > > > Launcher.java:338) >> > > > > > > > > > > > > > > > > at java.lang.ClassLoader. >> > > > > > > > > loadClass(ClassLoader.java: >> > > > > > > > > > > 357) >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Looks like you introduced a dependency on >> > > commons >> > > > > > lang >> > > > > > > > that >> > > > > > > > > > > > wasn't >> > > > > > > > > > > > > > part >> > > > > > > > > > > > > > > > of >> > > > > > > > > > > > > > > > > Storm, isn't it ? >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Keep faith, next time will be better :) >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Best regards, >> > > > > > > > > > > > > > > > > Alexandre >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > 2017-11-19 10:44 GMT+01:00 Stig Rohde >> > Døssing < >> > > > > > > > > > > > > > stigdoess...@gmail.com >> > > > > > > > > > > > > > > >: >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > I've put up a fix here >> > > > > https://github.com/apache/ >> > > > > > > > > > > > storm/pull/2426 >> > > > > > > > > > > > > . >> > > > > > > > > > > > > > > > > There's >> > > > > > > > > > > > > > > > > > an updated storm-kafka-client jar at >> > > > > > > > > > > > > > > > > > https://drive.google.com/file/d/ >> > > > > > > > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_ >> > > > > > > > > > > > > > > > > > v9fd/view?usp=sharing >> > > > > > > > > > > > > > > > > > if you'd like to try it out. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde >> > > Døssing < >> > > > > > > > > > > > > > > stigdoess...@gmail.com >> > > > > > > > > > > > > > > > >: >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Oops. The exception is not intentional, >> > > it's >> > > > a >> > > > > > bug. >> > > > > > > > In >> > > > > > > > > > > 1.2.0 >> > > > > > > > > > > > we >> > > > > > > > > > > > > > > check >> > > > > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > > > > > the "enable.auto.commit" key in the >> > > > > > > kafkaConsumerProp >> > > > > > > > > > map, >> > > > > > > > > > > > and >> > > > > > > > > > > > > if >> > > > > > > > > > > > > > > it >> > > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > set >> > > > > > > > > > > > > > > > > > > we warn in the log that it shouldn't >> be, >> > > > > because >> > > > > > > > users >> > > > > > > > > > > should >> > > > > > > > > > > > > use >> > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > KafkaSpoutConfig.Builder. >> > > > > setProcessingGuarantee >> > > > > > > > method >> > > > > > > > > > > > > instead. >> > > > > > > > > > > > > > > When >> > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > property is set we try to set the >> > > processing >> > > > > > > > guarantee >> > > > > > > > > to >> > > > > > > > > > > > match >> > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > pre-1.2.0 behavior, but I made a >> mistake >> > > and >> > > > > > > assumed >> > > > > > > > > the >> > > > > > > > > > > > > property >> > > > > > > > > > > > > > > key >> > > > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > a >> > > > > > > > > > > > > > > > > > > boolean when it might actually be >> either >> > a >> > > > > > boolean >> > > > > > > > or a >> > > > > > > > > > > > string. >> > > > > > > > > > > > > > > I'll >> > > > > > > > > > > > > > > > > put >> > > > > > > > > > > > > > > > > > up >> > > > > > > > > > > > > > > > > > > a fix ASAP. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 >> > > GMT+01:00 >> > > > > > > > > Alexandre >> > > > > > > > > > > > > > > Vermeerbergen >> > > > > > > > > > > > > > > > > < >> > > > > > > > > > > > > > > > > > > avermeerber...@gmail.com>: >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> Hello Stig, >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> Here's my first feedback on this Storm >> > > 1.2.0 >> > > > > > > preview >> > > > > > > > > on >> > > > > > > > > > my >> > > > > > > > > > > > > > > > Supervision >> > > > > > > > > > > > > > > > > > >> system based on Storm : I have 11 >> > > topologies >> > > > > KO >> > > > > > > (not >> > > > > > > > > > even >> > > > > > > > > > > > able >> > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > start), >> > > > > > > > > > > > > > > > > > >> and 4 topologies which seem to be >> > > > unaffected. >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> Details: >> > > > > > > > > > > > > > > > > > >> - I used the binaries posted on >> dropbox >> > by >> > > > > > > Jungteak >> > > > > > > > > and >> > > > > > > > > > > your >> > > > > > > > > > > > > > > binary >> > > > > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > > > > >> storm-kafka-client >> > > > > > > > > > > > > > > > > > >> - Rebuilt all our topologies using >> these >> > > > > > > > > storm-core.jar >> > > > > > > > > > & >> > > > > > > > > > > > > > > > > > >> storm-kafka-client.jar from these >> > > > > > > > storm-1.2.0-snapshot >> > > > > > > > > > > (and >> > > > > > > > > > > > I >> > > > > > > > > > > > > > got >> > > > > > > > > > > > > > > > > > >> absolutely no error at build time). >> > > > > > > > > > > > > > > > > > >> Meaning that I have new "big jar" >> > files >> > > > > > > including >> > > > > > > > > this >> > > > > > > > > > > > newer >> > > > > > > > > > > > > > > > > > >> storm-kafka-client.jar (except for 1 >> > > > topology >> > > > > > > which >> > > > > > > > > > > depends >> > > > > > > > > > > > on >> > > > > > > > > > > > > > > > another >> > > > > > > > > > > > > > > > > > >> build system) >> > > > > > > > > > > > > > > > > > >> - The 11 topologies which fail to >> start >> > > show >> > > > > up >> > > > > > > the >> > > > > > > > > > > > following >> > > > > > > > > > > > > > > trace >> > > > > > > > > > > > > > > > > when >> > > > > > > > > > > > > > > > > > >> they are submitted: >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> Running: java -client -Ddaemon.name= >> > > > > > > > -Dstorm.options= >> > > > > > > > > > > > > > > > > > >> -Dstorm.home=/usr/local/Storm/ >> > > storm-stable >> > > > > > > > > > > > > > > > > > >> -Dstorm.log.dir=/usr/local/ >> > > > > > Storm/storm-stable/logs >> > > > > > > > > > > > > > > > > > >> -Djava.library.path=/usr/ >> > > > > > > > > local/lib:/opt/local/lib:/usr/ >> > > > > > > > > > > lib >> > > > > > > > > > > > > > > > > > >> -Dstorm.conf.file= -cp >> > > > > > > > > > > > > > > > > > >> /usr/local/Storm/storm-stable/ >> > > > > > > > > > > > *:/usr/local/Storm/storm-stabl >> > > > > > > > > > > > > > > > > > >> e/lib/*:/usr/local/Storm/ >> > > > > > > > storm-stable/extlib/*:/usr/ >> > > > > > > > > > > local/ >> > > > > > > > > > > > > > > > > > >> Storm/StormTopologiesBJ.jar:/ >> > > > > > > > > > > usr/local/Storm/storm-stable/ >> > > > > > > > > > > > > > > > > > >> conf:/usr/local/Storm/storm- >> stable/bin >> > > > > > > > > > > > > > > > > > >> -Dstorm.jar=/usr/local/Storm/ >> > > > > > StormTopologiesBJ.jar >> > > > > > > > > > > > > > > > > > >> -Dstorm.dependency.jars= >> > > > > > > > > > > > > > > > > > >> -Dstorm.dependency.artifacts={} >> > > > > > > > > > > > > > > > > > >> com.acme.storm.evaluator. >> > > > > > > > SLAEventsInterceptorTopology >> > > > > > > > > > > > > > > > > > >> slaEventsInterceptor >> > > > > > > > > > > > > > > > > > >> Latest >> > > > > > > > > > > > > > > > > > >> 1022 [main] INFO c.a.s.e. >> > > > > > > > > SLAEventsInterceptorTopology >> > > > > > > > > > - >> > > > > > > > > > > > > > > Topology : >> > > > > > > > > > > > > > > > > > >> 'slaEventsInterceptor', >> > > > > KafkaOffsetTimeStrategy >> > > > > > = >> > > > > > > > > > 'Latest' >> > > > > > > > > > > > > > > > > > >> 1026 [main] WARN c.a.s.u.StormUtils - >> > > > > Couldn't >> > > > > > > read >> > > > > > > > > > > > > > > > > > >> /usr/local/Storm/flux/ >> > > slaEventsInterceptor_ >> > > > > > > > > > > > > > > kafkaConsumer.properties >> > > > > > > > > > > > > > > > > > file: >> > > > > > > > > > > > > > > > > > >> /usr/local/Storm/flux/ >> > > slaEventsInterceptor_ >> > > > > > > > > > > > > > > kafkaConsumer.properties >> > > > > > > > > > > > > > > > > (No >> > > > > > > > > > > > > > > > > > >> such file or directory) >> > > > > > > > > > > > > > > > > > >> 1037 [main] WARN >> > > > o.a.s.k.s.KafkaSpoutConfig - >> > > > > > Do >> > > > > > > > not >> > > > > > > > > > set >> > > > > > > > > > > > > > > > > > >> enable.auto.commit manually. Instead >> use >> > > > > > > > > > > > > > > > > > >> KafkaSpoutConfig.Builder. >> > > > > > setProcessingGuarantee. >> > > > > > > > This >> > > > > > > > > > > will >> > > > > > > > > > > > be >> > > > > > > > > > > > > > > > treated >> > > > > > > > > > > > > > > > > > as >> > > > > > > > > > > > > > > > > > >> an >> > > > > > > > > > > > > > > > > > >> error in the next major release. For >> now >> > > the >> > > > > > spout >> > > > > > > > > will >> > > > > > > > > > be >> > > > > > > > > > > > > > > > configured >> > > > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > >> behave like it would have in pre-1.2.0 >> > > > > releases. >> > > > > > > > > > > > > > > > > > >> Exception in thread "main" >> > > > > > > > > java.lang.ClassCastException: >> > > > > > > > > > > > > > > > > > java.lang.String >> > > > > > > > > > > > > > > > > > >> cannot be cast to java.lang.Boolean >> > > > > > > > > > > > > > > > > > >> at >> > > > > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout. >> > > > > KafkaSpoutConfig. >> > > > > > > > > > > > setAutoCommitM >> > > > > > > > > > > > > > > > > > >> ode(KafkaSpoutConfig.java:721) >> > > > > > > > > > > > > > > > > > >> at >> > > > > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout. >> > > > > > > > KafkaSpoutConfig.<init>( >> > > > > > > > > > > > KafkaSp >> > > > > > > > > > > > > > > > > > >> outConfig.java:97) >> > > > > > > > > > > > > > > > > > >> at >> > > > > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout. >> > > > > > > > > KafkaSpoutConfig$Builder. >> > > > > > > > > > > > build( >> > > > > > > > > > > > > > > > > > >> KafkaSpoutConfig.java:671) >> > > > > > > > > > > > > > > > > > >> at >> > > > > > > > > > > > > > > > > > >> com.acme.storm.evaluator. >> > > > > > > > > SLAEventsInterceptorTopology. >> > > > > > > > > > > > main(S >> > > > > > > > > > > > > > > > > > >> LAEventsInterceptorTopology.java:118) >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> The line part of our application which >> > > > > triggered >> > > > > > > > above >> > > > > > > > > > > stack >> > > > > > > > > > > > > > trace >> > > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > >> following one: >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> KafkaSpoutConfig<String, >> Event> >> > > > > > > > > > > > > spoutConfigForMainTopic >> > > > > > > > > > > > > > = >> > > > > > > > > > > > > > > > > > >> KafkaSpoutConfig >> > > > > > > > > > > > > > > > > > >> >> > > > .builder(elasticKafkaBrokers, >> > > > > > > > > > > > > KafkaTopics.MAIN) >> > > > > > > > > > > > > > > > > > >> .setValue( >> > > > > > > > > EventKafkaDeserializer.class) >> > > > > > > > > > > > > > > > > > >> >> .setGroupId(consumerId + >> > > > "_" + >> > > > > > > > > > > > > KafkaTopics.MAIN) >> > > > > > > > > > > > > > > > > > >> >> > > .setFirstPollOffsetStrategy( >> > > > > > > > strategy) >> > > > > > > > > > > > > > > > > > >> >> > > .setProp(kafkaConsumerProp) >> > > > > > > > > > > > > > > > > > >> .setRecordTranslator( >> > > > > > > > > > > > > > > > > > >> new >> > > > > > > > > > SLAEventsInterceptorKafkaRecor >> > > > > > > > > > > > > > > > > > dTranslator()) >> > > > > > > > > > > > > > > > > > >> .build(); >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> I understand that there's a WARN >> > message, >> > > > but >> > > > > if >> > > > > > > > it's >> > > > > > > > > a >> > > > > > > > > > > > > warning, >> > > > > > > > > > > > > > > > then >> > > > > > > > > > > > > > > > > > why >> > > > > > > > > > > > > > > > > > >> do we exit with such a brutal >> exception? >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> Best regards, >> > > > > > > > > > > > > > > > > > >> Alexandre Vermeerbergen >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde >> > > > Døssing >> > > > > < >> > > > > > > > > > > > > > > > stigdoess...@gmail.com >> > > > > > > > > > > > > > > > > >: >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > >> > Alexandre, >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > I've uploaded the 1.2.0-SNAPSHOT >> > > > > > > > storm-kafka-client >> > > > > > > > > > jar >> > > > > > > > > > > > here >> > > > > > > > > > > > > > > > > > >> > https://drive.google.com/file/d/ >> > > > > > > > > > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_ >> > > > > > > > > > > > > > > > > > >> > v9fd/view?usp=sharing. >> > > > > > > > > > > > > > > > > > >> > You can probably keep the 1.1.0 >> > versions >> > > > of >> > > > > > > > > > storm-hbase >> > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > storm-hdfs, >> > > > > > > > > > > > > > > > > > >> > since the core Storm API hasn't >> > changed. >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > If you want to build the jars >> > yourself, >> > > > > clone >> > > > > > > the >> > > > > > > > > > > > repository >> > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > > >> > https://github.com/apache/storm and >> > > check >> > > > > out >> > > > > > > the >> > > > > > > > > > > > > 1.x-branch >> > > > > > > > > > > > > > > > > branch. >> > > > > > > > > > > > > > > > > > >> You >> > > > > > > > > > > > > > > > > > >> > can build the entire project by >> > running >> > > > "mvn >> > > > > > > clean >> > > > > > > > > > > install >> > > > > > > > > > > > > > > > > > -DskipTests" >> > > > > > > > > > > > > > > > > > >> in >> > > > > > > > > > > > > > > > > > >> > the root. The individual module jars >> > > will >> > > > be >> > > > > > > > > available >> > > > > > > > > > > in >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > target >> > > > > > > > > > > > > > > > > > >> > directories for each module, e.g. >> > > > > > > > > > storm-kafka-client.jar >> > > > > > > > > > > > > will >> > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > > >> > external/storm-kafka-client/target. >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > If you need to build the regular >> Storm >> > > > > > > > distribution >> > > > > > > > > > (the >> > > > > > > > > > > > > > tarball >> > > > > > > > > > > > > > > > > > >> Jungtaek >> > > > > > > > > > > > > > > > > > >> > sent you), you can go to >> > > storm-dist/binary >> > > > > and >> > > > > > > run >> > > > > > > > > > "mvn >> > > > > > > > > > > > > clean >> > > > > > > > > > > > > > > > > install >> > > > > > > > > > > > > > > > > > >> > -Dgpg.skip". The tar/zip will be in >> > > > > > > > > > > > > storm-dist/binary/target. >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > Further reference in case you need >> it >> > > > > > > > > > > > > > > > > > >> > https://github.com/apache/ >> > > > > > > > > storm/blob/master/DEVELOPER >> > > > > > > > > > . >> > > > > > > > > > > > > > > > > > >> > md#build-the-code-and-run-the-tests >> > > > > > > > > > > > > > > > > > >> > . >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre >> > > > > > > > Vermeerbergen < >> > > > > > > > > > > > > > > > > > >> > avermeerber...@gmail.com >> > > > > > > > > > > > > > > > > > >> > >: >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > > Hello Jungtaek, >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > > Thanks for the link to 1.2.0 >> preview >> > > > > > binaries. >> > > > > > > > > > > > > > > > > > >> > > However, since we heavily depend >> on >> > > > Storm >> > > > > > > Kafka >> > > > > > > > > > > Client, >> > > > > > > > > > > > > > would >> > > > > > > > > > > > > > > > you >> > > > > > > > > > > > > > > > > > >> please >> > > > > > > > > > > > > > > > > > >> > > either add it or remind me how to >> > > build >> > > > > the >> > > > > > > jars >> > > > > > > > > of >> > > > > > > > > > > this >> > > > > > > > > > > > > > > > external >> > > > > > > > > > > > > > > > > > >> lib, at >> > > > > > > > > > > > > > > > > > >> > > same "snapshot" version as the >> rest >> > ? >> > > > > > > > > > > > > > > > > > >> > > On a side note, some of our >> > topologies >> > > > > also >> > > > > > > rely >> > > > > > > > > on >> > > > > > > > > > > > storm >> > > > > > > > > > > > > > > hbase >> > > > > > > > > > > > > > > > & >> > > > > > > > > > > > > > > > > > >> storm >> > > > > > > > > > > > > > > > > > >> > > hdfs : does it matters if these >> > later >> > > > ones >> > > > > > > stay >> > > > > > > > in >> > > > > > > > > > > 1.1.0 >> > > > > > > > > > > > > > > version >> > > > > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > > > > >> this >> > > > > > > > > > > > > > > > > > >> > > test? >> > > > > > > > > > > > > > > > > > >> > > Indeed, I want to focus on Storm >> > 1.2.0 >> > > > > > "core" >> > > > > > > + >> > > > > > > > > > Storm >> > > > > > > > > > > > > Kafka >> > > > > > > > > > > > > > > > Client >> > > > > > > > > > > > > > > > > > >> 1.2.0 >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > > Best regards, >> > > > > > > > > > > > > > > > > > >> > > Alexandre Vermeerbergen >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > > 2017-11-18 <20%2017%2011%2018> >> 7:41 >> > > > > > GMT+01:00 >> > > > > > > > > > Jungtaek >> > > > > > > > > > > > > Lim < >> > > > > > > > > > > > > > > > > > >> kabh...@gmail.com>: >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > > > Alexandre, >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > https://www.dropbox.com/s/ >> > > > > > > > > > > > mg2gnunk24oesyc/apache-storm- >> > > > > > > > > > > > > > > > > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0 >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > Above link is custom binary >> > > > distribution >> > > > > > of >> > > > > > > > > > current >> > > > > > > > > > > > > > > 1.x-branch >> > > > > > > > > > > > > > > > > > >> > (SNAPSHOT >> > > > > > > > > > > > > > > > > > >> > > of >> > > > > > > > > > > > > > > > > > >> > > > 1.2.0). Could you run the test >> in >> > > your >> > > > > > > > > environment >> > > > > > > > > > > > first >> > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > let >> > > > > > > > > > > > > > > > > > us >> > > > > > > > > > > > > > > > > > >> > know >> > > > > > > > > > > > > > > > > > >> > > > about the result? Regardless of >> > > > > including >> > > > > > > > > metrics >> > > > > > > > > > V2 >> > > > > > > > > > > > to >> > > > > > > > > > > > > > > 1.2.0 >> > > > > > > > > > > > > > > > or >> > > > > > > > > > > > > > > > > > >> not, >> > > > > > > > > > > > > > > > > > >> > > your >> > > > > > > > > > > > > > > > > > >> > > > test report should be valuable >> for >> > > us. >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > And please refer my analysis of >> > > > current >> > > > > > > > metrics: >> > > > > > > > > > > > > > > > > > >> > > > https://cwiki.apache.org/ >> > > > > > > > > > confluence/display/STORM/ >> > > > > > > > > > > > > > > > > > >> > > Limitations+of+current+ >> > > > > > > > > > > > > > > > > > >> > > > metrics+feature >> > > > > > > > > > > > > > > > > > >> > > > to >> > > > > > > > > > > > > > > > > > >> > > > see why we want to move toward >> to >> > > > > Metrics >> > > > > > > V2. >> > > > > > > > > > > Current >> > > > > > > > > > > > PR >> > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > > Metrics >> > > > > > > > > > > > > > > > > > >> V2 >> > > > > > > > > > > > > > > > > > >> > is >> > > > > > > > > > > > > > > > > > >> > > > for initial state and it could >> > cover >> > > > > only >> > > > > > > some >> > > > > > > > > of >> > > > > > > > > > > > issues >> > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > list, >> > > > > > > > > > > > > > > > > > >> but I >> > > > > > > > > > > > > > > > > > >> > > > expect we will address another >> > > issues >> > > > as >> > > > > > > well >> > > > > > > > > > based >> > > > > > > > > > > on >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > > PR. >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > Thanks in advance! >> > > > > > > > > > > > > > > > > > >> > > > Jungtaek Lim (HeartSaVioR) >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, >> > Alexandre >> > > > > > > > > > Vermeerbergen < >> > > > > > > > > > > > > > > > > > >> > > > avermeerber...@gmail.com>님이 >> > > > > > > > > > > > > > > > > > >> > > > 작성: >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > > > Hello Hugo, >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > As I already posted, I was >> > getting >> > > > > ready >> > > > > > > to >> > > > > > > > > test >> > > > > > > > > > > the >> > > > > > > > > > > > > > > > upcoming >> > > > > > > > > > > > > > > > > > RC, >> > > > > > > > > > > > > > > > > > >> so >> > > > > > > > > > > > > > > > > > >> > > > yeah : >> > > > > > > > > > > > > > > > > > >> > > > > as soon as I have some >> binaries >> > > for >> > > > > > > testing, >> > > > > > > > > I >> > > > > > > > > > > can >> > > > > > > > > > > > > run >> > > > > > > > > > > > > > > any >> > > > > > > > > > > > > > > > > > Storm >> > > > > > > > > > > > > > > > > > >> > 1.2.0 >> > > > > > > > > > > > > > > > > > >> > > > > preview on my pre-production >> > (well >> > > > > > > stressed) >> > > > > > > > > > > > > > environment. >> > > > > > > > > > > > > > > Is >> > > > > > > > > > > > > > > > > > there >> > > > > > > > > > > > > > > > > > >> > some >> > > > > > > > > > > > > > > > > > >> > > > URL >> > > > > > > > > > > > > > > > > > >> > > > > from where I can download such >> > > > > > "snapshot" >> > > > > > > > > > > binaries? >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Best regards, >> > > > > > > > > > > > > > > > > > >> > > > > Alexandre Vermeerbergen >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > PS: By the way, I have no >> > interest >> > > > in >> > > > > > the >> > > > > > > > > > > > > > metrics-related >> > > > > > > > > > > > > > > > > > >> > enhancements >> > > > > > > > > > > > > > > > > > >> > > > you >> > > > > > > > > > > > > > > > > > >> > > > > mentionned - I can understand >> it >> > > is >> > > > of >> > > > > > > > > interest >> > > > > > > > > > > for >> > > > > > > > > > > > > > people >> > > > > > > > > > > > > > > > > aware >> > > > > > > > > > > > > > > > > > >> of >> > > > > > > > > > > > > > > > > > >> > it >> > > > > > > > > > > > > > > > > > >> > > > and >> > > > > > > > > > > > > > > > > > >> > > > > who are expecting "something" >> > > > related >> > > > > to >> > > > > > > it >> > > > > > > > - >> > > > > > > > > > but >> > > > > > > > > > > I >> > > > > > > > > > > > > > would >> > > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > sad >> > > > > > > > > > > > > > > > > > >> to >> > > > > > > > > > > > > > > > > > >> > > see a >> > > > > > > > > > > > > > > > > > >> > > > > postponing of the imminent >> Storm >> > > > 1.2.0 >> > > > > > > > release >> > > > > > > > > > for >> > > > > > > > > > > > > > > something >> > > > > > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > > > do >> > > > > > > > > > > > > > > > > > >> > not >> > > > > > > > > > > > > > > > > > >> > > > use >> > > > > > > > > > > > > > > > > > >> > > > > at all. >> > > > > > > > > > > > > > > > > > >> > > > > PPS: I would love to learn >> more >> > > > about >> > > > > > > these >> > > > > > > > > > > > "metrics", >> > > > > > > > > > > > > > > maybe >> > > > > > > > > > > > > > > > > > it's >> > > > > > > > > > > > > > > > > > >> > > > something >> > > > > > > > > > > > > > > > > > >> > > > > worth which never quite >> > > > understood... >> > > > > > any >> > > > > > > > link >> > > > > > > > > > > with >> > > > > > > > > > > > > > clear >> > > > > > > > > > > > > > > > > > >> explanation >> > > > > > > > > > > > > > > > > > >> > > of >> > > > > > > > > > > > > > > > > > >> > > > > this feature? >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 >> Hugo >> > Da >> > > > > Cruz >> > > > > > > > Louro >> > > > > > > > > < >> > > > > > > > > > > > > > > > > > >> > hlo...@hortonworks.com >> > > > > > > > > > > > > > > > > > >> > > >: >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > I am also in agreement that >> we >> > > > > should >> > > > > > > not >> > > > > > > > > > delay >> > > > > > > > > > > > the >> > > > > > > > > > > > > > > 1.2.0 >> > > > > > > > > > > > > > > > > > >> release >> > > > > > > > > > > > > > > > > > >> > for >> > > > > > > > > > > > > > > > > > >> > > > too >> > > > > > > > > > > > > > > > > > >> > > > > > long, but in order to >> release >> > > it a >> > > > > few >> > > > > > > > > things >> > > > > > > > > > > need >> > > > > > > > > > > > > to >> > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > kept >> > > > > > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > > >> > > mind: >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > - If we want to avoid >> > releasing >> > > > > 1.3.0 >> > > > > > > > soon >> > > > > > > > > or >> > > > > > > > > > > at >> > > > > > > > > > > > > all, >> > > > > > > > > > > > > > > > that >> > > > > > > > > > > > > > > > > > >> means >> > > > > > > > > > > > > > > > > > >> > > that >> > > > > > > > > > > > > > > > > > >> > > > we >> > > > > > > > > > > > > > > > > > >> > > > > > should include in 1.2.0 as >> > many >> > > > > > > important >> > > > > > > > > > > features >> > > > > > > > > > > > > > (e.g. >> > > > > > > > > > > > > > > > > > >> metricsV2) >> > > > > > > > > > > > > > > > > > >> > > as >> > > > > > > > > > > > > > > > > > >> > > > > > possible within a reasonable >> > > time >> > > > > > frame. >> > > > > > > > > Also >> > > > > > > > > > to >> > > > > > > > > > > > > avoid >> > > > > > > > > > > > > > > as >> > > > > > > > > > > > > > > > > much >> > > > > > > > > > > > > > > > > > >> back >> > > > > > > > > > > > > > > > > > >> > > > > porting >> > > > > > > > > > > > > > > > > > >> > > > > > as possible one should >> really >> > > get >> > > > > all >> > > > > > > > > features >> > > > > > > > > > > > that >> > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > > foresee >> > > > > > > > > > > > > > > > > > >> we >> > > > > > > > > > > > > > > > > > >> > > want >> > > > > > > > > > > > > > > > > > >> > > > to >> > > > > > > > > > > > > > > > > > >> > > > > > maintain in 1.2.0 right >> away, >> > > and >> > > > > then >> > > > > > > > > simply >> > > > > > > > > > > > > maintain >> > > > > > > > > > > > > > > > them >> > > > > > > > > > > > > > > > > > >> there. >> > > > > > > > > > > > > > > > > > >> > > > > However >> > > > > > > > > > > > > > > > > > >> > > > > > it’s a fact of life that if >> a >> > > bug >> > > > is >> > > > > > > found >> > > > > > > > > and >> > > > > > > > > > > it >> > > > > > > > > > > > > is a >> > > > > > > > > > > > > > > > > > blocker, >> > > > > > > > > > > > > > > > > > >> it >> > > > > > > > > > > > > > > > > > >> > > will >> > > > > > > > > > > > > > > > > > >> > > > > > have to be back ported. >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > - STORM-2153 details that >> the >> > > new >> > > > > > > metrics >> > > > > > > > > > > > > requirements >> > > > > > > > > > > > > > > > were >> > > > > > > > > > > > > > > > > > >> driven >> > > > > > > > > > > > > > > > > > >> > by >> > > > > > > > > > > > > > > > > > >> > > > the >> > > > > > > > > > > > > > > > > > >> > > > > > users. Therefore if this >> > feature >> > > > is >> > > > > > > highly >> > > > > > > > > > > > important >> > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > sought >> > > > > > > > > > > > > > > > > > >> > > after, >> > > > > > > > > > > > > > > > > > >> > > > it >> > > > > > > > > > > > > > > > > > >> > > > > > may not make much sense to >> > > release >> > > > > > 1.2.0 >> > > > > > > > > > without >> > > > > > > > > > > > it. >> > > > > > > > > > > > > > If >> > > > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > > do >> > > > > > > > > > > > > > > > > > >> > release >> > > > > > > > > > > > > > > > > > >> > > > > 1.2.0 >> > > > > > > > > > > > > > > > > > >> > > > > > without the metrics, does it >> > > mean >> > > > > that >> > > > > > > > > metrics >> > > > > > > > > > > > will >> > > > > > > > > > > > > go >> > > > > > > > > > > > > > > > only >> > > > > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > > >> > 2.0? I >> > > > > > > > > > > > > > > > > > >> > > > > would >> > > > > > > > > > > > > > > > > > >> > > > > > conjecture that most >> > production >> > > > > > > > deployments >> > > > > > > > > > will >> > > > > > > > > > > > > take >> > > > > > > > > > > > > > a >> > > > > > > > > > > > > > > > > while >> > > > > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > >> > > > upgrade >> > > > > > > > > > > > > > > > > > >> > > > > to >> > > > > > > > > > > > > > > > > > >> > > > > > 2.0 even after it is >> released. >> > > > That >> > > > > > > means >> > > > > > > > > that >> > > > > > > > > > > > they >> > > > > > > > > > > > > > will >> > > > > > > > > > > > > > > > > still >> > > > > > > > > > > > > > > > > > >> be >> > > > > > > > > > > > > > > > > > >> > > > running >> > > > > > > > > > > > > > > > > > >> > > > > > Storm without the benefit of >> > the >> > > > new >> > > > > > > > > metrics. >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > - Several fixes have got in >> > > > > > > > > > storm-kafka-client. >> > > > > > > > > > > > > There >> > > > > > > > > > > > > > > > have >> > > > > > > > > > > > > > > > > > >> been a >> > > > > > > > > > > > > > > > > > >> > > lot >> > > > > > > > > > > > > > > > > > >> > > > of >> > > > > > > > > > > > > > > > > > >> > > > > > changes and I wonder to >> which >> > > > degree >> > > > > > > they >> > > > > > > > > have >> > > > > > > > > > > > been >> > > > > > > > > > > > > > > system >> > > > > > > > > > > > > > > > > > >> tested >> > > > > > > > > > > > > > > > > > >> > in >> > > > > > > > > > > > > > > > > > >> > > > > > addition the existing unit >> > tests >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > @Alexandre, since you are >> > using >> > > > > > > > > > > storm-kafka-client >> > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > have >> > > > > > > > > > > > > > > > > > >> filed >> > > > > > > > > > > > > > > > > > >> > > some >> > > > > > > > > > > > > > > > > > >> > > > > > bugs and driven some feature >> > > > > > requests, I >> > > > > > > > > would >> > > > > > > > > > > > like >> > > > > > > > > > > > > to >> > > > > > > > > > > > > > > ask >> > > > > > > > > > > > > > > > > if >> > > > > > > > > > > > > > > > > > >> you >> > > > > > > > > > > > > > > > > > >> > > could >> > > > > > > > > > > > > > > > > > >> > > > > > help us within what is >> > > reasonably >> > > > > > > possible >> > > > > > > > > for >> > > > > > > > > > > you >> > > > > > > > > > > > > > with >> > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > >> > > following: >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > - system test the latest >> > > > > > > > storm-kafka-client >> > > > > > > > > > > > changes >> > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > your >> > > > > > > > > > > > > > > > > > >> > > > > > test/pre-production >> > environment >> > > > > > > > > > > > > > > > > > >> > > > > > - provide some information >> on >> > > > about >> > > > > > > your >> > > > > > > > > > setup, >> > > > > > > > > > > > > > > > > > >> > > > > > - how you are using >> > > > > > > > storm-kafka-client >> > > > > > > > > > > > > > > > > > >> > > > > > - how it is performing >> > > > > > > > > > > > > > > > > > >> > > > > > - Kafka brokers, >> number >> > of >> > > > > > > > > > > > topics/partitions, >> > > > > > > > > > > > > > > storm >> > > > > > > > > > > > > > > > > > >> > > parallelism, >> > > > > > > > > > > > > > > > > > >> > > > > and >> > > > > > > > > > > > > > > > > > >> > > > > > some info about your network >> > > > > > > > > > > > > > > > > > >> > > > > > - if it is reasonable to do >> so >> > > > share >> > > > > > > some >> > > > > > > > of >> > > > > > > > > > > your >> > > > > > > > > > > > > > tests >> > > > > > > > > > > > > > > > such >> > > > > > > > > > > > > > > > > > >> that >> > > > > > > > > > > > > > > > > > >> > we >> > > > > > > > > > > > > > > > > > >> > > > can >> > > > > > > > > > > > > > > > > > >> > > > > > also test it at our end. >> > > > > > > > > > > > > > > > > > >> > > > > > - Tell us specific things >> that >> > > you >> > > > > > would >> > > > > > > > > like >> > > > > > > > > > us >> > > > > > > > > > > > to >> > > > > > > > > > > > > > test >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > Thanks, >> > > > > > > > > > > > > > > > > > >> > > > > > Hugo >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 >> > PM, >> > > > Stig >> > > > > > > Rohde >> > > > > > > > > > > > Døssing < >> > > > > > > > > > > > > > > > > > >> > > > > stigdoess...@gmail.com> >> > > > > > > > > > > > > > > > > > >> > > > > > wrote: >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > I agree with Jungtaek, if >> > > > metrics >> > > > > v2 >> > > > > > > can >> > > > > > > > > go >> > > > > > > > > > in >> > > > > > > > > > > > > very >> > > > > > > > > > > > > > > soon >> > > > > > > > > > > > > > > > > it >> > > > > > > > > > > > > > > > > > >> > should >> > > > > > > > > > > > > > > > > > >> > > go >> > > > > > > > > > > > > > > > > > >> > > > > in, >> > > > > > > > > > > > > > > > > > >> > > > > > > otherwise I'd rather >> release >> > > > 1.2.0 >> > > > > > now >> > > > > > > > and >> > > > > > > > > > > work >> > > > > > > > > > > > on >> > > > > > > > > > > > > > > > getting >> > > > > > > > > > > > > > > > > > >> 2.0.0 >> > > > > > > > > > > > > > > > > > >> > > > ready >> > > > > > > > > > > > > > > > > > >> > > > > > for >> > > > > > > > > > > > > > > > > > >> > > > > > > release. >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > 1.x and master have >> drifted >> > > > pretty >> > > > > > far >> > > > > > > > > > apart, >> > > > > > > > > > > > and >> > > > > > > > > > > > > > it's >> > > > > > > > > > > > > > > > > > >> causing a >> > > > > > > > > > > > > > > > > > >> > > lot >> > > > > > > > > > > > > > > > > > >> > > > of >> > > > > > > > > > > > > > > > > > >> > > > > > > porting work at this >> point. >> > > 1.x >> > > > > > still >> > > > > > > > has >> > > > > > > > > a >> > > > > > > > > > > lot >> > > > > > > > > > > > of >> > > > > > > > > > > > > > > > Clojure >> > > > > > > > > > > > > > > > > > >> code, >> > > > > > > > > > > > > > > > > > >> > > > > > storm-core >> > > > > > > > > > > > > > > > > > >> > > > > > > hasn't been split yet in >> > that >> > > > > branch >> > > > > > > and >> > > > > > > > > > it's >> > > > > > > > > > > > also >> > > > > > > > > > > > > > > > > targeting >> > > > > > > > > > > > > > > > > > >> JDK >> > > > > > > > > > > > > > > > > > >> > > 1.7. >> > > > > > > > > > > > > > > > > > >> > > > > > It's >> > > > > > > > > > > > > > > > > > >> > > > > > > seems rare at this point >> > that >> > > a >> > > > PR >> > > > > > > > cherry >> > > > > > > > > > > picks >> > > > > > > > > > > > > > > cleanly >> > > > > > > > > > > > > > > > > onto >> > > > > > > > > > > > > > > > > > >> 1.x >> > > > > > > > > > > > > > > > > > >> > > from >> > > > > > > > > > > > > > > > > > >> > > > > > > master. >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > I've linked the list of >> > issues >> > > > > that >> > > > > > > are >> > > > > > > > > only >> > > > > > > > > > > > fixed >> > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > 2.0.0 >> > > > > > > > > > > > > > > > > > >> and >> > > > > > > > > > > > > > > > > > >> > not >> > > > > > > > > > > > > > > > > > >> > > > > 1.x, >> > > > > > > > > > > > > > > > > > >> > > > > > to >> > > > > > > > > > > > > > > > > > >> > > > > > > illustrate how far ahead >> > 2.0.0 >> > > > is. >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > >> https://pste.eu/p/9CJT.html >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 >> > Arun >> > > > > Iyer >> > > > > > < >> > > > > > > > > > > > > > > > > ai...@hortonworks.com >> > > > > > > > > > > > > > > > > > >: >> > > > > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > >> Hi Taylor, >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> Is it >> > > > https://github.com/apache/ >> > > > > > > > > > > > storm/pull/2203 >> > > > > > > > > > > > > ? >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> I think it would be great >> > to >> > > > get >> > > > > > this >> > > > > > > > in >> > > > > > > > > > 1.2 >> > > > > > > > > > > > > > release. >> > > > > > > > > > > > > > > > Can >> > > > > > > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > > > >> try >> > > > > > > > > > > > > > > > > > >> > > to >> > > > > > > > > > > > > > > > > > >> > > > > > >> address the issues in a >> > week >> > > or >> > > > > so >> > > > > > > and >> > > > > > > > > get >> > > > > > > > > > > this >> > > > > > > > > > > > > in? >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> Thanks, >> > > > > > > > > > > > > > > > > > >> > > > > > >> Arun >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> On 11/17/17, 7:13 AM, "P. >> > > > Taylor >> > > > > > > > Goetz" < >> > > > > > > > > > > > > > > > > ptgo...@gmail.com >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > wrote: >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >>> The original idea for >> the >> > > 1.2 >> > > > > > > release >> > > > > > > > > was >> > > > > > > > > > > that >> > > > > > > > > > > > > > would >> > > > > > > > > > > > > > > > > > >> introduce >> > > > > > > > > > > > > > > > > > >> > > the >> > > > > > > > > > > > > > > > > > >> > > > > > >> metrics v2 work (and >> there >> > is >> > > > > > > > additional >> > > > > > > > > > work >> > > > > > > > > > > > > > > required >> > > > > > > > > > > > > > > > > > there, >> > > > > > > > > > > > > > > > > > >> > so I >> > > > > > > > > > > > > > > > > > >> > > > > > >> understand a desire not >> to >> > > > delay >> > > > > > the >> > > > > > > > > > > release). >> > > > > > > > > > > > Do >> > > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > want >> > > > > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > >> > stick >> > > > > > > > > > > > > > > > > > >> > > > > with >> > > > > > > > > > > > > > > > > > >> > > > > > >> that, or deviate? If the >> > > > latter, >> > > > > > > would >> > > > > > > > we >> > > > > > > > > > do >> > > > > > > > > > > a >> > > > > > > > > > > > > 1.3 >> > > > > > > > > > > > > > > > > release >> > > > > > > > > > > > > > > > > > >> for >> > > > > > > > > > > > > > > > > > >> > > > > metrics? >> > > > > > > > > > > > > > > > > > >> > > > > > >>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>> As far as a 1.1.2 >> release >> > > I’m >> > > > > fine >> > > > > > > > with >> > > > > > > > > > > > > releasing >> > > > > > > > > > > > > > > that >> > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > > >> any >> > > > > > > > > > > > > > > > > > >> > > time. >> > > > > > > > > > > > > > > > > > >> > > > > > >>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>> -Taylor >> > > > > > > > > > > > > > > > > > >> > > > > > >>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> On Nov 15, 2017, at >> 6:24 >> > > PM, >> > > > > > > Jungtaek >> > > > > > > > > > Lim < >> > > > > > > > > > > > > > > > > > >> kabh...@gmail.com> >> > > > > > > > > > > > > > > > > > >> > > > > wrote: >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> I think we could start >> > > > release >> > > > > > > phase >> > > > > > > > > for >> > > > > > > > > > > both >> > > > > > > > > > > > > > 1.1.2 >> > > > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > >> 1.2.0 >> > > > > > > > > > > > > > > > > > >> > > when >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> >> > https://github.com/apache/ >> > > > > > > > > > storm/pull/2423 >> > > > > > > > > > > > will >> > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > > merged. >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> Thanks, >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> Jungtaek Lim >> > (HeartSaVioR) >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 >> > 7:00, >> > > > > > > Alexandre >> > > > > > > > > > > > > > Vermeerbergen >> > > > > > > > > > > > > > > < >> > > > > > > > > > > > > > > > > > >> > > > > > >> avermeerber...@gmail.com >> > >님이 >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> 작성: >> > > > > > > > > > > > > > > > > > >> > > > > > >>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> Hello, >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> I'd love to see a >> Storm >> > > > 1.2.0 >> > > > > > > > > released: >> > > > > > > > > > > > it's a >> > > > > > > > > > > > > > > > perfect >> > > > > > > > > > > > > > > > > > >> > schedule >> > > > > > > > > > > > > > > > > > >> > > > for >> > > > > > > > > > > > > > > > > > >> > > > > > me >> > > > > > > > > > > > > > > > > > >> > > > > > >> for >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> test any Release >> > Candidate >> > > > > that >> > > > > > > > might >> > > > > > > > > be >> > > > > > > > > > > > > > available >> > > > > > > > > > > > > > > > if >> > > > > > > > > > > > > > > > > it >> > > > > > > > > > > > > > > > > > >> > > happens >> > > > > > > > > > > > > > > > > > >> > > > > > soon. >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> Best regards, >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> Alexandre >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> 2017-11-15 7:29 >> > GMT+01:00 >> > > > Arun >> > > > > > > > > Mahadevan >> > > > > > > > > > > < >> > > > > > > > > > > > > > > > > > >> ar...@apache.org>: >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> Hi, >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> Looks like we are >> only >> > > > > waiting >> > > > > > on >> > > > > > > > > > > > > > > > > > >> > https://issues.apache.org/ >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> jira/browse/STORM-2546 >> > . >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> Are there any other >> > > issues >> > > > > > which >> > > > > > > > are >> > > > > > > > > > > > blockers >> > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > > > Storm >> > > > > > > > > > > > > > > > > > >> > 1.2.0? >> > > > > > > > > > > > > > > > > > >> > > > > Would >> > > > > > > > > > > > > > > > > > >> > > > > > >> be >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> great to see the >> 1.2.0 >> > > > > release >> > > > > > > out >> > > > > > > > > soon >> > > > > > > > > > > as >> > > > > > > > > > > > it >> > > > > > > > > > > > > > > has a >> > > > > > > > > > > > > > > > > lot >> > > > > > > > > > > > > > > > > > >> of >> > > > > > > > > > > > > > > > > > >> > > > > critical >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> fixes. >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> Thanks, >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> Arun >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>>>> >> > > > > > > > > > > > > > > > > > >> > > > > > >>> >> > > > > > > > > > > > > > > > > > >> > > > > > >> >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >>
