Hello All,
Good news today: I found & solved what was preventing my topologies from
consuming since I had upgraded from storm-kafka-client 1.1.0 to
storm-kafka-client.1.2.0-lastestsnapopfromstig.
The reasons from our own BasicKafkaSpout class, which is our homebrewed
Kafka spout based on same configuration settings as the official
storm-kafka-client Kafka spout.
The issue came from the fact that, at least storm-kafka-client 1.1.0
doesn't exposes a way to get the list of consumed topics, so we had to
parse the value returned by KafkaSpoutConfig<K,
V>.getSubscription().getTopicsString() in order to extract the list of
topics.
Here's where the issue lies:
public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) {
this.kafkaBrokers = (String) config.getKafkaProps().get(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
this.consumerId = config.getConsumerGroupId();
logger.debug("topics are: {}", config.getSubscription()
.getTopicsString());
String topicsStr = config.getSubscription().getTopicsString();
this.topics = topicsStr.substring(1, topicsStr.length() -
1).split(",");
=> Bug here with storm-kafka-client 1.2.0 snapshot, because if
there is only one topic,
the value of topicsStr which was "[mytopic]" with 1.1.0 is now
"mytopic"
and here's a fixed (and ugly) version which works with both 1.1.0 & 1.2.0
snapshot of the same code snippet:
public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) {
this.kafkaBrokers = (String) config.getKafkaProps().get(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
this.consumerId = config.getConsumerGroupId();
logger.debug("topics are: {}", config.getSubscription()
.getTopicsString());
String topicsStr = config.getSubscription().getTopicsString();
if (topicsStr.startsWith("[") && topicsStr.endsWith("]")) {
this.topics = topicsStr.substring(1, topicsStr.length() -
1).split(",");
}
else {
this.topics = topicsStr.split(",");
}
Maybe it would be good to either restore this string representation as it
was in 1.1.0, or if not your intention, they document the change.
Now I guess your question is : "why on earth did you implement your own
Kafka Spout?"
The answer is simple: we were waiting for 1.2.0 which contains a fix to
storm-kafka-client to that Spout statistics are visible in Storm UI when
this spout is used in autocommit mode. Our own spout was design on purpose
to have same config as torm-kafka-client so as to make switch from our
implementation to official storm-kafka-client-1.2.0 as simple as simple (as
a matter of fact, it's a simple property to change on our side). We
couldn't wait for 1.2.0 because we had to switch to Kafka 0.10.2.0 as soon
as possible.
Next steps for me related to Storm 1.2.0 preview tests:
* Try our topologies with the official storm-kafka-client-1.2.0 in order to
test non-regression
* Try our topologies with the official storm-kafka-client-1.2.0 on our
pre-production in order to test non-regression of performances.
Hope it helps, stay tuned :)
Best regards,
Alexandre Vermeerbergen
2017-11-20 8:49 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> Alexandre,
>
> Could you also post the BasicKafkaSpout source file? I'm curious what it's
> doing.
>
> 2017-11-20 7:50 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com>
> :
>
> > Hello Jungtaek,
> >
> > OK I will activate these traces, but since we need to capture the Spouts'
> > initialization traces, how should I activate these traces?
> >
> > Indeed, if I use one of the techniques shown here
> > https://community.hortonworks.com/articles/36151/debugging-
> > an-apache-storm-topology.html
> > then I'm afraid I have to wait until the topology is deployed before
> > setting its trace level.
> >
> > Would you please clarify how to activate traces enabling to get spout's
> > initialization.
> > (please be as specific as possible for our current case - I can even
> modify
> > some code if that's the only to get this "early" activation, but then
> > please specify which lines of code I need to add)
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> >
> > 2017-11-20 1:20 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
> >
> > > It would be much appreciated if you could change topology log level to
> > the
> > > following: level: DEBUG, logger name: 'ROOT' or 'org.apache', timeout:
> > long
> > > enough (say 1800 or 3600), and kill worker which contains Spout in UI
> or
> > > console.
> > >
> > > Above instruction enables logging with DEBUG level, and Kafka spout
> being
> > > restarted, so we could see initialization phase of Kafka spout.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 11월 20일 (월) 오전 8:09, Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com>님이
> > > 작성:
> > >
> > > > Hello Stig,
> > > >
> > > > Thanks again for your latest fix.
> > > >
> > > > I have no longer any exception when submitting my topologies, but
> then
> > > they
> > > > read nothing from my Kafka topics.
> > > >
> > > > So I made another test: I reverted from your latest
> storm-kafka-client
> > > > 1.2.0 snapshot jar to storm-kafka-client-1.1.0.jar in my topologies,
> > but
> > > I
> > > > kept an installation based on Storm 1.2.0 snapshot => Kafka
> consumption
> > > is
> > > > OK
> > > >
> > > > If I revert again to your latest storm-kafka-client 1.2.0 snapshot
> jar,
> > > > then Kafka consumption never starts.
> > > >
> > > > I have no exceptions in logs worker's artifact logs, I'm unsure what
> to
> > > try
> > > > now...
> > > >
> > > > Any trace which I could activate?
> > > >
> > > > Best regards,
> > > > Alexandre Vermeerbergen
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2017-11-19 16:48 GMT+01:00 Stig Rohde Døssing <
> stigdoess...@gmail.com
> > >:
> > > >
> > > > > Thanks, I've addressed the issue here
> > > > > https://github.com/apache/storm/pull/2428 and uploaded a new jar
> at
> > > the
> > > > > same link here https://drive.google.com/file/d/
> > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > v9fd/view?usp=sharing. I went over the PR that made these changes,
> > and
> > > I
> > > > > don't believe anything else breaks backward compatibility, but
> we'll
> > > see.
> > > > >
> > > > > 2017-11-19 15:48 GMT+01:00 Alexandre Vermeerbergen <
> > > > > avermeerber...@gmail.com
> > > > > >:
> > > > >
> > > > > > Hi Stig,
> > > > > >
> > > > > > Here's the source of our "BasicKafkaSpout" class's constructor (I
> > can
> > > > > send
> > > > > > the full source if needed),
> > > > > >
> > > > > >
> > > > > > public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) {
> > > > > > this.kafkaBrokers = (String) config.getKafkaProps().get(
> > > > > > ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
> > > > > > this.consumerId = config.getConsumerGroupId();
> > > > > > logger.debug("topics are: {}", config.getSubscription()
> > > > > > .getTopicsString());
> > > > > > String topicsStr = config.getSubscription().
> > > getTopicsString();
> > > > > > this.topics = topicsStr.substring(1, topicsStr.length() -
> > > > > > 1).split(",");
> > > > > > switch (config.getFirstPollOffsetStrategy()) {
> > > > > > case UNCOMMITTED_EARLIEST:
> > > > > > case EARLIEST:
> > > > > > this.strategy = "earliest";
> > > > > > break;
> > > > > > case UNCOMMITTED_LATEST:
> > > > > > case LATEST:
> > > > > > default:
> > > > > > this.strategy = "latest";
> > > > > > }
> > > > > > this.keyDeserializer = config.getKeyDeserializer().
> > > getClass();
> > > > > > this.valueDeserializer = config.getValueDeserializer().
> > > > > getClass();
> > > > > > this.translator = config.getTranslator();
> > > > > > this.consumerProps = new Properties();
> > > > > > if (config.getKafkaProps() != null) {
> > > > > > consumerProps.putAll(config.getKafkaProps());
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > line 72 is the following one:
> > > > > >
> > > > > > this.keyDeserializer = config.getKeyDeserializer().
> > > getClass();
> > > > > >
> > > > > > Hope it helps,
> > > > > >
> > > > > > Alexandre Vermeerbergen
> > > > > >
> > > > > > 2017-11-19 15:37 GMT+01:00 Stig Rohde Døssing <
> > > stigdoess...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Alexandre,
> > > > > > >
> > > > > > > I'm sorry this is giving you so much trouble. Looking at the
> > stack
> > > > > trace
> > > > > > > you posted, it seems like the NPE is coming from
> BasicKafkaSpout
> > > line
> > > > > 72.
> > > > > > > Can you post that line (and maybe some of the surrounding
> code)?
> > > > > > >
> > > > > > > 2017-11-19 15:13 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > avermeerber...@gmail.com
> > > > > > > >:
> > > > > > >
> > > > > > > > Hi Stig,
> > > > > > > >
> > > > > > > > After having included commons-lang-2.5.jar in my topologies,
> I
> > > > still
> > > > > > have
> > > > > > > > 11 topologies failing with this new message, this time a
> > > > > > > > NullPointerException:
> > > > > > > >
> > > > > > > > Running: /usr/local/jdk/bin/java -client -Ddaemon.name=
> > > > > > -Dstorm.options=
> > > > > > > > -Dstorm.home=/usr/local/Storm/storm-stable
> > > > > > > > -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> > > > > > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > > > > > > > -Dstorm.conf.file= -cp
> > > > > > > > /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-
> > > > > > > > stable/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/
> > > > > > > > local/Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-
> > > > > > > > stable/conf:/usr/local/Storm/storm-stable/bin
> > > > > > > > -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> > > > > > > > -Dstorm.dependency.jars=
> > > > > > > > -Dstorm.dependency.artifacts={}
> > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology
> > > > > > > slaEventsInterceptor
> > > > > > > > 1939 [main] WARN c.a.s.u.StormUtils - Couldn't read
> > > > > > > > /usr/local/Storm/flux/slaEventsInterceptor_
> > > kafkaConsumer.properties
> > > > > > > file:
> > > > > > > > /usr/local/Storm/flux/slaEventsInterceptor_
> > > kafkaConsumer.properties
> > > > > > (No
> > > > > > > > such file or directory)
> > > > > > > > 1978 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
> > > > > > > > enable.auto.commit manually. Instead use
> > > > > > > > KafkaSpoutConfig.Builder.setProcessingGuarantee. This will
> be
> > > > > treated
> > > > > > as
> > > > > > > > an
> > > > > > > > error in the next major release. For now the spout will be
> > > > configured
> > > > > > to
> > > > > > > > behave like it would have in pre-1.2.0 releases.
> > > > > > > > 1982 [main] INFO c.a.s.u.StormUtils - Use Kafka Spout:
> > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout
> > > > > > > > Exception in thread "main" java.lang.NullPointerException
> > > > > > > > at
> > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(
> > > > > > > > BasicKafkaSpout.java:72)
> > > > > > > > at com.acme.storm.util.StormUtils.getKafkaSpout(
> > > > > > > > StormUtils.java:888)
> > > > > > > > at
> > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(
> > > > > > > > SLAEventsInterceptorTopology.java:120)
> > > > > > > >
> > > > > > > > and here's the line from our source from which this exception
> > was
> > > > > > > > triggered:
> > > > > > > >
> > > > > > > > IRichSpout kafkaSpoutForMainTopic = StormUtils
> > > > > > > > .getKafkaSpout(spoutConfigForMainTopic,
> > > > > > > kafkaSpoutClass);
> > > > > > > >
> > > > > > > > Looks like upgrade from 1.1.0 (our current production setup)
> to
> > > > 1.2.0
> > > > > > is
> > > > > > > a
> > > > > > > > bit more difficult than I expected.
> > > > > > > > But anyway, I greatly appreciate Storm, so I hope my
> feedbacks
> > > will
> > > > > > help
> > > > > > > > keeping it as good as possible :)
> > > > > > > >
> > > > > > > > Please let me know if further details are required to solve
> > this
> > > > > issue
> > > > > > > with
> > > > > > > > storm-kafka-client 1.2.0 snapshot.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Alexandre
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-11-19 13:26 GMT+01:00 Stig Rohde Døssing <
> > > > > stigdoess...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Using Maven and controlling dependencies aren't mutually
> > > > exclusive.
> > > > > > > Tools
> > > > > > > > > like Nexus (https://help.sonatype.com/
> > > display/NXRM2/Procurement+
> > > > > > Suite)
> > > > > > > > > allow you to control access to dependencies without
> manually
> > > > > putting
> > > > > > > jars
> > > > > > > > > in an SCM.
> > > > > > > > >
> > > > > > > > > I realize this doesn't help you right now, just thought I'd
> > > > mention
> > > > > > it
> > > > > > > so
> > > > > > > > > you don't get the idea that I'm advocating a build process
> > that
> > > > > > blindly
> > > > > > > > > includes new dependencies.
> > > > > > > > >
> > > > > > > > > 2017-11-19 13:05 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > > > avermeerber...@gmail.com
> > > > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Believe it or not, we don't internally use Maven : we
> > > strictly
> > > > > > > control
> > > > > > > > > our
> > > > > > > > > > dependencies.
> > > > > > > > > >
> > > > > > > > > > So okay, I have delivered a "storm-commons-lang2.jar" in
> > our
> > > > SCM
> > > > > > > system
> > > > > > > > > in
> > > > > > > > > > order to specifically add it to our topologies big jars
> > > > depending
> > > > > > on
> > > > > > > > > > storm-kafka-client (feeling sad).
> > > > > > > > > >
> > > > > > > > > > I'll tell you the outcome ASAP.
> > > > > > > > > >
> > > > > > > > > > Best regard,
> > > > > > > > > > Alexandre Vermeerbergen
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2017-11-19 12:57 GMT+01:00 Stig Rohde Døssing <
> > > > > > > stigdoess...@gmail.com
> > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > I also don't think that adding dependencies
> constitutes a
> > > > > > breaking
> > > > > > > > > > change,
> > > > > > > > > > > nor should it. Dependency management tooling like Maven
> > > will
> > > > > > handle
> > > > > > > > > > pulling
> > > > > > > > > > > the right dependencies automatically.
> > > > > > > > > > >
> > > > > > > > > > > 2017-11-19 12:34 GMT+01:00 Jungtaek Lim <
> > kabh...@gmail.com
> > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Alexandre,
> > > > > > > > > > > >
> > > > > > > > > > > > There's so much pain on using relocated artifact in
> > other
> > > > > > module
> > > > > > > > in a
> > > > > > > > > > > > project. You need to refer the class as relocated
> name,
> > > and
> > > > > > your
> > > > > > > > IDE
> > > > > > > > > > (at
> > > > > > > > > > > > least IntelliJ) complains and the compilation in IDE
> > will
> > > > not
> > > > > > > > > succeed.
> > > > > > > > > > > (The
> > > > > > > > > > > > build via Maven will succeed though.)
> > > > > > > > > > > > There was same issue on 2.0.0 as well, and for now we
> > > don't
> > > > > > shade
> > > > > > > > > > > anything
> > > > > > > > > > > > in 2.0.0. We need to discuss which things to shade
> > before
> > > > > > > > discussing
> > > > > > > > > > > about
> > > > > > > > > > > > releasing, indeed.
> > > > > > > > > > > >
> > > > > > > > > > > > And normally we have been allowing adding dependency
> as
> > > > minor
> > > > > > > > > release.
> > > > > > > > > > I
> > > > > > > > > > > > understand someone could say that it is breaking
> > > > dependency,
> > > > > > and
> > > > > > > > > should
> > > > > > > > > > > > lead another relocation as well. But my 2 cents the
> > > > > requirement
> > > > > > > is
> > > > > > > > > too
> > > > > > > > > > > > restrictive, and may end up with Guava-like
> versioning.
> > > > > > > (releasing
> > > > > > > > > > major
> > > > > > > > > > > > versions sooner...)
> > > > > > > > > > > > I think relocating is not an ideal solution, and
> > > eventually
> > > > > we
> > > > > > > may
> > > > > > > > > have
> > > > > > > > > > > > nice solution (like classloader isolation) and then
> we
> > > > could
> > > > > > sort
> > > > > > > > out
> > > > > > > > > > all
> > > > > > > > > > > > the related issues. Before that, I could not block
> > adding
> > > > > > > > > dependencies
> > > > > > > > > > in
> > > > > > > > > > > > minor release if it is necessary.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Jungtaek Lim (HeartSaVioR)
> > > > > > > > > > > >
> > > > > > > > > > > > 2017년 11월 19일 (일) 오후 8:02, Stig Rohde Døssing <
> > > > > > > > > stigdoess...@gmail.com
> > > > > > > > > > >님이
> > > > > > > > > > > > 작성:
> > > > > > > > > > > >
> > > > > > > > > > > > > Alexandre,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for trying. I think this time it's not a
> > problem
> > > > > with
> > > > > > > > > > > > > storm-kafka-client. It has a dependency on
> > commons-lang
> > > > > 2.5,
> > > > > > > > which
> > > > > > > > > is
> > > > > > > > > > > > > declared in the storm-kafka-client pom. We usually
> > > > > recommend
> > > > > > > that
> > > > > > > > > > > people
> > > > > > > > > > > > > use Maven/Gradle/Ivy or similar systems to ensure
> > that
> > > > they
> > > > > > get
> > > > > > > > all
> > > > > > > > > > > > > dependencies when they build their topologies. In
> > such
> > > a
> > > > > case
> > > > > > > > Maven
> > > > > > > > > > (or
> > > > > > > > > > > > > similar) would see that your topology depends on
> > > > > > > > > storm-kafka-client,
> > > > > > > > > > > and
> > > > > > > > > > > > > also download and include commons-lang in the
> > topology
> > > > jar.
> > > > > > > When
> > > > > > > > > you
> > > > > > > > > > > put
> > > > > > > > > > > > > together dependencies manually you need to include
> > > > > > commons-lang
> > > > > > > > > > > manually
> > > > > > > > > > > > as
> > > > > > > > > > > > > well.
> > > > > > > > > > > > >
> > > > > > > > > > > > > You can get the commons-lang 2.5 jar here
> > > > > > > > > > > > >
> > > > > > > > > > > > > http://central.maven.org/
> > maven2/commons-lang/commons-
> > > > > > > > > > > > lang/2.5/commons-lang-2.5.jar
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-11-19 11:49 GMT+01:00 Alexandre Vermeerbergen
> <
> > > > > > > > > > > > > avermeerber...@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hello Stig,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Than you very much for you quick answer & fix.
> > > > > > > > > > > > > > Unfortunaly, I still have topologies failing to
> > start
> > > > but
> > > > > > > with
> > > > > > > > a
> > > > > > > > > > > > > different
> > > > > > > > > > > > > > exception this time:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Exception in thread "main" java.lang.
> > > > > NoClassDefFoundError:
> > > > > > > > > > > > > > org/apache/commons/lang/StringUtils
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > org.apache.storm.kafka.spout.NamedTopicFilter.
> > > > > > > getTopicsString(
> > > > > > > > > > > > > > NamedTopicFilter.java:66)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > org.apache.storm.kafka.spout.
> > > > > ManualPartitionSubscription.
> > > > > > > > > > > > getTopicsString(
> > > > > > > > > > > > > > ManualPartitionSubscription.java:70)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > com.acme.storm.evaluator.
> > > spout.BasicKafkaSpout.<init>(
> > > > > > > > > > > > > > BasicKafkaSpout.java:59)
> > > > > > > > > > > > > > at
> > > > com.acme.storm.util.StormUtils.getKafkaSpout(
> > > > > > > > > > > > > > StormUtils.java:888)
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > com.acme.storm.evaluator.
> > > SLAEventsInterceptorTopology.
> > > > > > main(
> > > > > > > > > > > > > > SLAEventsInterceptorTopology.java:120)
> > > > > > > > > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > > > > > > > > org.apache.commons.lang.StringUtils
> > > > > > > > > > > > > > at java.net.URLClassLoader.
> > > > > > > > > findClass(URLClassLoader.java:
> > > > > > > > > > > 381)
> > > > > > > > > > > > > > at java.lang.ClassLoader.
> > > > > > loadClass(ClassLoader.java:
> > > > > > > > 424)
> > > > > > > > > > > > > > at sun.misc.Launcher$
> > > AppClassLoader.loadClass(
> > > > > > > > > > > > Launcher.java:338)
> > > > > > > > > > > > > > at java.lang.ClassLoader.
> > > > > > loadClass(ClassLoader.java:
> > > > > > > > 357)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Looks like you introduced a dependency on commons
> > > lang
> > > > > that
> > > > > > > > > wasn't
> > > > > > > > > > > part
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > Storm, isn't it ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Keep faith, next time will be better :)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Alexandre
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <
> > > > > > > > > > > stigdoess...@gmail.com
> > > > > > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I've put up a fix here
> > https://github.com/apache/
> > > > > > > > > storm/pull/2426
> > > > > > > > > > .
> > > > > > > > > > > > > > There's
> > > > > > > > > > > > > > > an updated storm-kafka-client jar at
> > > > > > > > > > > > > > > https://drive.google.com/file/d/
> > > > > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > > > > > > > v9fd/view?usp=sharing
> > > > > > > > > > > > > > > if you'd like to try it out.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <
> > > > > > > > > > > > stigdoess...@gmail.com
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Oops. The exception is not intentional, it's
> a
> > > bug.
> > > > > In
> > > > > > > > 1.2.0
> > > > > > > > > we
> > > > > > > > > > > > check
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > the "enable.auto.commit" key in the
> > > > kafkaConsumerProp
> > > > > > > map,
> > > > > > > > > and
> > > > > > > > > > if
> > > > > > > > > > > > it
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > set
> > > > > > > > > > > > > > > > we warn in the log that it shouldn't be,
> > because
> > > > > users
> > > > > > > > should
> > > > > > > > > > use
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > KafkaSpoutConfig.Builder.
> > setProcessingGuarantee
> > > > > method
> > > > > > > > > > instead.
> > > > > > > > > > > > When
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > property is set we try to set the processing
> > > > > guarantee
> > > > > > to
> > > > > > > > > match
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > pre-1.2.0 behavior, but I made a mistake and
> > > > assumed
> > > > > > the
> > > > > > > > > > property
> > > > > > > > > > > > key
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > boolean when it might actually be either a
> > > boolean
> > > > > or a
> > > > > > > > > string.
> > > > > > > > > > > > I'll
> > > > > > > > > > > > > > put
> > > > > > > > > > > > > > > up
> > > > > > > > > > > > > > > > a fix ASAP.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00
> > > > > > Alexandre
> > > > > > > > > > > > Vermeerbergen
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > avermeerber...@gmail.com>:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Hello Stig,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Here's my first feedback on this Storm 1.2.0
> > > > preview
> > > > > > on
> > > > > > > my
> > > > > > > > > > > > > Supervision
> > > > > > > > > > > > > > > >> system based on Storm : I have 11 topologies
> > KO
> > > > (not
> > > > > > > even
> > > > > > > > > able
> > > > > > > > > > > to
> > > > > > > > > > > > > > > start),
> > > > > > > > > > > > > > > >> and 4 topologies which seem to be
> unaffected.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Details:
> > > > > > > > > > > > > > > >> - I used the binaries posted on dropbox by
> > > > Jungteak
> > > > > > and
> > > > > > > > your
> > > > > > > > > > > > binary
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> storm-kafka-client
> > > > > > > > > > > > > > > >> - Rebuilt all our topologies using these
> > > > > > storm-core.jar
> > > > > > > &
> > > > > > > > > > > > > > > >> storm-kafka-client.jar from these
> > > > > storm-1.2.0-snapshot
> > > > > > > > (and
> > > > > > > > > I
> > > > > > > > > > > got
> > > > > > > > > > > > > > > >> absolutely no error at build time).
> > > > > > > > > > > > > > > >> Meaning that I have new "big jar" files
> > > > including
> > > > > > this
> > > > > > > > > newer
> > > > > > > > > > > > > > > >> storm-kafka-client.jar (except for 1
> topology
> > > > which
> > > > > > > > depends
> > > > > > > > > on
> > > > > > > > > > > > > another
> > > > > > > > > > > > > > > >> build system)
> > > > > > > > > > > > > > > >> - The 11 topologies which fail to start show
> > up
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > trace
> > > > > > > > > > > > > > when
> > > > > > > > > > > > > > > >> they are submitted:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Running: java -client -Ddaemon.name=
> > > > > -Dstorm.options=
> > > > > > > > > > > > > > > >> -Dstorm.home=/usr/local/Storm/storm-stable
> > > > > > > > > > > > > > > >> -Dstorm.log.dir=/usr/local/
> > > Storm/storm-stable/logs
> > > > > > > > > > > > > > > >> -Djava.library.path=/usr/
> > > > > > local/lib:/opt/local/lib:/usr/
> > > > > > > > lib
> > > > > > > > > > > > > > > >> -Dstorm.conf.file= -cp
> > > > > > > > > > > > > > > >> /usr/local/Storm/storm-stable/
> > > > > > > > > *:/usr/local/Storm/storm-stabl
> > > > > > > > > > > > > > > >> e/lib/*:/usr/local/Storm/
> > > > > storm-stable/extlib/*:/usr/
> > > > > > > > local/
> > > > > > > > > > > > > > > >> Storm/StormTopologiesBJ.jar:/
> > > > > > > > usr/local/Storm/storm-stable/
> > > > > > > > > > > > > > > >> conf:/usr/local/Storm/storm-stable/bin
> > > > > > > > > > > > > > > >> -Dstorm.jar=/usr/local/Storm/
> > > StormTopologiesBJ.jar
> > > > > > > > > > > > > > > >> -Dstorm.dependency.jars=
> > > > > > > > > > > > > > > >> -Dstorm.dependency.artifacts={}
> > > > > > > > > > > > > > > >> com.acme.storm.evaluator.
> > > > > SLAEventsInterceptorTopology
> > > > > > > > > > > > > > > >> slaEventsInterceptor
> > > > > > > > > > > > > > > >> Latest
> > > > > > > > > > > > > > > >> 1022 [main] INFO c.a.s.e.
> > > > > > SLAEventsInterceptorTopology
> > > > > > > -
> > > > > > > > > > > > Topology :
> > > > > > > > > > > > > > > >> 'slaEventsInterceptor',
> > KafkaOffsetTimeStrategy
> > > =
> > > > > > > 'Latest'
> > > > > > > > > > > > > > > >> 1026 [main] WARN c.a.s.u.StormUtils -
> > Couldn't
> > > > read
> > > > > > > > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > > > > > > > kafkaConsumer.properties
> > > > > > > > > > > > > > > file:
> > > > > > > > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > > > > > > > kafkaConsumer.properties
> > > > > > > > > > > > > > (No
> > > > > > > > > > > > > > > >> such file or directory)
> > > > > > > > > > > > > > > >> 1037 [main] WARN
> o.a.s.k.s.KafkaSpoutConfig -
> > > Do
> > > > > not
> > > > > > > set
> > > > > > > > > > > > > > > >> enable.auto.commit manually. Instead use
> > > > > > > > > > > > > > > >> KafkaSpoutConfig.Builder.
> > > setProcessingGuarantee.
> > > > > This
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > > treated
> > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > >> an
> > > > > > > > > > > > > > > >> error in the next major release. For now the
> > > spout
> > > > > > will
> > > > > > > be
> > > > > > > > > > > > > configured
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> behave like it would have in pre-1.2.0
> > releases.
> > > > > > > > > > > > > > > >> Exception in thread "main"
> > > > > > java.lang.ClassCastException:
> > > > > > > > > > > > > > > java.lang.String
> > > > > > > > > > > > > > > >> cannot be cast to java.lang.Boolean
> > > > > > > > > > > > > > > >> at
> > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout.
> > KafkaSpoutConfig.
> > > > > > > > > setAutoCommitM
> > > > > > > > > > > > > > > >> ode(KafkaSpoutConfig.java:721)
> > > > > > > > > > > > > > > >> at
> > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout.
> > > > > KafkaSpoutConfig.<init>(
> > > > > > > > > KafkaSp
> > > > > > > > > > > > > > > >> outConfig.java:97)
> > > > > > > > > > > > > > > >> at
> > > > > > > > > > > > > > > >> org.apache.storm.kafka.spout.
> > > > > > KafkaSpoutConfig$Builder.
> > > > > > > > > build(
> > > > > > > > > > > > > > > >> KafkaSpoutConfig.java:671)
> > > > > > > > > > > > > > > >> at
> > > > > > > > > > > > > > > >> com.acme.storm.evaluator.
> > > > > > SLAEventsInterceptorTopology.
> > > > > > > > > main(S
> > > > > > > > > > > > > > > >> LAEventsInterceptorTopology.java:118)
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> The line part of our application which
> > triggered
> > > > > above
> > > > > > > > stack
> > > > > > > > > > > trace
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> following one:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> KafkaSpoutConfig<String, Event>
> > > > > > > > > > spoutConfigForMainTopic
> > > > > > > > > > > =
> > > > > > > > > > > > > > > >> KafkaSpoutConfig
> > > > > > > > > > > > > > > >>
> .builder(elasticKafkaBrokers,
> > > > > > > > > > KafkaTopics.MAIN)
> > > > > > > > > > > > > > > >> .setValue(
> > > > > > EventKafkaDeserializer.class)
> > > > > > > > > > > > > > > >> .setGroupId(consumerId +
> "_" +
> > > > > > > > > > KafkaTopics.MAIN)
> > > > > > > > > > > > > > > >> .setFirstPollOffsetStrategy(
> > > > > strategy)
> > > > > > > > > > > > > > > >> .setProp(kafkaConsumerProp)
> > > > > > > > > > > > > > > >> .setRecordTranslator(
> > > > > > > > > > > > > > > >> new
> > > > > > > SLAEventsInterceptorKafkaRecor
> > > > > > > > > > > > > > > dTranslator())
> > > > > > > > > > > > > > > >> .build();
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> I understand that there's a WARN message,
> but
> > if
> > > > > it's
> > > > > > a
> > > > > > > > > > warning,
> > > > > > > > > > > > > then
> > > > > > > > > > > > > > > why
> > > > > > > > > > > > > > > >> do we exit with such a brutal exception?
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Best regards,
> > > > > > > > > > > > > > > >> Alexandre Vermeerbergen
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde
> Døssing
> > <
> > > > > > > > > > > > > stigdoess...@gmail.com
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> > Alexandre,
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > I've uploaded the 1.2.0-SNAPSHOT
> > > > > storm-kafka-client
> > > > > > > jar
> > > > > > > > > here
> > > > > > > > > > > > > > > >> > https://drive.google.com/file/d/
> > > > > > > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > > > > > > > >> > v9fd/view?usp=sharing.
> > > > > > > > > > > > > > > >> > You can probably keep the 1.1.0 versions
> of
> > > > > > > storm-hbase
> > > > > > > > > and
> > > > > > > > > > > > > > > storm-hdfs,
> > > > > > > > > > > > > > > >> > since the core Storm API hasn't changed.
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > If you want to build the jars yourself,
> > clone
> > > > the
> > > > > > > > > repository
> > > > > > > > > > > at
> > > > > > > > > > > > > > > >> > https://github.com/apache/storm and check
> > out
> > > > the
> > > > > > > > > > 1.x-branch
> > > > > > > > > > > > > > branch.
> > > > > > > > > > > > > > > >> You
> > > > > > > > > > > > > > > >> > can build the entire project by running
> "mvn
> > > > clean
> > > > > > > > install
> > > > > > > > > > > > > > > -DskipTests"
> > > > > > > > > > > > > > > >> in
> > > > > > > > > > > > > > > >> > the root. The individual module jars will
> be
> > > > > > available
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > target
> > > > > > > > > > > > > > > >> > directories for each module, e.g.
> > > > > > > storm-kafka-client.jar
> > > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > >> > external/storm-kafka-client/target.
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > If you need to build the regular Storm
> > > > > distribution
> > > > > > > (the
> > > > > > > > > > > tarball
> > > > > > > > > > > > > > > >> Jungtaek
> > > > > > > > > > > > > > > >> > sent you), you can go to storm-dist/binary
> > and
> > > > run
> > > > > > > "mvn
> > > > > > > > > > clean
> > > > > > > > > > > > > > install
> > > > > > > > > > > > > > > >> > -Dgpg.skip". The tar/zip will be in
> > > > > > > > > > storm-dist/binary/target.
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Further reference in case you need it
> > > > > > > > > > > > > > > >> > https://github.com/apache/
> > > > > > storm/blob/master/DEVELOPER
> > > > > > > .
> > > > > > > > > > > > > > > >> > md#build-the-code-and-run-the-tests
> > > > > > > > > > > > > > > >> > .
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre
> > > > > Vermeerbergen <
> > > > > > > > > > > > > > > >> > avermeerber...@gmail.com
> > > > > > > > > > > > > > > >> > >:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > > Hello Jungtaek,
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Thanks for the link to 1.2.0 preview
> > > binaries.
> > > > > > > > > > > > > > > >> > > However, since we heavily depend on
> Storm
> > > > Kafka
> > > > > > > > Client,
> > > > > > > > > > > would
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > > >> please
> > > > > > > > > > > > > > > >> > > either add it or remind me how to build
> > the
> > > > jars
> > > > > > of
> > > > > > > > this
> > > > > > > > > > > > > external
> > > > > > > > > > > > > > > >> lib, at
> > > > > > > > > > > > > > > >> > > same "snapshot" version as the rest ?
> > > > > > > > > > > > > > > >> > > On a side note, some of our topologies
> > also
> > > > rely
> > > > > > on
> > > > > > > > > storm
> > > > > > > > > > > > hbase
> > > > > > > > > > > > > &
> > > > > > > > > > > > > > > >> storm
> > > > > > > > > > > > > > > >> > > hdfs : does it matters if these later
> ones
> > > > stay
> > > > > in
> > > > > > > > 1.1.0
> > > > > > > > > > > > version
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> this
> > > > > > > > > > > > > > > >> > > test?
> > > > > > > > > > > > > > > >> > > Indeed, I want to focus on Storm 1.2.0
> > > "core"
> > > > +
> > > > > > > Storm
> > > > > > > > > > Kafka
> > > > > > > > > > > > > Client
> > > > > > > > > > > > > > > >> 1.2.0
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Best regards,
> > > > > > > > > > > > > > > >> > > Alexandre Vermeerbergen
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > 2017-11-18 <20%2017%2011%2018> 7:41
> > > GMT+01:00
> > > > > > > Jungtaek
> > > > > > > > > > Lim <
> > > > > > > > > > > > > > > >> kabh...@gmail.com>:
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > > Alexandre,
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > https://www.dropbox.com/s/
> > > > > > > > > mg2gnunk24oesyc/apache-storm-
> > > > > > > > > > > > > > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > Above link is custom binary
> distribution
> > > of
> > > > > > > current
> > > > > > > > > > > > 1.x-branch
> > > > > > > > > > > > > > > >> > (SNAPSHOT
> > > > > > > > > > > > > > > >> > > of
> > > > > > > > > > > > > > > >> > > > 1.2.0). Could you run the test in your
> > > > > > environment
> > > > > > > > > first
> > > > > > > > > > > and
> > > > > > > > > > > > > let
> > > > > > > > > > > > > > > us
> > > > > > > > > > > > > > > >> > know
> > > > > > > > > > > > > > > >> > > > about the result? Regardless of
> > including
> > > > > > metrics
> > > > > > > V2
> > > > > > > > > to
> > > > > > > > > > > > 1.2.0
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > >> not,
> > > > > > > > > > > > > > > >> > > your
> > > > > > > > > > > > > > > >> > > > test report should be valuable for us.
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > And please refer my analysis of
> current
> > > > > metrics:
> > > > > > > > > > > > > > > >> > > > https://cwiki.apache.org/
> > > > > > > confluence/display/STORM/
> > > > > > > > > > > > > > > >> > > Limitations+of+current+
> > > > > > > > > > > > > > > >> > > > metrics+feature
> > > > > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > > > > >> > > > see why we want to move toward to
> > Metrics
> > > > V2.
> > > > > > > > Current
> > > > > > > > > PR
> > > > > > > > > > > on
> > > > > > > > > > > > > > > Metrics
> > > > > > > > > > > > > > > >> V2
> > > > > > > > > > > > > > > >> > is
> > > > > > > > > > > > > > > >> > > > for initial state and it could cover
> > only
> > > > some
> > > > > > of
> > > > > > > > > issues
> > > > > > > > > > > in
> > > > > > > > > > > > > > list,
> > > > > > > > > > > > > > > >> but I
> > > > > > > > > > > > > > > >> > > > expect we will address another issues
> as
> > > > well
> > > > > > > based
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > PR.
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > Thanks in advance!
> > > > > > > > > > > > > > > >> > > > Jungtaek Lim (HeartSaVioR)
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre
> > > > > > > Vermeerbergen <
> > > > > > > > > > > > > > > >> > > > avermeerber...@gmail.com>님이
> > > > > > > > > > > > > > > >> > > > 작성:
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > > Hello Hugo,
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > As I already posted, I was getting
> > ready
> > > > to
> > > > > > test
> > > > > > > > the
> > > > > > > > > > > > > upcoming
> > > > > > > > > > > > > > > RC,
> > > > > > > > > > > > > > > >> so
> > > > > > > > > > > > > > > >> > > > yeah :
> > > > > > > > > > > > > > > >> > > > > as soon as I have some binaries for
> > > > testing,
> > > > > > I
> > > > > > > > can
> > > > > > > > > > run
> > > > > > > > > > > > any
> > > > > > > > > > > > > > > Storm
> > > > > > > > > > > > > > > >> > 1.2.0
> > > > > > > > > > > > > > > >> > > > > preview on my pre-production (well
> > > > stressed)
> > > > > > > > > > > environment.
> > > > > > > > > > > > Is
> > > > > > > > > > > > > > > there
> > > > > > > > > > > > > > > >> > some
> > > > > > > > > > > > > > > >> > > > URL
> > > > > > > > > > > > > > > >> > > > > from where I can download such
> > > "snapshot"
> > > > > > > > binaries?
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > > > > > > >> > > > > Alexandre Vermeerbergen
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > PS: By the way, I have no interest
> in
> > > the
> > > > > > > > > > > metrics-related
> > > > > > > > > > > > > > > >> > enhancements
> > > > > > > > > > > > > > > >> > > > you
> > > > > > > > > > > > > > > >> > > > > mentionned - I can understand it is
> of
> > > > > > interest
> > > > > > > > for
> > > > > > > > > > > people
> > > > > > > > > > > > > > aware
> > > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > > >> > it
> > > > > > > > > > > > > > > >> > > > and
> > > > > > > > > > > > > > > >> > > > > who are expecting "something"
> related
> > to
> > > > it
> > > > > -
> > > > > > > but
> > > > > > > > I
> > > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > sad
> > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > >> > > see a
> > > > > > > > > > > > > > > >> > > > > postponing of the imminent Storm
> 1.2.0
> > > > > release
> > > > > > > for
> > > > > > > > > > > > something
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > >> > not
> > > > > > > > > > > > > > > >> > > > use
> > > > > > > > > > > > > > > >> > > > > at all.
> > > > > > > > > > > > > > > >> > > > > PPS: I would love to learn more
> about
> > > > these
> > > > > > > > > "metrics",
> > > > > > > > > > > > maybe
> > > > > > > > > > > > > > > it's
> > > > > > > > > > > > > > > >> > > > something
> > > > > > > > > > > > > > > >> > > > > worth which never quite
> understood...
> > > any
> > > > > link
> > > > > > > > with
> > > > > > > > > > > clear
> > > > > > > > > > > > > > > >> explanation
> > > > > > > > > > > > > > > >> > > of
> > > > > > > > > > > > > > > >> > > > > this feature?
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da
> > Cruz
> > > > > Louro
> > > > > > <
> > > > > > > > > > > > > > > >> > hlo...@hortonworks.com
> > > > > > > > > > > > > > > >> > > >:
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > > I am also in agreement that we
> > should
> > > > not
> > > > > > > delay
> > > > > > > > > the
> > > > > > > > > > > > 1.2.0
> > > > > > > > > > > > > > > >> release
> > > > > > > > > > > > > > > >> > for
> > > > > > > > > > > > > > > >> > > > too
> > > > > > > > > > > > > > > >> > > > > > long, but in order to release it a
> > few
> > > > > > things
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > >> > > mind:
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > - If we want to avoid releasing
> > 1.3.0
> > > > > soon
> > > > > > or
> > > > > > > > at
> > > > > > > > > > all,
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> means
> > > > > > > > > > > > > > > >> > > that
> > > > > > > > > > > > > > > >> > > > we
> > > > > > > > > > > > > > > >> > > > > > should include in 1.2.0 as many
> > > > important
> > > > > > > > features
> > > > > > > > > > > (e.g.
> > > > > > > > > > > > > > > >> metricsV2)
> > > > > > > > > > > > > > > >> > > as
> > > > > > > > > > > > > > > >> > > > > > possible within a reasonable time
> > > frame.
> > > > > > Also
> > > > > > > to
> > > > > > > > > > avoid
> > > > > > > > > > > > as
> > > > > > > > > > > > > > much
> > > > > > > > > > > > > > > >> back
> > > > > > > > > > > > > > > >> > > > > porting
> > > > > > > > > > > > > > > >> > > > > > as possible one should really get
> > all
> > > > > > features
> > > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > > foresee
> > > > > > > > > > > > > > > >> we
> > > > > > > > > > > > > > > >> > > want
> > > > > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > > > > >> > > > > > maintain in 1.2.0 right away, and
> > then
> > > > > > simply
> > > > > > > > > > maintain
> > > > > > > > > > > > > them
> > > > > > > > > > > > > > > >> there.
> > > > > > > > > > > > > > > >> > > > > However
> > > > > > > > > > > > > > > >> > > > > > it’s a fact of life that if a bug
> is
> > > > found
> > > > > > and
> > > > > > > > it
> > > > > > > > > > is a
> > > > > > > > > > > > > > > blocker,
> > > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > > >> > > will
> > > > > > > > > > > > > > > >> > > > > > have to be back ported.
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > - STORM-2153 details that the new
> > > > metrics
> > > > > > > > > > requirements
> > > > > > > > > > > > > were
> > > > > > > > > > > > > > > >> driven
> > > > > > > > > > > > > > > >> > by
> > > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > > >> > > > > > users. Therefore if this feature
> is
> > > > highly
> > > > > > > > > important
> > > > > > > > > > > and
> > > > > > > > > > > > > > > sought
> > > > > > > > > > > > > > > >> > > after,
> > > > > > > > > > > > > > > >> > > > it
> > > > > > > > > > > > > > > >> > > > > > may not make much sense to release
> > > 1.2.0
> > > > > > > without
> > > > > > > > > it.
> > > > > > > > > > > If
> > > > > > > > > > > > we
> > > > > > > > > > > > > > do
> > > > > > > > > > > > > > > >> > release
> > > > > > > > > > > > > > > >> > > > > 1.2.0
> > > > > > > > > > > > > > > >> > > > > > without the metrics, does it mean
> > that
> > > > > > metrics
> > > > > > > > > will
> > > > > > > > > > go
> > > > > > > > > > > > > only
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > >> > 2.0? I
> > > > > > > > > > > > > > > >> > > > > would
> > > > > > > > > > > > > > > >> > > > > > conjecture that most production
> > > > > deployments
> > > > > > > will
> > > > > > > > > > take
> > > > > > > > > > > a
> > > > > > > > > > > > > > while
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> > > > upgrade
> > > > > > > > > > > > > > > >> > > > > to
> > > > > > > > > > > > > > > >> > > > > > 2.0 even after it is released.
> That
> > > > means
> > > > > > that
> > > > > > > > > they
> > > > > > > > > > > will
> > > > > > > > > > > > > > still
> > > > > > > > > > > > > > > >> be
> > > > > > > > > > > > > > > >> > > > running
> > > > > > > > > > > > > > > >> > > > > > Storm without the benefit of the
> new
> > > > > > metrics.
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > - Several fixes have got in
> > > > > > > storm-kafka-client.
> > > > > > > > > > There
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > >> been a
> > > > > > > > > > > > > > > >> > > lot
> > > > > > > > > > > > > > > >> > > > of
> > > > > > > > > > > > > > > >> > > > > > changes and I wonder to which
> degree
> > > > they
> > > > > > have
> > > > > > > > > been
> > > > > > > > > > > > system
> > > > > > > > > > > > > > > >> tested
> > > > > > > > > > > > > > > >> > in
> > > > > > > > > > > > > > > >> > > > > > addition the existing unit tests
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > @Alexandre, since you are using
> > > > > > > > storm-kafka-client
> > > > > > > > > > and
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > >> filed
> > > > > > > > > > > > > > > >> > > some
> > > > > > > > > > > > > > > >> > > > > > bugs and driven some feature
> > > requests, I
> > > > > > would
> > > > > > > > > like
> > > > > > > > > > to
> > > > > > > > > > > > ask
> > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > >> you
> > > > > > > > > > > > > > > >> > > could
> > > > > > > > > > > > > > > >> > > > > > help us within what is reasonably
> > > > possible
> > > > > > for
> > > > > > > > you
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> > > following:
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > - system test the latest
> > > > > storm-kafka-client
> > > > > > > > > changes
> > > > > > > > > > > in
> > > > > > > > > > > > > your
> > > > > > > > > > > > > > > >> > > > > > test/pre-production environment
> > > > > > > > > > > > > > > >> > > > > > - provide some information on
> about
> > > > your
> > > > > > > setup,
> > > > > > > > > > > > > > > >> > > > > > - how you are using
> > > > > storm-kafka-client
> > > > > > > > > > > > > > > >> > > > > > - how it is performing
> > > > > > > > > > > > > > > >> > > > > > - Kafka brokers, number of
> > > > > > > > > topics/partitions,
> > > > > > > > > > > > storm
> > > > > > > > > > > > > > > >> > > parallelism,
> > > > > > > > > > > > > > > >> > > > > and
> > > > > > > > > > > > > > > >> > > > > > some info about your network
> > > > > > > > > > > > > > > >> > > > > > - if it is reasonable to do so
> share
> > > > some
> > > > > of
> > > > > > > > your
> > > > > > > > > > > tests
> > > > > > > > > > > > > such
> > > > > > > > > > > > > > > >> that
> > > > > > > > > > > > > > > >> > we
> > > > > > > > > > > > > > > >> > > > can
> > > > > > > > > > > > > > > >> > > > > > also test it at our end.
> > > > > > > > > > > > > > > >> > > > > > - Tell us specific things that you
> > > would
> > > > > > like
> > > > > > > us
> > > > > > > > > to
> > > > > > > > > > > test
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > > > > > > > >> > > > > > Hugo
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 PM,
> Stig
> > > > Rohde
> > > > > > > > > Døssing <
> > > > > > > > > > > > > > > >> > > > > stigdoess...@gmail.com>
> > > > > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > I agree with Jungtaek, if
> metrics
> > v2
> > > > can
> > > > > > go
> > > > > > > in
> > > > > > > > > > very
> > > > > > > > > > > > soon
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > >> > should
> > > > > > > > > > > > > > > >> > > go
> > > > > > > > > > > > > > > >> > > > > in,
> > > > > > > > > > > > > > > >> > > > > > > otherwise I'd rather release
> 1.2.0
> > > now
> > > > > and
> > > > > > > > work
> > > > > > > > > on
> > > > > > > > > > > > > getting
> > > > > > > > > > > > > > > >> 2.0.0
> > > > > > > > > > > > > > > >> > > > ready
> > > > > > > > > > > > > > > >> > > > > > for
> > > > > > > > > > > > > > > >> > > > > > > release.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 1.x and master have drifted
> pretty
> > > far
> > > > > > > apart,
> > > > > > > > > and
> > > > > > > > > > > it's
> > > > > > > > > > > > > > > >> causing a
> > > > > > > > > > > > > > > >> > > lot
> > > > > > > > > > > > > > > >> > > > of
> > > > > > > > > > > > > > > >> > > > > > > porting work at this point. 1.x
> > > still
> > > > > has
> > > > > > a
> > > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > > Clojure
> > > > > > > > > > > > > > > >> code,
> > > > > > > > > > > > > > > >> > > > > > storm-core
> > > > > > > > > > > > > > > >> > > > > > > hasn't been split yet in that
> > branch
> > > > and
> > > > > > > it's
> > > > > > > > > also
> > > > > > > > > > > > > > targeting
> > > > > > > > > > > > > > > >> JDK
> > > > > > > > > > > > > > > >> > > 1.7.
> > > > > > > > > > > > > > > >> > > > > > It's
> > > > > > > > > > > > > > > >> > > > > > > seems rare at this point that a
> PR
> > > > > cherry
> > > > > > > > picks
> > > > > > > > > > > > cleanly
> > > > > > > > > > > > > > onto
> > > > > > > > > > > > > > > >> 1.x
> > > > > > > > > > > > > > > >> > > from
> > > > > > > > > > > > > > > >> > > > > > > master.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > I've linked the list of issues
> > that
> > > > are
> > > > > > only
> > > > > > > > > fixed
> > > > > > > > > > > in
> > > > > > > > > > > > > > 2.0.0
> > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > >> > not
> > > > > > > > > > > > > > > >> > > > > 1.x,
> > > > > > > > > > > > > > > >> > > > > > to
> > > > > > > > > > > > > > > >> > > > > > > illustrate how far ahead 2.0.0
> is.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > https://pste.eu/p/9CJT.html
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun
> > Iyer
> > > <
> > > > > > > > > > > > > > ai...@hortonworks.com
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >> Hi Taylor,
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >> Is it
> https://github.com/apache/
> > > > > > > > > storm/pull/2203
> > > > > > > > > > ?
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >> I think it would be great to
> get
> > > this
> > > > > in
> > > > > > > 1.2
> > > > > > > > > > > release.
> > > > > > > > > > > > > Can
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > >> try
> > > > > > > > > > > > > > > >> > > to
> > > > > > > > > > > > > > > >> > > > > > >> address the issues in a week or
> > so
> > > > and
> > > > > > get
> > > > > > > > this
> > > > > > > > > > in?
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >> Thanks,
> > > > > > > > > > > > > > > >> > > > > > >> Arun
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >> On 11/17/17, 7:13 AM, "P.
> Taylor
> > > > > Goetz" <
> > > > > > > > > > > > > > ptgo...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > wrote:
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > > >>> The original idea for the 1.2
> > > > release
> > > > > > was
> > > > > > > > that
> > > > > > > > > > > would
> > > > > > > > > > > > > > > >> introduce
> > > > > > > > > > > > > > > >> > > the
> > > > > > > > > > > > > > > >> > > > > > >> metrics v2 work (and there is
> > > > > additional
> > > > > > > work
> > > > > > > > > > > > required
> > > > > > > > > > > > > > > there,
> > > > > > > > > > > > > > > >> > so I
> > > > > > > > > > > > > > > >> > > > > > >> understand a desire not to
> delay
> > > the
> > > > > > > > release).
> > > > > > > > > Do
> > > > > > > > > > > we
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> > stick
> > > > > > > > > > > > > > > >> > > > > with
> > > > > > > > > > > > > > > >> > > > > > >> that, or deviate? If the
> latter,
> > > > would
> > > > > we
> > > > > > > do
> > > > > > > > a
> > > > > > > > > > 1.3
> > > > > > > > > > > > > > release
> > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > >> > > > > metrics?
> > > > > > > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > > > > > > >> > > > > > >>> As far as a 1.1.2 release I’m
> > fine
> > > > > with
> > > > > > > > > > releasing
> > > > > > > > > > > > that
> > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > >> any
> > > > > > > > > > > > > > > >> > > time.
> > > > > > > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > > > > > > >> > > > > > >>> -Taylor
> > > > > > > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > > > > > > >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM,
> > > > Jungtaek
> > > > > > > Lim <
> > > > > > > > > > > > > > > >> kabh...@gmail.com>
> > > > > > > > > > > > > > > >> > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > > > > > > >> > > > > > >>>> I think we could start
> release
> > > > phase
> > > > > > for
> > > > > > > > both
> > > > > > > > > > > 1.1.2
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> 1.2.0
> > > > > > > > > > > > > > > >> > > when
> > > > > > > > > > > > > > > >> > > > > > >>>> https://github.com/apache/
> > > > > > > storm/pull/2423
> > > > > > > > > will
> > > > > > > > > > > be
> > > > > > > > > > > > > > > merged.
> > > > > > > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > > > > > > >> > > > > > >>>> Thanks,
> > > > > > > > > > > > > > > >> > > > > > >>>> Jungtaek Lim (HeartSaVioR)
> > > > > > > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > > > > > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00,
> > > > Alexandre
> > > > > > > > > > > Vermeerbergen
> > > > > > > > > > > > <
> > > > > > > > > > > > > > > >> > > > > > >> avermeerber...@gmail.com>님이
> > > > > > > > > > > > > > > >> > > > > > >>>> 작성:
> > > > > > > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>> Hello,
> > > > > > > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>> I'd love to see a Storm
> 1.2.0
> > > > > > released:
> > > > > > > > > it's a
> > > > > > > > > > > > > perfect
> > > > > > > > > > > > > > > >> > schedule
> > > > > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > > > > >> > > > > > me
> > > > > > > > > > > > > > > >> > > > > > >> for
> > > > > > > > > > > > > > > >> > > > > > >>>>> test any Release Candidate
> > that
> > > > > might
> > > > > > be
> > > > > > > > > > > available
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > >> > > happens
> > > > > > > > > > > > > > > >> > > > > > soon.
> > > > > > > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>> Best regards,
> > > > > > > > > > > > > > > >> > > > > > >>>>> Alexandre
> > > > > > > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00
> Arun
> > > > > > Mahadevan
> > > > > > > > <
> > > > > > > > > > > > > > > >> ar...@apache.org>:
> > > > > > > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>> Hi,
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>> Looks like we are only
> > waiting
> > > on
> > > > > > > > > > > > > > > >> > https://issues.apache.org/
> > > > > > > > > > > > > > > >> > > > > > >>>>>> jira/browse/STORM-2546 .
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>> Are there any other issues
> > > which
> > > > > are
> > > > > > > > > blockers
> > > > > > > > > > > for
> > > > > > > > > > > > > > Storm
> > > > > > > > > > > > > > > >> > 1.2.0?
> > > > > > > > > > > > > > > >> > > > > Would
> > > > > > > > > > > > > > > >> > > > > > >> be
> > > > > > > > > > > > > > > >> > > > > > >>>>>> great to see the 1.2.0
> > release
> > > > out
> > > > > > soon
> > > > > > > > as
> > > > > > > > > it
> > > > > > > > > > > > has a
> > > > > > > > > > > > > > lot
> > > > > > > > > > > > > > > >> of
> > > > > > > > > > > > > > > >> > > > > critical
> > > > > > > > > > > > > > > >> > > > > > >>>>> fixes.
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>> Thanks,
> > > > > > > > > > > > > > > >> > > > > > >>>>>> Arun
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > > > > > > >> > > > > > >>
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
