Hi Stig,
Here's the source of our "BasicKafkaSpout" class's constructor (I can send
the full source if needed),
public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) {
this.kafkaBrokers = (String) config.getKafkaProps().get(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
this.consumerId = config.getConsumerGroupId();
logger.debug("topics are: {}", config.getSubscription()
.getTopicsString());
String topicsStr = config.getSubscription().getTopicsString();
this.topics = topicsStr.substring(1, topicsStr.length() -
1).split(",");
switch (config.getFirstPollOffsetStrategy()) {
case UNCOMMITTED_EARLIEST:
case EARLIEST:
this.strategy = "earliest";
break;
case UNCOMMITTED_LATEST:
case LATEST:
default:
this.strategy = "latest";
}
this.keyDeserializer = config.getKeyDeserializer().getClass();
this.valueDeserializer = config.getValueDeserializer().getClass();
this.translator = config.getTranslator();
this.consumerProps = new Properties();
if (config.getKafkaProps() != null) {
consumerProps.putAll(config.getKafkaProps());
}
}
line 72 is the following one:
this.keyDeserializer = config.getKeyDeserializer().getClass();
Hope it helps,
Alexandre Vermeerbergen
2017-11-19 15:37 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> Alexandre,
>
> I'm sorry this is giving you so much trouble. Looking at the stack trace
> you posted, it seems like the NPE is coming from BasicKafkaSpout line 72.
> Can you post that line (and maybe some of the surrounding code)?
>
> 2017-11-19 15:13 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com
> >:
>
> > Hi Stig,
> >
> > After having included commons-lang-2.5.jar in my topologies, I still have
> > 11 topologies failing with this new message, this time a
> > NullPointerException:
> >
> > Running: /usr/local/jdk/bin/java -client -Ddaemon.name= -Dstorm.options=
> > -Dstorm.home=/usr/local/Storm/storm-stable
> > -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > -Dstorm.conf.file= -cp
> > /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-
> > stable/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/
> > local/Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-
> > stable/conf:/usr/local/Storm/storm-stable/bin
> > -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> > -Dstorm.dependency.jars=
> > -Dstorm.dependency.artifacts={}
> > com.acme.storm.evaluator.SLAEventsInterceptorTopology
> slaEventsInterceptor
> > 1939 [main] WARN c.a.s.u.StormUtils - Couldn't read
> > /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties
> file:
> > /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties (No
> > such file or directory)
> > 1978 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
> > enable.auto.commit manually. Instead use
> > KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated as
> > an
> > error in the next major release. For now the spout will be configured to
> > behave like it would have in pre-1.2.0 releases.
> > 1982 [main] INFO c.a.s.u.StormUtils - Use Kafka Spout:
> > com.acme.storm.evaluator.spout.BasicKafkaSpout
> > Exception in thread "main" java.lang.NullPointerException
> > at
> > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(
> > BasicKafkaSpout.java:72)
> > at com.acme.storm.util.StormUtils.getKafkaSpout(
> > StormUtils.java:888)
> > at
> > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(
> > SLAEventsInterceptorTopology.java:120)
> >
> > and here's the line from our source from which this exception was
> > triggered:
> >
> > IRichSpout kafkaSpoutForMainTopic = StormUtils
> > .getKafkaSpout(spoutConfigForMainTopic,
> kafkaSpoutClass);
> >
> > Looks like upgrade from 1.1.0 (our current production setup) to 1.2.0 is
> a
> > bit more difficult than I expected.
> > But anyway, I greatly appreciate Storm, so I hope my feedbacks will help
> > keeping it as good as possible :)
> >
> > Please let me know if further details are required to solve this issue
> with
> > storm-kafka-client 1.2.0 snapshot.
> >
> > Best regards,
> > Alexandre
> >
> >
> > 2017-11-19 13:26 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> >
> > > Using Maven and controlling dependencies aren't mutually exclusive.
> Tools
> > > like Nexus (https://help.sonatype.com/display/NXRM2/Procurement+Suite)
> > > allow you to control access to dependencies without manually putting
> jars
> > > in an SCM.
> > >
> > > I realize this doesn't help you right now, just thought I'd mention it
> so
> > > you don't get the idea that I'm advocating a build process that blindly
> > > includes new dependencies.
> > >
> > > 2017-11-19 13:05 GMT+01:00 Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com
> > > >:
> > >
> > > > Believe it or not, we don't internally use Maven : we strictly
> control
> > > our
> > > > dependencies.
> > > >
> > > > So okay, I have delivered a "storm-commons-lang2.jar" in our SCM
> system
> > > in
> > > > order to specifically add it to our topologies big jars depending on
> > > > storm-kafka-client (feeling sad).
> > > >
> > > > I'll tell you the outcome ASAP.
> > > >
> > > > Best regard,
> > > > Alexandre Vermeerbergen
> > > >
> > > >
> > > > 2017-11-19 12:57 GMT+01:00 Stig Rohde Døssing <
> stigdoess...@gmail.com
> > >:
> > > >
> > > > > I also don't think that adding dependencies constitutes a breaking
> > > > change,
> > > > > nor should it. Dependency management tooling like Maven will handle
> > > > pulling
> > > > > the right dependencies automatically.
> > > > >
> > > > > 2017-11-19 12:34 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
> > > > >
> > > > > > Alexandre,
> > > > > >
> > > > > > There's so much pain on using relocated artifact in other module
> > in a
> > > > > > project. You need to refer the class as relocated name, and your
> > IDE
> > > > (at
> > > > > > least IntelliJ) complains and the compilation in IDE will not
> > > succeed.
> > > > > (The
> > > > > > build via Maven will succeed though.)
> > > > > > There was same issue on 2.0.0 as well, and for now we don't shade
> > > > > anything
> > > > > > in 2.0.0. We need to discuss which things to shade before
> > discussing
> > > > > about
> > > > > > releasing, indeed.
> > > > > >
> > > > > > And normally we have been allowing adding dependency as minor
> > > release.
> > > > I
> > > > > > understand someone could say that it is breaking dependency, and
> > > should
> > > > > > lead another relocation as well. But my 2 cents the requirement
> is
> > > too
> > > > > > restrictive, and may end up with Guava-like versioning.
> (releasing
> > > > major
> > > > > > versions sooner...)
> > > > > > I think relocating is not an ideal solution, and eventually we
> may
> > > have
> > > > > > nice solution (like classloader isolation) and then we could sort
> > out
> > > > all
> > > > > > the related issues. Before that, I could not block adding
> > > dependencies
> > > > in
> > > > > > minor release if it is necessary.
> > > > > >
> > > > > > Thanks,
> > > > > > Jungtaek Lim (HeartSaVioR)
> > > > > >
> > > > > > 2017년 11월 19일 (일) 오후 8:02, Stig Rohde Døssing <
> > > stigdoess...@gmail.com
> > > > >님이
> > > > > > 작성:
> > > > > >
> > > > > > > Alexandre,
> > > > > > >
> > > > > > > Thanks for trying. I think this time it's not a problem with
> > > > > > > storm-kafka-client. It has a dependency on commons-lang 2.5,
> > which
> > > is
> > > > > > > declared in the storm-kafka-client pom. We usually recommend
> that
> > > > > people
> > > > > > > use Maven/Gradle/Ivy or similar systems to ensure that they get
> > all
> > > > > > > dependencies when they build their topologies. In such a case
> > Maven
> > > > (or
> > > > > > > similar) would see that your topology depends on
> > > storm-kafka-client,
> > > > > and
> > > > > > > also download and include commons-lang in the topology jar.
> When
> > > you
> > > > > put
> > > > > > > together dependencies manually you need to include commons-lang
> > > > > manually
> > > > > > as
> > > > > > > well.
> > > > > > >
> > > > > > > You can get the commons-lang 2.5 jar here
> > > > > > >
> > > > > > > http://central.maven.org/maven2/commons-lang/commons-
> > > > > > lang/2.5/commons-lang-2.5.jar
> > > > > > >
> > > > > > > 2017-11-19 11:49 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > avermeerber...@gmail.com
> > > > > > > >:
> > > > > > >
> > > > > > > > Hello Stig,
> > > > > > > >
> > > > > > > > Than you very much for you quick answer & fix.
> > > > > > > > Unfortunaly, I still have topologies failing to start but
> with
> > a
> > > > > > > different
> > > > > > > > exception this time:
> > > > > > > >
> > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > > > org/apache/commons/lang/StringUtils
> > > > > > > > at
> > > > > > > > org.apache.storm.kafka.spout.NamedTopicFilter.
> getTopicsString(
> > > > > > > > NamedTopicFilter.java:66)
> > > > > > > > at
> > > > > > > > org.apache.storm.kafka.spout.ManualPartitionSubscription.
> > > > > > getTopicsString(
> > > > > > > > ManualPartitionSubscription.java:70)
> > > > > > > > at
> > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(
> > > > > > > > BasicKafkaSpout.java:59)
> > > > > > > > at com.acme.storm.util.StormUtils.getKafkaSpout(
> > > > > > > > StormUtils.java:888)
> > > > > > > > at
> > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(
> > > > > > > > SLAEventsInterceptorTopology.java:120)
> > > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > > org.apache.commons.lang.StringUtils
> > > > > > > > at java.net.URLClassLoader.
> > > findClass(URLClassLoader.java:
> > > > > 381)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:
> > 424)
> > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(
> > > > > > Launcher.java:338)
> > > > > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:
> > 357)
> > > > > > > >
> > > > > > > > Looks like you introduced a dependency on commons lang that
> > > wasn't
> > > > > part
> > > > > > > of
> > > > > > > > Storm, isn't it ?
> > > > > > > >
> > > > > > > > Keep faith, next time will be better :)
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Alexandre
> > > > > > > >
> > > > > > > > 2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <
> > > > > stigdoess...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > I've put up a fix here https://github.com/apache/
> > > storm/pull/2426
> > > > .
> > > > > > > > There's
> > > > > > > > > an updated storm-kafka-client jar at
> > > > > > > > > https://drive.google.com/file/d/
> > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > v9fd/view?usp=sharing
> > > > > > > > > if you'd like to try it out.
> > > > > > > > >
> > > > > > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <
> > > > > > stigdoess...@gmail.com
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Oops. The exception is not intentional, it's a bug. In
> > 1.2.0
> > > we
> > > > > > check
> > > > > > > > for
> > > > > > > > > > the "enable.auto.commit" key in the kafkaConsumerProp
> map,
> > > and
> > > > if
> > > > > > it
> > > > > > > is
> > > > > > > > > set
> > > > > > > > > > we warn in the log that it shouldn't be, because users
> > should
> > > > use
> > > > > > the
> > > > > > > > > > KafkaSpoutConfig.Builder.setProcessingGuarantee method
> > > > instead.
> > > > > > When
> > > > > > > > the
> > > > > > > > > > property is set we try to set the processing guarantee to
> > > match
> > > > > the
> > > > > > > > > > pre-1.2.0 behavior, but I made a mistake and assumed the
> > > > property
> > > > > > key
> > > > > > > > is
> > > > > > > > > a
> > > > > > > > > > boolean when it might actually be either a boolean or a
> > > string.
> > > > > > I'll
> > > > > > > > put
> > > > > > > > > up
> > > > > > > > > > a fix ASAP.
> > > > > > > > > >
> > > > > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00 Alexandre
> > > > > > Vermeerbergen
> > > > > > > > <
> > > > > > > > > > avermeerber...@gmail.com>:
> > > > > > > > > >
> > > > > > > > > >> Hello Stig,
> > > > > > > > > >>
> > > > > > > > > >> Here's my first feedback on this Storm 1.2.0 preview on
> my
> > > > > > > Supervision
> > > > > > > > > >> system based on Storm : I have 11 topologies KO (not
> even
> > > able
> > > > > to
> > > > > > > > > start),
> > > > > > > > > >> and 4 topologies which seem to be unaffected.
> > > > > > > > > >>
> > > > > > > > > >> Details:
> > > > > > > > > >> - I used the binaries posted on dropbox by Jungteak and
> > your
> > > > > > binary
> > > > > > > > for
> > > > > > > > > >> storm-kafka-client
> > > > > > > > > >> - Rebuilt all our topologies using these storm-core.jar
> &
> > > > > > > > > >> storm-kafka-client.jar from these storm-1.2.0-snapshot
> > (and
> > > I
> > > > > got
> > > > > > > > > >> absolutely no error at build time).
> > > > > > > > > >> Meaning that I have new "big jar" files including this
> > > newer
> > > > > > > > > >> storm-kafka-client.jar (except for 1 topology which
> > depends
> > > on
> > > > > > > another
> > > > > > > > > >> build system)
> > > > > > > > > >> - The 11 topologies which fail to start show up the
> > > following
> > > > > > trace
> > > > > > > > when
> > > > > > > > > >> they are submitted:
> > > > > > > > > >>
> > > > > > > > > >> Running: java -client -Ddaemon.name= -Dstorm.options=
> > > > > > > > > >> -Dstorm.home=/usr/local/Storm/storm-stable
> > > > > > > > > >> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> > > > > > > > > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/
> > lib
> > > > > > > > > >> -Dstorm.conf.file= -cp
> > > > > > > > > >> /usr/local/Storm/storm-stable/
> > > *:/usr/local/Storm/storm-stabl
> > > > > > > > > >> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/
> > local/
> > > > > > > > > >> Storm/StormTopologiesBJ.jar:/
> > usr/local/Storm/storm-stable/
> > > > > > > > > >> conf:/usr/local/Storm/storm-stable/bin
> > > > > > > > > >> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> > > > > > > > > >> -Dstorm.dependency.jars=
> > > > > > > > > >> -Dstorm.dependency.artifacts={}
> > > > > > > > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology
> > > > > > > > > >> slaEventsInterceptor
> > > > > > > > > >> Latest
> > > > > > > > > >> 1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology
> -
> > > > > > Topology :
> > > > > > > > > >> 'slaEventsInterceptor', KafkaOffsetTimeStrategy =
> 'Latest'
> > > > > > > > > >> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read
> > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > kafkaConsumer.properties
> > > > > > > > > file:
> > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > kafkaConsumer.properties
> > > > > > > > (No
> > > > > > > > > >> such file or directory)
> > > > > > > > > >> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not
> set
> > > > > > > > > >> enable.auto.commit manually. Instead use
> > > > > > > > > >> KafkaSpoutConfig.Builder.setProcessingGuarantee. This
> > will
> > > be
> > > > > > > treated
> > > > > > > > > as
> > > > > > > > > >> an
> > > > > > > > > >> error in the next major release. For now the spout will
> be
> > > > > > > configured
> > > > > > > > to
> > > > > > > > > >> behave like it would have in pre-1.2.0 releases.
> > > > > > > > > >> Exception in thread "main" java.lang.ClassCastException:
> > > > > > > > > java.lang.String
> > > > > > > > > >> cannot be cast to java.lang.Boolean
> > > > > > > > > >> at
> > > > > > > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.
> > > setAutoCommitM
> > > > > > > > > >> ode(KafkaSpoutConfig.java:721)
> > > > > > > > > >> at
> > > > > > > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(
> > > KafkaSp
> > > > > > > > > >> outConfig.java:97)
> > > > > > > > > >> at
> > > > > > > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.
> > > build(
> > > > > > > > > >> KafkaSpoutConfig.java:671)
> > > > > > > > > >> at
> > > > > > > > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology.
> > > main(S
> > > > > > > > > >> LAEventsInterceptorTopology.java:118)
> > > > > > > > > >>
> > > > > > > > > >> The line part of our application which triggered above
> > stack
> > > > > trace
> > > > > > > is
> > > > > > > > > the
> > > > > > > > > >> following one:
> > > > > > > > > >>
> > > > > > > > > >> KafkaSpoutConfig<String, Event>
> > > > spoutConfigForMainTopic
> > > > > =
> > > > > > > > > >> KafkaSpoutConfig
> > > > > > > > > >> .builder(elasticKafkaBrokers,
> > > > KafkaTopics.MAIN)
> > > > > > > > > >> .setValue(EventKafkaDeserializer.class)
> > > > > > > > > >> .setGroupId(consumerId + "_" +
> > > > KafkaTopics.MAIN)
> > > > > > > > > >> .setFirstPollOffsetStrategy(strategy)
> > > > > > > > > >> .setProp(kafkaConsumerProp)
> > > > > > > > > >> .setRecordTranslator(
> > > > > > > > > >> new
> SLAEventsInterceptorKafkaRecor
> > > > > > > > > dTranslator())
> > > > > > > > > >> .build();
> > > > > > > > > >>
> > > > > > > > > >> I understand that there's a WARN message, but if it's a
> > > > warning,
> > > > > > > then
> > > > > > > > > why
> > > > > > > > > >> do we exit with such a brutal exception?
> > > > > > > > > >>
> > > > > > > > > >> Best regards,
> > > > > > > > > >> Alexandre Vermeerbergen
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <
> > > > > > > stigdoess...@gmail.com
> > > > > > > > >:
> > > > > > > > > >>
> > > > > > > > > >> > Alexandre,
> > > > > > > > > >> >
> > > > > > > > > >> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client
> jar
> > > here
> > > > > > > > > >> > https://drive.google.com/file/d/
> > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > >> > v9fd/view?usp=sharing.
> > > > > > > > > >> > You can probably keep the 1.1.0 versions of
> storm-hbase
> > > and
> > > > > > > > > storm-hdfs,
> > > > > > > > > >> > since the core Storm API hasn't changed.
> > > > > > > > > >> >
> > > > > > > > > >> > If you want to build the jars yourself, clone the
> > > repository
> > > > > at
> > > > > > > > > >> > https://github.com/apache/storm and check out the
> > > > 1.x-branch
> > > > > > > > branch.
> > > > > > > > > >> You
> > > > > > > > > >> > can build the entire project by running "mvn clean
> > install
> > > > > > > > > -DskipTests"
> > > > > > > > > >> in
> > > > > > > > > >> > the root. The individual module jars will be available
> > in
> > > > the
> > > > > > > target
> > > > > > > > > >> > directories for each module, e.g.
> storm-kafka-client.jar
> > > > will
> > > > > be
> > > > > > > in
> > > > > > > > > >> > external/storm-kafka-client/target.
> > > > > > > > > >> >
> > > > > > > > > >> > If you need to build the regular Storm distribution
> (the
> > > > > tarball
> > > > > > > > > >> Jungtaek
> > > > > > > > > >> > sent you), you can go to storm-dist/binary and run
> "mvn
> > > > clean
> > > > > > > > install
> > > > > > > > > >> > -Dgpg.skip". The tar/zip will be in
> > > > storm-dist/binary/target.
> > > > > > > > > >> >
> > > > > > > > > >> > Further reference in case you need it
> > > > > > > > > >> > https://github.com/apache/storm/blob/master/DEVELOPER
> .
> > > > > > > > > >> > md#build-the-code-and-run-the-tests
> > > > > > > > > >> > .
> > > > > > > > > >> >
> > > > > > > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > > > >> > avermeerber...@gmail.com
> > > > > > > > > >> > >:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hello Jungtaek,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Thanks for the link to 1.2.0 preview binaries.
> > > > > > > > > >> > > However, since we heavily depend on Storm Kafka
> > Client,
> > > > > would
> > > > > > > you
> > > > > > > > > >> please
> > > > > > > > > >> > > either add it or remind me how to build the jars of
> > this
> > > > > > > external
> > > > > > > > > >> lib, at
> > > > > > > > > >> > > same "snapshot" version as the rest ?
> > > > > > > > > >> > > On a side note, some of our topologies also rely on
> > > storm
> > > > > > hbase
> > > > > > > &
> > > > > > > > > >> storm
> > > > > > > > > >> > > hdfs : does it matters if these later ones stay in
> > 1.1.0
> > > > > > version
> > > > > > > > for
> > > > > > > > > >> this
> > > > > > > > > >> > > test?
> > > > > > > > > >> > > Indeed, I want to focus on Storm 1.2.0 "core" +
> Storm
> > > > Kafka
> > > > > > > Client
> > > > > > > > > >> 1.2.0
> > > > > > > > > >> > >
> > > > > > > > > >> > > Best regards,
> > > > > > > > > >> > > Alexandre Vermeerbergen
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00
> Jungtaek
> > > > Lim <
> > > > > > > > > >> kabh...@gmail.com>:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Alexandre,
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > https://www.dropbox.com/s/
> > > mg2gnunk24oesyc/apache-storm-
> > > > > > > > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Above link is custom binary distribution of
> current
> > > > > > 1.x-branch
> > > > > > > > > >> > (SNAPSHOT
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > 1.2.0). Could you run the test in your environment
> > > first
> > > > > and
> > > > > > > let
> > > > > > > > > us
> > > > > > > > > >> > know
> > > > > > > > > >> > > > about the result? Regardless of including metrics
> V2
> > > to
> > > > > > 1.2.0
> > > > > > > or
> > > > > > > > > >> not,
> > > > > > > > > >> > > your
> > > > > > > > > >> > > > test report should be valuable for us.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > And please refer my analysis of current metrics:
> > > > > > > > > >> > > > https://cwiki.apache.org/
> confluence/display/STORM/
> > > > > > > > > >> > > Limitations+of+current+
> > > > > > > > > >> > > > metrics+feature
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > see why we want to move toward to Metrics V2.
> > Current
> > > PR
> > > > > on
> > > > > > > > > Metrics
> > > > > > > > > >> V2
> > > > > > > > > >> > is
> > > > > > > > > >> > > > for initial state and it could cover only some of
> > > issues
> > > > > in
> > > > > > > > list,
> > > > > > > > > >> but I
> > > > > > > > > >> > > > expect we will address another issues as well
> based
> > on
> > > > the
> > > > > > PR.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Thanks in advance!
> > > > > > > > > >> > > > Jungtaek Lim (HeartSaVioR)
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre
> Vermeerbergen <
> > > > > > > > > >> > > > avermeerber...@gmail.com>님이
> > > > > > > > > >> > > > 작성:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Hello Hugo,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > As I already posted, I was getting ready to test
> > the
> > > > > > > upcoming
> > > > > > > > > RC,
> > > > > > > > > >> so
> > > > > > > > > >> > > > yeah :
> > > > > > > > > >> > > > > as soon as I have some binaries for testing, I
> > can
> > > > run
> > > > > > any
> > > > > > > > > Storm
> > > > > > > > > >> > 1.2.0
> > > > > > > > > >> > > > > preview on my pre-production (well stressed)
> > > > > environment.
> > > > > > Is
> > > > > > > > > there
> > > > > > > > > >> > some
> > > > > > > > > >> > > > URL
> > > > > > > > > >> > > > > from where I can download such "snapshot"
> > binaries?
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best regards,
> > > > > > > > > >> > > > > Alexandre Vermeerbergen
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > PS: By the way, I have no interest in the
> > > > > metrics-related
> > > > > > > > > >> > enhancements
> > > > > > > > > >> > > > you
> > > > > > > > > >> > > > > mentionned - I can understand it is of interest
> > for
> > > > > people
> > > > > > > > aware
> > > > > > > > > >> of
> > > > > > > > > >> > it
> > > > > > > > > >> > > > and
> > > > > > > > > >> > > > > who are expecting "something" related to it -
> but
> > I
> > > > > would
> > > > > > be
> > > > > > > > sad
> > > > > > > > > >> to
> > > > > > > > > >> > > see a
> > > > > > > > > >> > > > > postponing of the imminent Storm 1.2.0 release
> for
> > > > > > something
> > > > > > > > we
> > > > > > > > > do
> > > > > > > > > >> > not
> > > > > > > > > >> > > > use
> > > > > > > > > >> > > > > at all.
> > > > > > > > > >> > > > > PPS: I would love to learn more about these
> > > "metrics",
> > > > > > maybe
> > > > > > > > > it's
> > > > > > > > > >> > > > something
> > > > > > > > > >> > > > > worth which never quite understood... any link
> > with
> > > > > clear
> > > > > > > > > >> explanation
> > > > > > > > > >> > > of
> > > > > > > > > >> > > > > this feature?
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro <
> > > > > > > > > >> > hlo...@hortonworks.com
> > > > > > > > > >> > > >:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > I am also in agreement that we should not
> delay
> > > the
> > > > > > 1.2.0
> > > > > > > > > >> release
> > > > > > > > > >> > for
> > > > > > > > > >> > > > too
> > > > > > > > > >> > > > > > long, but in order to release it a few things
> > need
> > > > to
> > > > > be
> > > > > > > > kept
> > > > > > > > > in
> > > > > > > > > >> > > mind:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > - If we want to avoid releasing 1.3.0 soon or
> > at
> > > > all,
> > > > > > > that
> > > > > > > > > >> means
> > > > > > > > > >> > > that
> > > > > > > > > >> > > > we
> > > > > > > > > >> > > > > > should include in 1.2.0 as many important
> > features
> > > > > (e.g.
> > > > > > > > > >> metricsV2)
> > > > > > > > > >> > > as
> > > > > > > > > >> > > > > > possible within a reasonable time frame. Also
> to
> > > > avoid
> > > > > > as
> > > > > > > > much
> > > > > > > > > >> back
> > > > > > > > > >> > > > > porting
> > > > > > > > > >> > > > > > as possible one should really get all features
> > > that
> > > > we
> > > > > > > > foresee
> > > > > > > > > >> we
> > > > > > > > > >> > > want
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > > maintain in 1.2.0 right away, and then simply
> > > > maintain
> > > > > > > them
> > > > > > > > > >> there.
> > > > > > > > > >> > > > > However
> > > > > > > > > >> > > > > > it’s a fact of life that if a bug is found and
> > it
> > > > is a
> > > > > > > > > blocker,
> > > > > > > > > >> it
> > > > > > > > > >> > > will
> > > > > > > > > >> > > > > > have to be back ported.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > - STORM-2153 details that the new metrics
> > > > requirements
> > > > > > > were
> > > > > > > > > >> driven
> > > > > > > > > >> > by
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > users. Therefore if this feature is highly
> > > important
> > > > > and
> > > > > > > > > sought
> > > > > > > > > >> > > after,
> > > > > > > > > >> > > > it
> > > > > > > > > >> > > > > > may not make much sense to release 1.2.0
> without
> > > it.
> > > > > If
> > > > > > we
> > > > > > > > do
> > > > > > > > > >> > release
> > > > > > > > > >> > > > > 1.2.0
> > > > > > > > > >> > > > > > without the metrics, does it mean that metrics
> > > will
> > > > go
> > > > > > > only
> > > > > > > > in
> > > > > > > > > >> > 2.0? I
> > > > > > > > > >> > > > > would
> > > > > > > > > >> > > > > > conjecture that most production deployments
> will
> > > > take
> > > > > a
> > > > > > > > while
> > > > > > > > > to
> > > > > > > > > >> > > > upgrade
> > > > > > > > > >> > > > > to
> > > > > > > > > >> > > > > > 2.0 even after it is released. That means that
> > > they
> > > > > will
> > > > > > > > still
> > > > > > > > > >> be
> > > > > > > > > >> > > > running
> > > > > > > > > >> > > > > > Storm without the benefit of the new metrics.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > - Several fixes have got in
> storm-kafka-client.
> > > > There
> > > > > > > have
> > > > > > > > > >> been a
> > > > > > > > > >> > > lot
> > > > > > > > > >> > > > of
> > > > > > > > > >> > > > > > changes and I wonder to which degree they have
> > > been
> > > > > > system
> > > > > > > > > >> tested
> > > > > > > > > >> > in
> > > > > > > > > >> > > > > > addition the existing unit tests
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > @Alexandre, since you are using
> > storm-kafka-client
> > > > and
> > > > > > > have
> > > > > > > > > >> filed
> > > > > > > > > >> > > some
> > > > > > > > > >> > > > > > bugs and driven some feature requests, I would
> > > like
> > > > to
> > > > > > ask
> > > > > > > > if
> > > > > > > > > >> you
> > > > > > > > > >> > > could
> > > > > > > > > >> > > > > > help us within what is reasonably possible for
> > you
> > > > > with
> > > > > > > the
> > > > > > > > > >> > > following:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > - system test the latest storm-kafka-client
> > > changes
> > > > > in
> > > > > > > your
> > > > > > > > > >> > > > > > test/pre-production environment
> > > > > > > > > >> > > > > > - provide some information on about your
> setup,
> > > > > > > > > >> > > > > > - how you are using storm-kafka-client
> > > > > > > > > >> > > > > > - how it is performing
> > > > > > > > > >> > > > > > - Kafka brokers, number of
> > > topics/partitions,
> > > > > > storm
> > > > > > > > > >> > > parallelism,
> > > > > > > > > >> > > > > and
> > > > > > > > > >> > > > > > some info about your network
> > > > > > > > > >> > > > > > - if it is reasonable to do so share some of
> > your
> > > > > tests
> > > > > > > such
> > > > > > > > > >> that
> > > > > > > > > >> > we
> > > > > > > > > >> > > > can
> > > > > > > > > >> > > > > > also test it at our end.
> > > > > > > > > >> > > > > > - Tell us specific things that you would like
> us
> > > to
> > > > > test
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > >> > > > > > Hugo
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde
> > > Døssing <
> > > > > > > > > >> > > > > stigdoess...@gmail.com>
> > > > > > > > > >> > > > > > wrote:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > I agree with Jungtaek, if metrics v2 can go
> in
> > > > very
> > > > > > soon
> > > > > > > > it
> > > > > > > > > >> > should
> > > > > > > > > >> > > go
> > > > > > > > > >> > > > > in,
> > > > > > > > > >> > > > > > > otherwise I'd rather release 1.2.0 now and
> > work
> > > on
> > > > > > > getting
> > > > > > > > > >> 2.0.0
> > > > > > > > > >> > > > ready
> > > > > > > > > >> > > > > > for
> > > > > > > > > >> > > > > > > release.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 1.x and master have drifted pretty far
> apart,
> > > and
> > > > > it's
> > > > > > > > > >> causing a
> > > > > > > > > >> > > lot
> > > > > > > > > >> > > > of
> > > > > > > > > >> > > > > > > porting work at this point. 1.x still has a
> > lot
> > > of
> > > > > > > Clojure
> > > > > > > > > >> code,
> > > > > > > > > >> > > > > > storm-core
> > > > > > > > > >> > > > > > > hasn't been split yet in that branch and
> it's
> > > also
> > > > > > > > targeting
> > > > > > > > > >> JDK
> > > > > > > > > >> > > 1.7.
> > > > > > > > > >> > > > > > It's
> > > > > > > > > >> > > > > > > seems rare at this point that a PR cherry
> > picks
> > > > > > cleanly
> > > > > > > > onto
> > > > > > > > > >> 1.x
> > > > > > > > > >> > > from
> > > > > > > > > >> > > > > > > master.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > I've linked the list of issues that are only
> > > fixed
> > > > > in
> > > > > > > > 2.0.0
> > > > > > > > > >> and
> > > > > > > > > >> > not
> > > > > > > > > >> > > > > 1.x,
> > > > > > > > > >> > > > > > to
> > > > > > > > > >> > > > > > > illustrate how far ahead 2.0.0 is.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > https://pste.eu/p/9CJT.html
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer <
> > > > > > > > ai...@hortonworks.com
> > > > > > > > > >:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > >> Hi Taylor,
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >> Is it https://github.com/apache/
> > > storm/pull/2203
> > > > ?
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >> I think it would be great to get this in
> 1.2
> > > > > release.
> > > > > > > Can
> > > > > > > > > we
> > > > > > > > > >> try
> > > > > > > > > >> > > to
> > > > > > > > > >> > > > > > >> address the issues in a week or so and get
> > this
> > > > in?
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >> Thanks,
> > > > > > > > > >> > > > > > >> Arun
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" <
> > > > > > > > ptgo...@gmail.com
> > > > > > > > > >
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > > >>> The original idea for the 1.2 release was
> > that
> > > > > would
> > > > > > > > > >> introduce
> > > > > > > > > >> > > the
> > > > > > > > > >> > > > > > >> metrics v2 work (and there is additional
> work
> > > > > > required
> > > > > > > > > there,
> > > > > > > > > >> > so I
> > > > > > > > > >> > > > > > >> understand a desire not to delay the
> > release).
> > > Do
> > > > > we
> > > > > > > want
> > > > > > > > > to
> > > > > > > > > >> > stick
> > > > > > > > > >> > > > > with
> > > > > > > > > >> > > > > > >> that, or deviate? If the latter, would we
> do
> > a
> > > > 1.3
> > > > > > > > release
> > > > > > > > > >> for
> > > > > > > > > >> > > > > metrics?
> > > > > > > > > >> > > > > > >>>
> > > > > > > > > >> > > > > > >>> As far as a 1.1.2 release I’m fine with
> > > > releasing
> > > > > > that
> > > > > > > > at
> > > > > > > > > >> any
> > > > > > > > > >> > > time.
> > > > > > > > > >> > > > > > >>>
> > > > > > > > > >> > > > > > >>> -Taylor
> > > > > > > > > >> > > > > > >>>
> > > > > > > > > >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek
> Lim <
> > > > > > > > > >> kabh...@gmail.com>
> > > > > > > > > >> > > > > wrote:
> > > > > > > > > >> > > > > > >>>>
> > > > > > > > > >> > > > > > >>>> I think we could start release phase for
> > both
> > > > > 1.1.2
> > > > > > > and
> > > > > > > > > >> 1.2.0
> > > > > > > > > >> > > when
> > > > > > > > > >> > > > > > >>>> https://github.com/apache/
> storm/pull/2423
> > > will
> > > > > be
> > > > > > > > > merged.
> > > > > > > > > >> > > > > > >>>>
> > > > > > > > > >> > > > > > >>>> Thanks,
> > > > > > > > > >> > > > > > >>>> Jungtaek Lim (HeartSaVioR)
> > > > > > > > > >> > > > > > >>>>
> > > > > > > > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre
> > > > > Vermeerbergen
> > > > > > <
> > > > > > > > > >> > > > > > >> avermeerber...@gmail.com>님이
> > > > > > > > > >> > > > > > >>>> 작성:
> > > > > > > > > >> > > > > > >>>>
> > > > > > > > > >> > > > > > >>>>> Hello,
> > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > >> > > > > > >>>>> I'd love to see a Storm 1.2.0 released:
> > > it's a
> > > > > > > perfect
> > > > > > > > > >> > schedule
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > me
> > > > > > > > > >> > > > > > >> for
> > > > > > > > > >> > > > > > >>>>> test any Release Candidate that might be
> > > > > available
> > > > > > > if
> > > > > > > > it
> > > > > > > > > >> > > happens
> > > > > > > > > >> > > > > > soon.
> > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > >> > > > > > >>>>> Best regards,
> > > > > > > > > >> > > > > > >>>>> Alexandre
> > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan
> > <
> > > > > > > > > >> ar...@apache.org>:
> > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > >> > > > > > >>>>>> Hi,
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>> Looks like we are only waiting on
> > > > > > > > > >> > https://issues.apache.org/
> > > > > > > > > >> > > > > > >>>>>> jira/browse/STORM-2546 .
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>> Are there any other issues which are
> > > blockers
> > > > > for
> > > > > > > > Storm
> > > > > > > > > >> > 1.2.0?
> > > > > > > > > >> > > > > Would
> > > > > > > > > >> > > > > > >> be
> > > > > > > > > >> > > > > > >>>>>> great to see the 1.2.0 release out soon
> > as
> > > it
> > > > > > has a
> > > > > > > > lot
> > > > > > > > > >> of
> > > > > > > > > >> > > > > critical
> > > > > > > > > >> > > > > > >>>>> fixes.
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>> Thanks,
> > > > > > > > > >> > > > > > >>>>>> Arun
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > >> > > > > > >>>
> > > > > > > > > >> > > > > > >>
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
