Thanks, I've addressed the issue here
https://github.com/apache/storm/pull/2428 and uploaded a new jar at the
same link here https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_
v9fd/view?usp=sharing. I went over the PR that made these changes, and I
don't believe anything else breaks backward compatibility, but we'll see.
2017-11-19 15:48 GMT+01:00 Alexandre Vermeerbergen <avermeerber...@gmail.com
>:
> Hi Stig,
>
> Here's the source of our "BasicKafkaSpout" class's constructor (I can send
> the full source if needed),
>
>
> public BasicKafkaSpout(KafkaSpoutConfig<K, V> config) {
> this.kafkaBrokers = (String) config.getKafkaProps().get(
> ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG);
> this.consumerId = config.getConsumerGroupId();
> logger.debug("topics are: {}", config.getSubscription()
> .getTopicsString());
> String topicsStr = config.getSubscription().getTopicsString();
> this.topics = topicsStr.substring(1, topicsStr.length() -
> 1).split(",");
> switch (config.getFirstPollOffsetStrategy()) {
> case UNCOMMITTED_EARLIEST:
> case EARLIEST:
> this.strategy = "earliest";
> break;
> case UNCOMMITTED_LATEST:
> case LATEST:
> default:
> this.strategy = "latest";
> }
> this.keyDeserializer = config.getKeyDeserializer().getClass();
> this.valueDeserializer = config.getValueDeserializer().getClass();
> this.translator = config.getTranslator();
> this.consumerProps = new Properties();
> if (config.getKafkaProps() != null) {
> consumerProps.putAll(config.getKafkaProps());
> }
> }
>
> line 72 is the following one:
>
> this.keyDeserializer = config.getKeyDeserializer().getClass();
>
> Hope it helps,
>
> Alexandre Vermeerbergen
>
> 2017-11-19 15:37 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
>
> > Alexandre,
> >
> > I'm sorry this is giving you so much trouble. Looking at the stack trace
> > you posted, it seems like the NPE is coming from BasicKafkaSpout line 72.
> > Can you post that line (and maybe some of the surrounding code)?
> >
> > 2017-11-19 15:13 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerber...@gmail.com
> > >:
> >
> > > Hi Stig,
> > >
> > > After having included commons-lang-2.5.jar in my topologies, I still
> have
> > > 11 topologies failing with this new message, this time a
> > > NullPointerException:
> > >
> > > Running: /usr/local/jdk/bin/java -client -Ddaemon.name=
> -Dstorm.options=
> > > -Dstorm.home=/usr/local/Storm/storm-stable
> > > -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> > > -Dstorm.conf.file= -cp
> > > /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-
> > > stable/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/
> > > local/Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-
> > > stable/conf:/usr/local/Storm/storm-stable/bin
> > > -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> > > -Dstorm.dependency.jars=
> > > -Dstorm.dependency.artifacts={}
> > > com.acme.storm.evaluator.SLAEventsInterceptorTopology
> > slaEventsInterceptor
> > > 1939 [main] WARN c.a.s.u.StormUtils - Couldn't read
> > > /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties
> > file:
> > > /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties
> (No
> > > such file or directory)
> > > 1978 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
> > > enable.auto.commit manually. Instead use
> > > KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated
> as
> > > an
> > > error in the next major release. For now the spout will be configured
> to
> > > behave like it would have in pre-1.2.0 releases.
> > > 1982 [main] INFO c.a.s.u.StormUtils - Use Kafka Spout:
> > > com.acme.storm.evaluator.spout.BasicKafkaSpout
> > > Exception in thread "main" java.lang.NullPointerException
> > > at
> > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(
> > > BasicKafkaSpout.java:72)
> > > at com.acme.storm.util.StormUtils.getKafkaSpout(
> > > StormUtils.java:888)
> > > at
> > > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(
> > > SLAEventsInterceptorTopology.java:120)
> > >
> > > and here's the line from our source from which this exception was
> > > triggered:
> > >
> > > IRichSpout kafkaSpoutForMainTopic = StormUtils
> > > .getKafkaSpout(spoutConfigForMainTopic,
> > kafkaSpoutClass);
> > >
> > > Looks like upgrade from 1.1.0 (our current production setup) to 1.2.0
> is
> > a
> > > bit more difficult than I expected.
> > > But anyway, I greatly appreciate Storm, so I hope my feedbacks will
> help
> > > keeping it as good as possible :)
> > >
> > > Please let me know if further details are required to solve this issue
> > with
> > > storm-kafka-client 1.2.0 snapshot.
> > >
> > > Best regards,
> > > Alexandre
> > >
> > >
> > > 2017-11-19 13:26 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com
> >:
> > >
> > > > Using Maven and controlling dependencies aren't mutually exclusive.
> > Tools
> > > > like Nexus (https://help.sonatype.com/display/NXRM2/Procurement+
> Suite)
> > > > allow you to control access to dependencies without manually putting
> > jars
> > > > in an SCM.
> > > >
> > > > I realize this doesn't help you right now, just thought I'd mention
> it
> > so
> > > > you don't get the idea that I'm advocating a build process that
> blindly
> > > > includes new dependencies.
> > > >
> > > > 2017-11-19 13:05 GMT+01:00 Alexandre Vermeerbergen <
> > > > avermeerber...@gmail.com
> > > > >:
> > > >
> > > > > Believe it or not, we don't internally use Maven : we strictly
> > control
> > > > our
> > > > > dependencies.
> > > > >
> > > > > So okay, I have delivered a "storm-commons-lang2.jar" in our SCM
> > system
> > > > in
> > > > > order to specifically add it to our topologies big jars depending
> on
> > > > > storm-kafka-client (feeling sad).
> > > > >
> > > > > I'll tell you the outcome ASAP.
> > > > >
> > > > > Best regard,
> > > > > Alexandre Vermeerbergen
> > > > >
> > > > >
> > > > > 2017-11-19 12:57 GMT+01:00 Stig Rohde Døssing <
> > stigdoess...@gmail.com
> > > >:
> > > > >
> > > > > > I also don't think that adding dependencies constitutes a
> breaking
> > > > > change,
> > > > > > nor should it. Dependency management tooling like Maven will
> handle
> > > > > pulling
> > > > > > the right dependencies automatically.
> > > > > >
> > > > > > 2017-11-19 12:34 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
> > > > > >
> > > > > > > Alexandre,
> > > > > > >
> > > > > > > There's so much pain on using relocated artifact in other
> module
> > > in a
> > > > > > > project. You need to refer the class as relocated name, and
> your
> > > IDE
> > > > > (at
> > > > > > > least IntelliJ) complains and the compilation in IDE will not
> > > > succeed.
> > > > > > (The
> > > > > > > build via Maven will succeed though.)
> > > > > > > There was same issue on 2.0.0 as well, and for now we don't
> shade
> > > > > > anything
> > > > > > > in 2.0.0. We need to discuss which things to shade before
> > > discussing
> > > > > > about
> > > > > > > releasing, indeed.
> > > > > > >
> > > > > > > And normally we have been allowing adding dependency as minor
> > > > release.
> > > > > I
> > > > > > > understand someone could say that it is breaking dependency,
> and
> > > > should
> > > > > > > lead another relocation as well. But my 2 cents the requirement
> > is
> > > > too
> > > > > > > restrictive, and may end up with Guava-like versioning.
> > (releasing
> > > > > major
> > > > > > > versions sooner...)
> > > > > > > I think relocating is not an ideal solution, and eventually we
> > may
> > > > have
> > > > > > > nice solution (like classloader isolation) and then we could
> sort
> > > out
> > > > > all
> > > > > > > the related issues. Before that, I could not block adding
> > > > dependencies
> > > > > in
> > > > > > > minor release if it is necessary.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jungtaek Lim (HeartSaVioR)
> > > > > > >
> > > > > > > 2017년 11월 19일 (일) 오후 8:02, Stig Rohde Døssing <
> > > > stigdoess...@gmail.com
> > > > > >님이
> > > > > > > 작성:
> > > > > > >
> > > > > > > > Alexandre,
> > > > > > > >
> > > > > > > > Thanks for trying. I think this time it's not a problem with
> > > > > > > > storm-kafka-client. It has a dependency on commons-lang 2.5,
> > > which
> > > > is
> > > > > > > > declared in the storm-kafka-client pom. We usually recommend
> > that
> > > > > > people
> > > > > > > > use Maven/Gradle/Ivy or similar systems to ensure that they
> get
> > > all
> > > > > > > > dependencies when they build their topologies. In such a case
> > > Maven
> > > > > (or
> > > > > > > > similar) would see that your topology depends on
> > > > storm-kafka-client,
> > > > > > and
> > > > > > > > also download and include commons-lang in the topology jar.
> > When
> > > > you
> > > > > > put
> > > > > > > > together dependencies manually you need to include
> commons-lang
> > > > > > manually
> > > > > > > as
> > > > > > > > well.
> > > > > > > >
> > > > > > > > You can get the commons-lang 2.5 jar here
> > > > > > > >
> > > > > > > > http://central.maven.org/maven2/commons-lang/commons-
> > > > > > > lang/2.5/commons-lang-2.5.jar
> > > > > > > >
> > > > > > > > 2017-11-19 11:49 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > > avermeerber...@gmail.com
> > > > > > > > >:
> > > > > > > >
> > > > > > > > > Hello Stig,
> > > > > > > > >
> > > > > > > > > Than you very much for you quick answer & fix.
> > > > > > > > > Unfortunaly, I still have topologies failing to start but
> > with
> > > a
> > > > > > > > different
> > > > > > > > > exception this time:
> > > > > > > > >
> > > > > > > > > Exception in thread "main" java.lang.NoClassDefFoundError:
> > > > > > > > > org/apache/commons/lang/StringUtils
> > > > > > > > > at
> > > > > > > > > org.apache.storm.kafka.spout.NamedTopicFilter.
> > getTopicsString(
> > > > > > > > > NamedTopicFilter.java:66)
> > > > > > > > > at
> > > > > > > > > org.apache.storm.kafka.spout.ManualPartitionSubscription.
> > > > > > > getTopicsString(
> > > > > > > > > ManualPartitionSubscription.java:70)
> > > > > > > > > at
> > > > > > > > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(
> > > > > > > > > BasicKafkaSpout.java:59)
> > > > > > > > > at com.acme.storm.util.StormUtils.getKafkaSpout(
> > > > > > > > > StormUtils.java:888)
> > > > > > > > > at
> > > > > > > > > com.acme.storm.evaluator.SLAEventsInterceptorTopology.
> main(
> > > > > > > > > SLAEventsInterceptorTopology.java:120)
> > > > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > > > org.apache.commons.lang.StringUtils
> > > > > > > > > at java.net.URLClassLoader.
> > > > findClass(URLClassLoader.java:
> > > > > > 381)
> > > > > > > > > at java.lang.ClassLoader.
> loadClass(ClassLoader.java:
> > > 424)
> > > > > > > > > at sun.misc.Launcher$AppClassLoader.loadClass(
> > > > > > > Launcher.java:338)
> > > > > > > > > at java.lang.ClassLoader.
> loadClass(ClassLoader.java:
> > > 357)
> > > > > > > > >
> > > > > > > > > Looks like you introduced a dependency on commons lang that
> > > > wasn't
> > > > > > part
> > > > > > > > of
> > > > > > > > > Storm, isn't it ?
> > > > > > > > >
> > > > > > > > > Keep faith, next time will be better :)
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Alexandre
> > > > > > > > >
> > > > > > > > > 2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <
> > > > > > stigdoess...@gmail.com
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > I've put up a fix here https://github.com/apache/
> > > > storm/pull/2426
> > > > > .
> > > > > > > > > There's
> > > > > > > > > > an updated storm-kafka-client jar at
> > > > > > > > > > https://drive.google.com/file/d/
> > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > > v9fd/view?usp=sharing
> > > > > > > > > > if you'd like to try it out.
> > > > > > > > > >
> > > > > > > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <
> > > > > > > stigdoess...@gmail.com
> > > > > > > > >:
> > > > > > > > > >
> > > > > > > > > > > Oops. The exception is not intentional, it's a bug. In
> > > 1.2.0
> > > > we
> > > > > > > check
> > > > > > > > > for
> > > > > > > > > > > the "enable.auto.commit" key in the kafkaConsumerProp
> > map,
> > > > and
> > > > > if
> > > > > > > it
> > > > > > > > is
> > > > > > > > > > set
> > > > > > > > > > > we warn in the log that it shouldn't be, because users
> > > should
> > > > > use
> > > > > > > the
> > > > > > > > > > > KafkaSpoutConfig.Builder.setProcessingGuarantee method
> > > > > instead.
> > > > > > > When
> > > > > > > > > the
> > > > > > > > > > > property is set we try to set the processing guarantee
> to
> > > > match
> > > > > > the
> > > > > > > > > > > pre-1.2.0 behavior, but I made a mistake and assumed
> the
> > > > > property
> > > > > > > key
> > > > > > > > > is
> > > > > > > > > > a
> > > > > > > > > > > boolean when it might actually be either a boolean or a
> > > > string.
> > > > > > > I'll
> > > > > > > > > put
> > > > > > > > > > up
> > > > > > > > > > > a fix ASAP.
> > > > > > > > > > >
> > > > > > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00
> Alexandre
> > > > > > > Vermeerbergen
> > > > > > > > > <
> > > > > > > > > > > avermeerber...@gmail.com>:
> > > > > > > > > > >
> > > > > > > > > > >> Hello Stig,
> > > > > > > > > > >>
> > > > > > > > > > >> Here's my first feedback on this Storm 1.2.0 preview
> on
> > my
> > > > > > > > Supervision
> > > > > > > > > > >> system based on Storm : I have 11 topologies KO (not
> > even
> > > > able
> > > > > > to
> > > > > > > > > > start),
> > > > > > > > > > >> and 4 topologies which seem to be unaffected.
> > > > > > > > > > >>
> > > > > > > > > > >> Details:
> > > > > > > > > > >> - I used the binaries posted on dropbox by Jungteak
> and
> > > your
> > > > > > > binary
> > > > > > > > > for
> > > > > > > > > > >> storm-kafka-client
> > > > > > > > > > >> - Rebuilt all our topologies using these
> storm-core.jar
> > &
> > > > > > > > > > >> storm-kafka-client.jar from these storm-1.2.0-snapshot
> > > (and
> > > > I
> > > > > > got
> > > > > > > > > > >> absolutely no error at build time).
> > > > > > > > > > >> Meaning that I have new "big jar" files including
> this
> > > > newer
> > > > > > > > > > >> storm-kafka-client.jar (except for 1 topology which
> > > depends
> > > > on
> > > > > > > > another
> > > > > > > > > > >> build system)
> > > > > > > > > > >> - The 11 topologies which fail to start show up the
> > > > following
> > > > > > > trace
> > > > > > > > > when
> > > > > > > > > > >> they are submitted:
> > > > > > > > > > >>
> > > > > > > > > > >> Running: java -client -Ddaemon.name= -Dstorm.options=
> > > > > > > > > > >> -Dstorm.home=/usr/local/Storm/storm-stable
> > > > > > > > > > >> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> > > > > > > > > > >> -Djava.library.path=/usr/
> local/lib:/opt/local/lib:/usr/
> > > lib
> > > > > > > > > > >> -Dstorm.conf.file= -cp
> > > > > > > > > > >> /usr/local/Storm/storm-stable/
> > > > *:/usr/local/Storm/storm-stabl
> > > > > > > > > > >> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/
> > > local/
> > > > > > > > > > >> Storm/StormTopologiesBJ.jar:/
> > > usr/local/Storm/storm-stable/
> > > > > > > > > > >> conf:/usr/local/Storm/storm-stable/bin
> > > > > > > > > > >> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> > > > > > > > > > >> -Dstorm.dependency.jars=
> > > > > > > > > > >> -Dstorm.dependency.artifacts={}
> > > > > > > > > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology
> > > > > > > > > > >> slaEventsInterceptor
> > > > > > > > > > >> Latest
> > > > > > > > > > >> 1022 [main] INFO c.a.s.e.
> SLAEventsInterceptorTopology
> > -
> > > > > > > Topology :
> > > > > > > > > > >> 'slaEventsInterceptor', KafkaOffsetTimeStrategy =
> > 'Latest'
> > > > > > > > > > >> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read
> > > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > > kafkaConsumer.properties
> > > > > > > > > > file:
> > > > > > > > > > >> /usr/local/Storm/flux/slaEventsInterceptor_
> > > > > > > kafkaConsumer.properties
> > > > > > > > > (No
> > > > > > > > > > >> such file or directory)
> > > > > > > > > > >> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not
> > set
> > > > > > > > > > >> enable.auto.commit manually. Instead use
> > > > > > > > > > >> KafkaSpoutConfig.Builder.setProcessingGuarantee. This
> > > will
> > > > be
> > > > > > > > treated
> > > > > > > > > > as
> > > > > > > > > > >> an
> > > > > > > > > > >> error in the next major release. For now the spout
> will
> > be
> > > > > > > > configured
> > > > > > > > > to
> > > > > > > > > > >> behave like it would have in pre-1.2.0 releases.
> > > > > > > > > > >> Exception in thread "main"
> java.lang.ClassCastException:
> > > > > > > > > > java.lang.String
> > > > > > > > > > >> cannot be cast to java.lang.Boolean
> > > > > > > > > > >> at
> > > > > > > > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.
> > > > setAutoCommitM
> > > > > > > > > > >> ode(KafkaSpoutConfig.java:721)
> > > > > > > > > > >> at
> > > > > > > > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(
> > > > KafkaSp
> > > > > > > > > > >> outConfig.java:97)
> > > > > > > > > > >> at
> > > > > > > > > > >> org.apache.storm.kafka.spout.
> KafkaSpoutConfig$Builder.
> > > > build(
> > > > > > > > > > >> KafkaSpoutConfig.java:671)
> > > > > > > > > > >> at
> > > > > > > > > > >> com.acme.storm.evaluator.
> SLAEventsInterceptorTopology.
> > > > main(S
> > > > > > > > > > >> LAEventsInterceptorTopology.java:118)
> > > > > > > > > > >>
> > > > > > > > > > >> The line part of our application which triggered above
> > > stack
> > > > > > trace
> > > > > > > > is
> > > > > > > > > > the
> > > > > > > > > > >> following one:
> > > > > > > > > > >>
> > > > > > > > > > >> KafkaSpoutConfig<String, Event>
> > > > > spoutConfigForMainTopic
> > > > > > =
> > > > > > > > > > >> KafkaSpoutConfig
> > > > > > > > > > >> .builder(elasticKafkaBrokers,
> > > > > KafkaTopics.MAIN)
> > > > > > > > > > >> .setValue(
> EventKafkaDeserializer.class)
> > > > > > > > > > >> .setGroupId(consumerId + "_" +
> > > > > KafkaTopics.MAIN)
> > > > > > > > > > >> .setFirstPollOffsetStrategy(strategy)
> > > > > > > > > > >> .setProp(kafkaConsumerProp)
> > > > > > > > > > >> .setRecordTranslator(
> > > > > > > > > > >> new
> > SLAEventsInterceptorKafkaRecor
> > > > > > > > > > dTranslator())
> > > > > > > > > > >> .build();
> > > > > > > > > > >>
> > > > > > > > > > >> I understand that there's a WARN message, but if it's
> a
> > > > > warning,
> > > > > > > > then
> > > > > > > > > > why
> > > > > > > > > > >> do we exit with such a brutal exception?
> > > > > > > > > > >>
> > > > > > > > > > >> Best regards,
> > > > > > > > > > >> Alexandre Vermeerbergen
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <
> > > > > > > > stigdoess...@gmail.com
> > > > > > > > > >:
> > > > > > > > > > >>
> > > > > > > > > > >> > Alexandre,
> > > > > > > > > > >> >
> > > > > > > > > > >> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client
> > jar
> > > > here
> > > > > > > > > > >> > https://drive.google.com/file/d/
> > > > > 1DgJWjhWwczYgZS82YGd63V3GT2G_
> > > > > > > > > > >> > v9fd/view?usp=sharing.
> > > > > > > > > > >> > You can probably keep the 1.1.0 versions of
> > storm-hbase
> > > > and
> > > > > > > > > > storm-hdfs,
> > > > > > > > > > >> > since the core Storm API hasn't changed.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If you want to build the jars yourself, clone the
> > > > repository
> > > > > > at
> > > > > > > > > > >> > https://github.com/apache/storm and check out the
> > > > > 1.x-branch
> > > > > > > > > branch.
> > > > > > > > > > >> You
> > > > > > > > > > >> > can build the entire project by running "mvn clean
> > > install
> > > > > > > > > > -DskipTests"
> > > > > > > > > > >> in
> > > > > > > > > > >> > the root. The individual module jars will be
> available
> > > in
> > > > > the
> > > > > > > > target
> > > > > > > > > > >> > directories for each module, e.g.
> > storm-kafka-client.jar
> > > > > will
> > > > > > be
> > > > > > > > in
> > > > > > > > > > >> > external/storm-kafka-client/target.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If you need to build the regular Storm distribution
> > (the
> > > > > > tarball
> > > > > > > > > > >> Jungtaek
> > > > > > > > > > >> > sent you), you can go to storm-dist/binary and run
> > "mvn
> > > > > clean
> > > > > > > > > install
> > > > > > > > > > >> > -Dgpg.skip". The tar/zip will be in
> > > > > storm-dist/binary/target.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Further reference in case you need it
> > > > > > > > > > >> > https://github.com/apache/
> storm/blob/master/DEVELOPER
> > .
> > > > > > > > > > >> > md#build-the-code-and-run-the-tests
> > > > > > > > > > >> > .
> > > > > > > > > > >> >
> > > > > > > > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen <
> > > > > > > > > > >> > avermeerber...@gmail.com
> > > > > > > > > > >> > >:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Hello Jungtaek,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Thanks for the link to 1.2.0 preview binaries.
> > > > > > > > > > >> > > However, since we heavily depend on Storm Kafka
> > > Client,
> > > > > > would
> > > > > > > > you
> > > > > > > > > > >> please
> > > > > > > > > > >> > > either add it or remind me how to build the jars
> of
> > > this
> > > > > > > > external
> > > > > > > > > > >> lib, at
> > > > > > > > > > >> > > same "snapshot" version as the rest ?
> > > > > > > > > > >> > > On a side note, some of our topologies also rely
> on
> > > > storm
> > > > > > > hbase
> > > > > > > > &
> > > > > > > > > > >> storm
> > > > > > > > > > >> > > hdfs : does it matters if these later ones stay in
> > > 1.1.0
> > > > > > > version
> > > > > > > > > for
> > > > > > > > > > >> this
> > > > > > > > > > >> > > test?
> > > > > > > > > > >> > > Indeed, I want to focus on Storm 1.2.0 "core" +
> > Storm
> > > > > Kafka
> > > > > > > > Client
> > > > > > > > > > >> 1.2.0
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Best regards,
> > > > > > > > > > >> > > Alexandre Vermeerbergen
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00
> > Jungtaek
> > > > > Lim <
> > > > > > > > > > >> kabh...@gmail.com>:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Alexandre,
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > https://www.dropbox.com/s/
> > > > mg2gnunk24oesyc/apache-storm-
> > > > > > > > > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Above link is custom binary distribution of
> > current
> > > > > > > 1.x-branch
> > > > > > > > > > >> > (SNAPSHOT
> > > > > > > > > > >> > > of
> > > > > > > > > > >> > > > 1.2.0). Could you run the test in your
> environment
> > > > first
> > > > > > and
> > > > > > > > let
> > > > > > > > > > us
> > > > > > > > > > >> > know
> > > > > > > > > > >> > > > about the result? Regardless of including
> metrics
> > V2
> > > > to
> > > > > > > 1.2.0
> > > > > > > > or
> > > > > > > > > > >> not,
> > > > > > > > > > >> > > your
> > > > > > > > > > >> > > > test report should be valuable for us.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > And please refer my analysis of current metrics:
> > > > > > > > > > >> > > > https://cwiki.apache.org/
> > confluence/display/STORM/
> > > > > > > > > > >> > > Limitations+of+current+
> > > > > > > > > > >> > > > metrics+feature
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > see why we want to move toward to Metrics V2.
> > > Current
> > > > PR
> > > > > > on
> > > > > > > > > > Metrics
> > > > > > > > > > >> V2
> > > > > > > > > > >> > is
> > > > > > > > > > >> > > > for initial state and it could cover only some
> of
> > > > issues
> > > > > > in
> > > > > > > > > list,
> > > > > > > > > > >> but I
> > > > > > > > > > >> > > > expect we will address another issues as well
> > based
> > > on
> > > > > the
> > > > > > > PR.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Thanks in advance!
> > > > > > > > > > >> > > > Jungtaek Lim (HeartSaVioR)
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre
> > Vermeerbergen <
> > > > > > > > > > >> > > > avermeerber...@gmail.com>님이
> > > > > > > > > > >> > > > 작성:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Hello Hugo,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > As I already posted, I was getting ready to
> test
> > > the
> > > > > > > > upcoming
> > > > > > > > > > RC,
> > > > > > > > > > >> so
> > > > > > > > > > >> > > > yeah :
> > > > > > > > > > >> > > > > as soon as I have some binaries for testing,
> I
> > > can
> > > > > run
> > > > > > > any
> > > > > > > > > > Storm
> > > > > > > > > > >> > 1.2.0
> > > > > > > > > > >> > > > > preview on my pre-production (well stressed)
> > > > > > environment.
> > > > > > > Is
> > > > > > > > > > there
> > > > > > > > > > >> > some
> > > > > > > > > > >> > > > URL
> > > > > > > > > > >> > > > > from where I can download such "snapshot"
> > > binaries?
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best regards,
> > > > > > > > > > >> > > > > Alexandre Vermeerbergen
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > PS: By the way, I have no interest in the
> > > > > > metrics-related
> > > > > > > > > > >> > enhancements
> > > > > > > > > > >> > > > you
> > > > > > > > > > >> > > > > mentionned - I can understand it is of
> interest
> > > for
> > > > > > people
> > > > > > > > > aware
> > > > > > > > > > >> of
> > > > > > > > > > >> > it
> > > > > > > > > > >> > > > and
> > > > > > > > > > >> > > > > who are expecting "something" related to it -
> > but
> > > I
> > > > > > would
> > > > > > > be
> > > > > > > > > sad
> > > > > > > > > > >> to
> > > > > > > > > > >> > > see a
> > > > > > > > > > >> > > > > postponing of the imminent Storm 1.2.0 release
> > for
> > > > > > > something
> > > > > > > > > we
> > > > > > > > > > do
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > use
> > > > > > > > > > >> > > > > at all.
> > > > > > > > > > >> > > > > PPS: I would love to learn more about these
> > > > "metrics",
> > > > > > > maybe
> > > > > > > > > > it's
> > > > > > > > > > >> > > > something
> > > > > > > > > > >> > > > > worth which never quite understood... any link
> > > with
> > > > > > clear
> > > > > > > > > > >> explanation
> > > > > > > > > > >> > > of
> > > > > > > > > > >> > > > > this feature?
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro
> <
> > > > > > > > > > >> > hlo...@hortonworks.com
> > > > > > > > > > >> > > >:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > I am also in agreement that we should not
> > delay
> > > > the
> > > > > > > 1.2.0
> > > > > > > > > > >> release
> > > > > > > > > > >> > for
> > > > > > > > > > >> > > > too
> > > > > > > > > > >> > > > > > long, but in order to release it a few
> things
> > > need
> > > > > to
> > > > > > be
> > > > > > > > > kept
> > > > > > > > > > in
> > > > > > > > > > >> > > mind:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > - If we want to avoid releasing 1.3.0 soon
> or
> > > at
> > > > > all,
> > > > > > > > that
> > > > > > > > > > >> means
> > > > > > > > > > >> > > that
> > > > > > > > > > >> > > > we
> > > > > > > > > > >> > > > > > should include in 1.2.0 as many important
> > > features
> > > > > > (e.g.
> > > > > > > > > > >> metricsV2)
> > > > > > > > > > >> > > as
> > > > > > > > > > >> > > > > > possible within a reasonable time frame.
> Also
> > to
> > > > > avoid
> > > > > > > as
> > > > > > > > > much
> > > > > > > > > > >> back
> > > > > > > > > > >> > > > > porting
> > > > > > > > > > >> > > > > > as possible one should really get all
> features
> > > > that
> > > > > we
> > > > > > > > > foresee
> > > > > > > > > > >> we
> > > > > > > > > > >> > > want
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > > maintain in 1.2.0 right away, and then
> simply
> > > > > maintain
> > > > > > > > them
> > > > > > > > > > >> there.
> > > > > > > > > > >> > > > > However
> > > > > > > > > > >> > > > > > it’s a fact of life that if a bug is found
> and
> > > it
> > > > > is a
> > > > > > > > > > blocker,
> > > > > > > > > > >> it
> > > > > > > > > > >> > > will
> > > > > > > > > > >> > > > > > have to be back ported.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > - STORM-2153 details that the new metrics
> > > > > requirements
> > > > > > > > were
> > > > > > > > > > >> driven
> > > > > > > > > > >> > by
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > users. Therefore if this feature is highly
> > > > important
> > > > > > and
> > > > > > > > > > sought
> > > > > > > > > > >> > > after,
> > > > > > > > > > >> > > > it
> > > > > > > > > > >> > > > > > may not make much sense to release 1.2.0
> > without
> > > > it.
> > > > > > If
> > > > > > > we
> > > > > > > > > do
> > > > > > > > > > >> > release
> > > > > > > > > > >> > > > > 1.2.0
> > > > > > > > > > >> > > > > > without the metrics, does it mean that
> metrics
> > > > will
> > > > > go
> > > > > > > > only
> > > > > > > > > in
> > > > > > > > > > >> > 2.0? I
> > > > > > > > > > >> > > > > would
> > > > > > > > > > >> > > > > > conjecture that most production deployments
> > will
> > > > > take
> > > > > > a
> > > > > > > > > while
> > > > > > > > > > to
> > > > > > > > > > >> > > > upgrade
> > > > > > > > > > >> > > > > to
> > > > > > > > > > >> > > > > > 2.0 even after it is released. That means
> that
> > > > they
> > > > > > will
> > > > > > > > > still
> > > > > > > > > > >> be
> > > > > > > > > > >> > > > running
> > > > > > > > > > >> > > > > > Storm without the benefit of the new
> metrics.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > - Several fixes have got in
> > storm-kafka-client.
> > > > > There
> > > > > > > > have
> > > > > > > > > > >> been a
> > > > > > > > > > >> > > lot
> > > > > > > > > > >> > > > of
> > > > > > > > > > >> > > > > > changes and I wonder to which degree they
> have
> > > > been
> > > > > > > system
> > > > > > > > > > >> tested
> > > > > > > > > > >> > in
> > > > > > > > > > >> > > > > > addition the existing unit tests
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > @Alexandre, since you are using
> > > storm-kafka-client
> > > > > and
> > > > > > > > have
> > > > > > > > > > >> filed
> > > > > > > > > > >> > > some
> > > > > > > > > > >> > > > > > bugs and driven some feature requests, I
> would
> > > > like
> > > > > to
> > > > > > > ask
> > > > > > > > > if
> > > > > > > > > > >> you
> > > > > > > > > > >> > > could
> > > > > > > > > > >> > > > > > help us within what is reasonably possible
> for
> > > you
> > > > > > with
> > > > > > > > the
> > > > > > > > > > >> > > following:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > - system test the latest storm-kafka-client
> > > > changes
> > > > > > in
> > > > > > > > your
> > > > > > > > > > >> > > > > > test/pre-production environment
> > > > > > > > > > >> > > > > > - provide some information on about your
> > setup,
> > > > > > > > > > >> > > > > > - how you are using storm-kafka-client
> > > > > > > > > > >> > > > > > - how it is performing
> > > > > > > > > > >> > > > > > - Kafka brokers, number of
> > > > topics/partitions,
> > > > > > > storm
> > > > > > > > > > >> > > parallelism,
> > > > > > > > > > >> > > > > and
> > > > > > > > > > >> > > > > > some info about your network
> > > > > > > > > > >> > > > > > - if it is reasonable to do so share some of
> > > your
> > > > > > tests
> > > > > > > > such
> > > > > > > > > > >> that
> > > > > > > > > > >> > we
> > > > > > > > > > >> > > > can
> > > > > > > > > > >> > > > > > also test it at our end.
> > > > > > > > > > >> > > > > > - Tell us specific things that you would
> like
> > us
> > > > to
> > > > > > test
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Thanks,
> > > > > > > > > > >> > > > > > Hugo
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde
> > > > Døssing <
> > > > > > > > > > >> > > > > stigdoess...@gmail.com>
> > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > I agree with Jungtaek, if metrics v2 can
> go
> > in
> > > > > very
> > > > > > > soon
> > > > > > > > > it
> > > > > > > > > > >> > should
> > > > > > > > > > >> > > go
> > > > > > > > > > >> > > > > in,
> > > > > > > > > > >> > > > > > > otherwise I'd rather release 1.2.0 now and
> > > work
> > > > on
> > > > > > > > getting
> > > > > > > > > > >> 2.0.0
> > > > > > > > > > >> > > > ready
> > > > > > > > > > >> > > > > > for
> > > > > > > > > > >> > > > > > > release.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 1.x and master have drifted pretty far
> > apart,
> > > > and
> > > > > > it's
> > > > > > > > > > >> causing a
> > > > > > > > > > >> > > lot
> > > > > > > > > > >> > > > of
> > > > > > > > > > >> > > > > > > porting work at this point. 1.x still has
> a
> > > lot
> > > > of
> > > > > > > > Clojure
> > > > > > > > > > >> code,
> > > > > > > > > > >> > > > > > storm-core
> > > > > > > > > > >> > > > > > > hasn't been split yet in that branch and
> > it's
> > > > also
> > > > > > > > > targeting
> > > > > > > > > > >> JDK
> > > > > > > > > > >> > > 1.7.
> > > > > > > > > > >> > > > > > It's
> > > > > > > > > > >> > > > > > > seems rare at this point that a PR cherry
> > > picks
> > > > > > > cleanly
> > > > > > > > > onto
> > > > > > > > > > >> 1.x
> > > > > > > > > > >> > > from
> > > > > > > > > > >> > > > > > > master.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > I've linked the list of issues that are
> only
> > > > fixed
> > > > > > in
> > > > > > > > > 2.0.0
> > > > > > > > > > >> and
> > > > > > > > > > >> > not
> > > > > > > > > > >> > > > > 1.x,
> > > > > > > > > > >> > > > > > to
> > > > > > > > > > >> > > > > > > illustrate how far ahead 2.0.0 is.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > https://pste.eu/p/9CJT.html
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer <
> > > > > > > > > ai...@hortonworks.com
> > > > > > > > > > >:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > >> Hi Taylor,
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >> Is it https://github.com/apache/
> > > > storm/pull/2203
> > > > > ?
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >> I think it would be great to get this in
> > 1.2
> > > > > > release.
> > > > > > > > Can
> > > > > > > > > > we
> > > > > > > > > > >> try
> > > > > > > > > > >> > > to
> > > > > > > > > > >> > > > > > >> address the issues in a week or so and
> get
> > > this
> > > > > in?
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >> Thanks,
> > > > > > > > > > >> > > > > > >> Arun
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" <
> > > > > > > > > ptgo...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > > >>> The original idea for the 1.2 release
> was
> > > that
> > > > > > would
> > > > > > > > > > >> introduce
> > > > > > > > > > >> > > the
> > > > > > > > > > >> > > > > > >> metrics v2 work (and there is additional
> > work
> > > > > > > required
> > > > > > > > > > there,
> > > > > > > > > > >> > so I
> > > > > > > > > > >> > > > > > >> understand a desire not to delay the
> > > release).
> > > > Do
> > > > > > we
> > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > >> > stick
> > > > > > > > > > >> > > > > with
> > > > > > > > > > >> > > > > > >> that, or deviate? If the latter, would we
> > do
> > > a
> > > > > 1.3
> > > > > > > > > release
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > > metrics?
> > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > >> > > > > > >>> As far as a 1.1.2 release I’m fine with
> > > > > releasing
> > > > > > > that
> > > > > > > > > at
> > > > > > > > > > >> any
> > > > > > > > > > >> > > time.
> > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > >> > > > > > >>> -Taylor
> > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek
> > Lim <
> > > > > > > > > > >> kabh...@gmail.com>
> > > > > > > > > > >> > > > > wrote:
> > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > >> > > > > > >>>> I think we could start release phase
> for
> > > both
> > > > > > 1.1.2
> > > > > > > > and
> > > > > > > > > > >> 1.2.0
> > > > > > > > > > >> > > when
> > > > > > > > > > >> > > > > > >>>> https://github.com/apache/
> > storm/pull/2423
> > > > will
> > > > > > be
> > > > > > > > > > merged.
> > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > >> > > > > > >>>> Thanks,
> > > > > > > > > > >> > > > > > >>>> Jungtaek Lim (HeartSaVioR)
> > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre
> > > > > > Vermeerbergen
> > > > > > > <
> > > > > > > > > > >> > > > > > >> avermeerber...@gmail.com>님이
> > > > > > > > > > >> > > > > > >>>> 작성:
> > > > > > > > > > >> > > > > > >>>>
> > > > > > > > > > >> > > > > > >>>>> Hello,
> > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > >> > > > > > >>>>> I'd love to see a Storm 1.2.0
> released:
> > > > it's a
> > > > > > > > perfect
> > > > > > > > > > >> > schedule
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > > > me
> > > > > > > > > > >> > > > > > >> for
> > > > > > > > > > >> > > > > > >>>>> test any Release Candidate that might
> be
> > > > > > available
> > > > > > > > if
> > > > > > > > > it
> > > > > > > > > > >> > > happens
> > > > > > > > > > >> > > > > > soon.
> > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > >> > > > > > >>>>> Best regards,
> > > > > > > > > > >> > > > > > >>>>> Alexandre
> > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun
> Mahadevan
> > > <
> > > > > > > > > > >> ar...@apache.org>:
> > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > >> > > > > > >>>>>> Hi,
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>> Looks like we are only waiting on
> > > > > > > > > > >> > https://issues.apache.org/
> > > > > > > > > > >> > > > > > >>>>>> jira/browse/STORM-2546 .
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>> Are there any other issues which are
> > > > blockers
> > > > > > for
> > > > > > > > > Storm
> > > > > > > > > > >> > 1.2.0?
> > > > > > > > > > >> > > > > Would
> > > > > > > > > > >> > > > > > >> be
> > > > > > > > > > >> > > > > > >>>>>> great to see the 1.2.0 release out
> soon
> > > as
> > > > it
> > > > > > > has a
> > > > > > > > > lot
> > > > > > > > > > >> of
> > > > > > > > > > >> > > > > critical
> > > > > > > > > > >> > > > > > >>>>> fixes.
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>> Thanks,
> > > > > > > > > > >> > > > > > >>>>>> Arun
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>>
> > > > > > > > > > >> > > > > > >>>>>
> > > > > > > > > > >> > > > > > >>>
> > > > > > > > > > >> > > > > > >>
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
