Hello Stig,
Than you very much for you quick answer & fix.
Unfortunaly, I still have topologies failing to start but with a different
exception this time:
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/lang/StringUtils
at
org.apache.storm.kafka.spout.NamedTopicFilter.getTopicsString(NamedTopicFilter.java:66)
at
org.apache.storm.kafka.spout.ManualPartitionSubscription.getTopicsString(ManualPartitionSubscription.java:70)
at
com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>(BasicKafkaSpout.java:59)
at com.acme.storm.util.StormUtils.getKafkaSpout(StormUtils.java:888)
at
com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(SLAEventsInterceptorTopology.java:120)
Caused by: java.lang.ClassNotFoundException:
org.apache.commons.lang.StringUtils
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Looks like you introduced a dependency on commons lang that wasn't part of
Storm, isn't it ?
Keep faith, next time will be better :)
Best regards,
Alexandre
2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> I've put up a fix here https://github.com/apache/storm/pull/2426. There's
> an updated storm-kafka-client jar at
> https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_
> v9fd/view?usp=sharing
> if you'd like to try it out.
>
> 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
>
> > Oops. The exception is not intentional, it's a bug. In 1.2.0 we check for
> > the "enable.auto.commit" key in the kafkaConsumerProp map, and if it is
> set
> > we warn in the log that it shouldn't be, because users should use the
> > KafkaSpoutConfig.Builder.setProcessingGuarantee method instead. When the
> > property is set we try to set the processing guarantee to match the
> > pre-1.2.0 behavior, but I made a mistake and assumed the property key is
> a
> > boolean when it might actually be either a boolean or a string. I'll put
> up
> > a fix ASAP.
> >
> > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00 Alexandre Vermeerbergen <
> > avermeerber...@gmail.com>:
> >
> >> Hello Stig,
> >>
> >> Here's my first feedback on this Storm 1.2.0 preview on my Supervision
> >> system based on Storm : I have 11 topologies KO (not even able to
> start),
> >> and 4 topologies which seem to be unaffected.
> >>
> >> Details:
> >> - I used the binaries posted on dropbox by Jungteak and your binary for
> >> storm-kafka-client
> >> - Rebuilt all our topologies using these storm-core.jar &
> >> storm-kafka-client.jar from these storm-1.2.0-snapshot (and I got
> >> absolutely no error at build time).
> >> Meaning that I have new "big jar" files including this newer
> >> storm-kafka-client.jar (except for 1 topology which depends on another
> >> build system)
> >> - The 11 topologies which fail to start show up the following trace when
> >> they are submitted:
> >>
> >> Running: java -client -Ddaemon.name= -Dstorm.options=
> >> -Dstorm.home=/usr/local/Storm/storm-stable
> >> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
> >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
> >> -Dstorm.conf.file= -cp
> >> /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-stabl
> >> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/local/
> >> Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-stable/
> >> conf:/usr/local/Storm/storm-stable/bin
> >> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
> >> -Dstorm.dependency.jars=
> >> -Dstorm.dependency.artifacts={}
> >> com.acme.storm.evaluator.SLAEventsInterceptorTopology
> >> slaEventsInterceptor
> >> Latest
> >> 1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology - Topology :
> >> 'slaEventsInterceptor', KafkaOffsetTimeStrategy = 'Latest'
> >> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read
> >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties
> file:
> >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties (No
> >> such file or directory)
> >> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
> >> enable.auto.commit manually. Instead use
> >> KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated
> as
> >> an
> >> error in the next major release. For now the spout will be configured to
> >> behave like it would have in pre-1.2.0 releases.
> >> Exception in thread "main" java.lang.ClassCastException:
> java.lang.String
> >> cannot be cast to java.lang.Boolean
> >> at
> >> org.apache.storm.kafka.spout.KafkaSpoutConfig.setAutoCommitM
> >> ode(KafkaSpoutConfig.java:721)
> >> at
> >> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(KafkaSp
> >> outConfig.java:97)
> >> at
> >> org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.build(
> >> KafkaSpoutConfig.java:671)
> >> at
> >> com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(S
> >> LAEventsInterceptorTopology.java:118)
> >>
> >> The line part of our application which triggered above stack trace is
> the
> >> following one:
> >>
> >> KafkaSpoutConfig<String, Event> spoutConfigForMainTopic =
> >> KafkaSpoutConfig
> >> .builder(elasticKafkaBrokers, KafkaTopics.MAIN)
> >> .setValue(EventKafkaDeserializer.class)
> >> .setGroupId(consumerId + "_" + KafkaTopics.MAIN)
> >> .setFirstPollOffsetStrategy(strategy)
> >> .setProp(kafkaConsumerProp)
> >> .setRecordTranslator(
> >> new SLAEventsInterceptorKafkaRecor
> dTranslator())
> >> .build();
> >>
> >> I understand that there's a WARN message, but if it's a warning, then
> why
> >> do we exit with such a brutal exception?
> >>
> >> Best regards,
> >> Alexandre Vermeerbergen
> >>
> >>
> >>
> >>
> >> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> >>
> >> > Alexandre,
> >> >
> >> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client jar here
> >> > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_
> >> > v9fd/view?usp=sharing.
> >> > You can probably keep the 1.1.0 versions of storm-hbase and
> storm-hdfs,
> >> > since the core Storm API hasn't changed.
> >> >
> >> > If you want to build the jars yourself, clone the repository at
> >> > https://github.com/apache/storm and check out the 1.x-branch branch.
> >> You
> >> > can build the entire project by running "mvn clean install
> -DskipTests"
> >> in
> >> > the root. The individual module jars will be available in the target
> >> > directories for each module, e.g. storm-kafka-client.jar will be in
> >> > external/storm-kafka-client/target.
> >> >
> >> > If you need to build the regular Storm distribution (the tarball
> >> Jungtaek
> >> > sent you), you can go to storm-dist/binary and run "mvn clean install
> >> > -Dgpg.skip". The tar/zip will be in storm-dist/binary/target.
> >> >
> >> > Further reference in case you need it
> >> > https://github.com/apache/storm/blob/master/DEVELOPER.
> >> > md#build-the-code-and-run-the-tests
> >> > .
> >> >
> >> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen <
> >> > avermeerber...@gmail.com
> >> > >:
> >> >
> >> > > Hello Jungtaek,
> >> > >
> >> > > Thanks for the link to 1.2.0 preview binaries.
> >> > > However, since we heavily depend on Storm Kafka Client, would you
> >> please
> >> > > either add it or remind me how to build the jars of this external
> >> lib, at
> >> > > same "snapshot" version as the rest ?
> >> > > On a side note, some of our topologies also rely on storm hbase &
> >> storm
> >> > > hdfs : does it matters if these later ones stay in 1.1.0 version for
> >> this
> >> > > test?
> >> > > Indeed, I want to focus on Storm 1.2.0 "core" + Storm Kafka Client
> >> 1.2.0
> >> > >
> >> > > Best regards,
> >> > > Alexandre Vermeerbergen
> >> > >
> >> > >
> >> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00 Jungtaek Lim <
> >> kabh...@gmail.com>:
> >> > >
> >> > > > Alexandre,
> >> > > >
> >> > > > https://www.dropbox.com/s/mg2gnunk24oesyc/apache-storm-
> >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0
> >> > > >
> >> > > > Above link is custom binary distribution of current 1.x-branch
> >> > (SNAPSHOT
> >> > > of
> >> > > > 1.2.0). Could you run the test in your environment first and let
> us
> >> > know
> >> > > > about the result? Regardless of including metrics V2 to 1.2.0 or
> >> not,
> >> > > your
> >> > > > test report should be valuable for us.
> >> > > >
> >> > > > And please refer my analysis of current metrics:
> >> > > > https://cwiki.apache.org/confluence/display/STORM/
> >> > > Limitations+of+current+
> >> > > > metrics+feature
> >> > > > to
> >> > > > see why we want to move toward to Metrics V2. Current PR on
> Metrics
> >> V2
> >> > is
> >> > > > for initial state and it could cover only some of issues in list,
> >> but I
> >> > > > expect we will address another issues as well based on the PR.
> >> > > >
> >> > > > Thanks in advance!
> >> > > > Jungtaek Lim (HeartSaVioR)
> >> > > >
> >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre Vermeerbergen <
> >> > > > avermeerber...@gmail.com>님이
> >> > > > 작성:
> >> > > >
> >> > > > > Hello Hugo,
> >> > > > >
> >> > > > > As I already posted, I was getting ready to test the upcoming
> RC,
> >> so
> >> > > > yeah :
> >> > > > > as soon as I have some binaries for testing, I can run any
> Storm
> >> > 1.2.0
> >> > > > > preview on my pre-production (well stressed) environment. Is
> there
> >> > some
> >> > > > URL
> >> > > > > from where I can download such "snapshot" binaries?
> >> > > > >
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Alexandre Vermeerbergen
> >> > > > >
> >> > > > > PS: By the way, I have no interest in the metrics-related
> >> > enhancements
> >> > > > you
> >> > > > > mentionned - I can understand it is of interest for people aware
> >> of
> >> > it
> >> > > > and
> >> > > > > who are expecting "something" related to it - but I would be sad
> >> to
> >> > > see a
> >> > > > > postponing of the imminent Storm 1.2.0 release for something we
> do
> >> > not
> >> > > > use
> >> > > > > at all.
> >> > > > > PPS: I would love to learn more about these "metrics", maybe
> it's
> >> > > > something
> >> > > > > worth which never quite understood... any link with clear
> >> explanation
> >> > > of
> >> > > > > this feature?
> >> > > > >
> >> > > > >
> >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro <
> >> > hlo...@hortonworks.com
> >> > > >:
> >> > > > >
> >> > > > > > I am also in agreement that we should not delay the 1.2.0
> >> release
> >> > for
> >> > > > too
> >> > > > > > long, but in order to release it a few things need to be kept
> in
> >> > > mind:
> >> > > > > >
> >> > > > > > - If we want to avoid releasing 1.3.0 soon or at all, that
> >> means
> >> > > that
> >> > > > we
> >> > > > > > should include in 1.2.0 as many important features (e.g.
> >> metricsV2)
> >> > > as
> >> > > > > > possible within a reasonable time frame. Also to avoid as much
> >> back
> >> > > > > porting
> >> > > > > > as possible one should really get all features that we foresee
> >> we
> >> > > want
> >> > > > to
> >> > > > > > maintain in 1.2.0 right away, and then simply maintain them
> >> there.
> >> > > > > However
> >> > > > > > it’s a fact of life that if a bug is found and it is a
> blocker,
> >> it
> >> > > will
> >> > > > > > have to be back ported.
> >> > > > > >
> >> > > > > > - STORM-2153 details that the new metrics requirements were
> >> driven
> >> > by
> >> > > > the
> >> > > > > > users. Therefore if this feature is highly important and
> sought
> >> > > after,
> >> > > > it
> >> > > > > > may not make much sense to release 1.2.0 without it. If we do
> >> > release
> >> > > > > 1.2.0
> >> > > > > > without the metrics, does it mean that metrics will go only in
> >> > 2.0? I
> >> > > > > would
> >> > > > > > conjecture that most production deployments will take a while
> to
> >> > > > upgrade
> >> > > > > to
> >> > > > > > 2.0 even after it is released. That means that they will still
> >> be
> >> > > > running
> >> > > > > > Storm without the benefit of the new metrics.
> >> > > > > >
> >> > > > > > - Several fixes have got in storm-kafka-client. There have
> >> been a
> >> > > lot
> >> > > > of
> >> > > > > > changes and I wonder to which degree they have been system
> >> tested
> >> > in
> >> > > > > > addition the existing unit tests
> >> > > > > >
> >> > > > > > @Alexandre, since you are using storm-kafka-client and have
> >> filed
> >> > > some
> >> > > > > > bugs and driven some feature requests, I would like to ask if
> >> you
> >> > > could
> >> > > > > > help us within what is reasonably possible for you with the
> >> > > following:
> >> > > > > >
> >> > > > > > - system test the latest storm-kafka-client changes in your
> >> > > > > > test/pre-production environment
> >> > > > > > - provide some information on about your setup,
> >> > > > > > - how you are using storm-kafka-client
> >> > > > > > - how it is performing
> >> > > > > > - Kafka brokers, number of topics/partitions, storm
> >> > > parallelism,
> >> > > > > and
> >> > > > > > some info about your network
> >> > > > > > - if it is reasonable to do so share some of your tests such
> >> that
> >> > we
> >> > > > can
> >> > > > > > also test it at our end.
> >> > > > > > - Tell us specific things that you would like us to test
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Hugo
> >> > > > > >
> >> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde Døssing <
> >> > > > > stigdoess...@gmail.com>
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > I agree with Jungtaek, if metrics v2 can go in very soon it
> >> > should
> >> > > go
> >> > > > > in,
> >> > > > > > > otherwise I'd rather release 1.2.0 now and work on getting
> >> 2.0.0
> >> > > > ready
> >> > > > > > for
> >> > > > > > > release.
> >> > > > > > >
> >> > > > > > > 1.x and master have drifted pretty far apart, and it's
> >> causing a
> >> > > lot
> >> > > > of
> >> > > > > > > porting work at this point. 1.x still has a lot of Clojure
> >> code,
> >> > > > > > storm-core
> >> > > > > > > hasn't been split yet in that branch and it's also targeting
> >> JDK
> >> > > 1.7.
> >> > > > > > It's
> >> > > > > > > seems rare at this point that a PR cherry picks cleanly onto
> >> 1.x
> >> > > from
> >> > > > > > > master.
> >> > > > > > >
> >> > > > > > > I've linked the list of issues that are only fixed in 2.0.0
> >> and
> >> > not
> >> > > > > 1.x,
> >> > > > > > to
> >> > > > > > > illustrate how far ahead 2.0.0 is.
> >> > > > > > >
> >> > > > > > > https://pste.eu/p/9CJT.html
> >> > > > > > >
> >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer <ai...@hortonworks.com
> >:
> >> > > > > > >
> >> > > > > > >> Hi Taylor,
> >> > > > > > >>
> >> > > > > > >> Is it https://github.com/apache/storm/pull/2203 ?
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> I think it would be great to get this in 1.2 release. Can
> we
> >> try
> >> > > to
> >> > > > > > >> address the issues in a week or so and get this in?
> >> > > > > > >>
> >> > > > > > >> Thanks,
> >> > > > > > >> Arun
> >> > > > > > >>
> >> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" <ptgo...@gmail.com
> >
> >> > > wrote:
> >> > > > > > >>
> >> > > > > > >>> The original idea for the 1.2 release was that would
> >> introduce
> >> > > the
> >> > > > > > >> metrics v2 work (and there is additional work required
> there,
> >> > so I
> >> > > > > > >> understand a desire not to delay the release). Do we want
> to
> >> > stick
> >> > > > > with
> >> > > > > > >> that, or deviate? If the latter, would we do a 1.3 release
> >> for
> >> > > > > metrics?
> >> > > > > > >>>
> >> > > > > > >>> As far as a 1.1.2 release I’m fine with releasing that at
> >> any
> >> > > time.
> >> > > > > > >>>
> >> > > > > > >>> -Taylor
> >> > > > > > >>>
> >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek Lim <
> >> kabh...@gmail.com>
> >> > > > > wrote:
> >> > > > > > >>>>
> >> > > > > > >>>> I think we could start release phase for both 1.1.2 and
> >> 1.2.0
> >> > > when
> >> > > > > > >>>> https://github.com/apache/storm/pull/2423 will be
> merged.
> >> > > > > > >>>>
> >> > > > > > >>>> Thanks,
> >> > > > > > >>>> Jungtaek Lim (HeartSaVioR)
> >> > > > > > >>>>
> >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre Vermeerbergen <
> >> > > > > > >> avermeerber...@gmail.com>님이
> >> > > > > > >>>> 작성:
> >> > > > > > >>>>
> >> > > > > > >>>>> Hello,
> >> > > > > > >>>>>
> >> > > > > > >>>>> I'd love to see a Storm 1.2.0 released: it's a perfect
> >> > schedule
> >> > > > for
> >> > > > > > me
> >> > > > > > >> for
> >> > > > > > >>>>> test any Release Candidate that might be available if it
> >> > > happens
> >> > > > > > soon.
> >> > > > > > >>>>>
> >> > > > > > >>>>> Best regards,
> >> > > > > > >>>>> Alexandre
> >> > > > > > >>>>>
> >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan <
> >> ar...@apache.org>:
> >> > > > > > >>>>>
> >> > > > > > >>>>>> Hi,
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Looks like we are only waiting on
> >> > https://issues.apache.org/
> >> > > > > > >>>>>> jira/browse/STORM-2546 .
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Are there any other issues which are blockers for Storm
> >> > 1.2.0?
> >> > > > > Would
> >> > > > > > >> be
> >> > > > > > >>>>>> great to see the 1.2.0 release out soon as it has a lot
> >> of
> >> > > > > critical
> >> > > > > > >>>>> fixes.
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Thanks,
> >> > > > > > >>>>>> Arun
> >> > > > > > >>>>>>
> >> > > > > > >>>>>>
> >> > > > > > >>>>>>
> >> > > > > > >>>>>
> >> > > > > > >>>
> >> > > > > > >>
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
