Alexandre, Thanks for trying. I think this time it's not a problem with storm-kafka-client. It has a dependency on commons-lang 2.5, which is declared in the storm-kafka-client pom. We usually recommend that people use Maven/Gradle/Ivy or similar systems to ensure that they get all dependencies when they build their topologies. In such a case Maven (or similar) would see that your topology depends on storm-kafka-client, and also download and include commons-lang in the topology jar. When you put together dependencies manually you need to include commons-lang manually as well.
You can get the commons-lang 2.5 jar here http://central.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.jar 2017-11-19 11:49 GMT+01:00 Alexandre Vermeerbergen <avermeerber...@gmail.com >: > Hello Stig, > > Than you very much for you quick answer & fix. > Unfortunaly, I still have topologies failing to start but with a different > exception this time: > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/commons/lang/StringUtils > at > org.apache.storm.kafka.spout.NamedTopicFilter.getTopicsString( > NamedTopicFilter.java:66) > at > org.apache.storm.kafka.spout.ManualPartitionSubscription.getTopicsString( > ManualPartitionSubscription.java:70) > at > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>( > BasicKafkaSpout.java:59) > at com.acme.storm.util.StormUtils.getKafkaSpout( > StormUtils.java:888) > at > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main( > SLAEventsInterceptorTopology.java:120) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.lang.StringUtils > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > Looks like you introduced a dependency on commons lang that wasn't part of > Storm, isn't it ? > > Keep faith, next time will be better :) > > Best regards, > Alexandre > > 2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: > > > I've put up a fix here https://github.com/apache/storm/pull/2426. > There's > > an updated storm-kafka-client jar at > > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_ > > v9fd/view?usp=sharing > > if you'd like to try it out. > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: > > > > > Oops. The exception is not intentional, it's a bug. In 1.2.0 we check > for > > > the "enable.auto.commit" key in the kafkaConsumerProp map, and if it is > > set > > > we warn in the log that it shouldn't be, because users should use the > > > KafkaSpoutConfig.Builder.setProcessingGuarantee method instead. When > the > > > property is set we try to set the processing guarantee to match the > > > pre-1.2.0 behavior, but I made a mistake and assumed the property key > is > > a > > > boolean when it might actually be either a boolean or a string. I'll > put > > up > > > a fix ASAP. > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00 Alexandre Vermeerbergen > < > > > avermeerber...@gmail.com>: > > > > > >> Hello Stig, > > >> > > >> Here's my first feedback on this Storm 1.2.0 preview on my Supervision > > >> system based on Storm : I have 11 topologies KO (not even able to > > start), > > >> and 4 topologies which seem to be unaffected. > > >> > > >> Details: > > >> - I used the binaries posted on dropbox by Jungteak and your binary > for > > >> storm-kafka-client > > >> - Rebuilt all our topologies using these storm-core.jar & > > >> storm-kafka-client.jar from these storm-1.2.0-snapshot (and I got > > >> absolutely no error at build time). > > >> Meaning that I have new "big jar" files including this newer > > >> storm-kafka-client.jar (except for 1 topology which depends on another > > >> build system) > > >> - The 11 topologies which fail to start show up the following trace > when > > >> they are submitted: > > >> > > >> Running: java -client -Ddaemon.name= -Dstorm.options= > > >> -Dstorm.home=/usr/local/Storm/storm-stable > > >> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs > > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib > > >> -Dstorm.conf.file= -cp > > >> /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-stabl > > >> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/local/ > > >> Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-stable/ > > >> conf:/usr/local/Storm/storm-stable/bin > > >> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar > > >> -Dstorm.dependency.jars= > > >> -Dstorm.dependency.artifacts={} > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology > > >> slaEventsInterceptor > > >> Latest > > >> 1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology - Topology : > > >> 'slaEventsInterceptor', KafkaOffsetTimeStrategy = 'Latest' > > >> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read > > >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties > > file: > > >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties > (No > > >> such file or directory) > > >> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set > > >> enable.auto.commit manually. Instead use > > >> KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated > > as > > >> an > > >> error in the next major release. For now the spout will be configured > to > > >> behave like it would have in pre-1.2.0 releases. > > >> Exception in thread "main" java.lang.ClassCastException: > > java.lang.String > > >> cannot be cast to java.lang.Boolean > > >> at > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.setAutoCommitM > > >> ode(KafkaSpoutConfig.java:721) > > >> at > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(KafkaSp > > >> outConfig.java:97) > > >> at > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.build( > > >> KafkaSpoutConfig.java:671) > > >> at > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(S > > >> LAEventsInterceptorTopology.java:118) > > >> > > >> The line part of our application which triggered above stack trace is > > the > > >> following one: > > >> > > >> KafkaSpoutConfig<String, Event> spoutConfigForMainTopic = > > >> KafkaSpoutConfig > > >> .builder(elasticKafkaBrokers, KafkaTopics.MAIN) > > >> .setValue(EventKafkaDeserializer.class) > > >> .setGroupId(consumerId + "_" + KafkaTopics.MAIN) > > >> .setFirstPollOffsetStrategy(strategy) > > >> .setProp(kafkaConsumerProp) > > >> .setRecordTranslator( > > >> new SLAEventsInterceptorKafkaRecor > > dTranslator()) > > >> .build(); > > >> > > >> I understand that there's a WARN message, but if it's a warning, then > > why > > >> do we exit with such a brutal exception? > > >> > > >> Best regards, > > >> Alexandre Vermeerbergen > > >> > > >> > > >> > > >> > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com > >: > > >> > > >> > Alexandre, > > >> > > > >> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client jar here > > >> > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_ > > >> > v9fd/view?usp=sharing. > > >> > You can probably keep the 1.1.0 versions of storm-hbase and > > storm-hdfs, > > >> > since the core Storm API hasn't changed. > > >> > > > >> > If you want to build the jars yourself, clone the repository at > > >> > https://github.com/apache/storm and check out the 1.x-branch > branch. > > >> You > > >> > can build the entire project by running "mvn clean install > > -DskipTests" > > >> in > > >> > the root. The individual module jars will be available in the target > > >> > directories for each module, e.g. storm-kafka-client.jar will be in > > >> > external/storm-kafka-client/target. > > >> > > > >> > If you need to build the regular Storm distribution (the tarball > > >> Jungtaek > > >> > sent you), you can go to storm-dist/binary and run "mvn clean > install > > >> > -Dgpg.skip". The tar/zip will be in storm-dist/binary/target. > > >> > > > >> > Further reference in case you need it > > >> > https://github.com/apache/storm/blob/master/DEVELOPER. > > >> > md#build-the-code-and-run-the-tests > > >> > . > > >> > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen < > > >> > avermeerber...@gmail.com > > >> > >: > > >> > > > >> > > Hello Jungtaek, > > >> > > > > >> > > Thanks for the link to 1.2.0 preview binaries. > > >> > > However, since we heavily depend on Storm Kafka Client, would you > > >> please > > >> > > either add it or remind me how to build the jars of this external > > >> lib, at > > >> > > same "snapshot" version as the rest ? > > >> > > On a side note, some of our topologies also rely on storm hbase & > > >> storm > > >> > > hdfs : does it matters if these later ones stay in 1.1.0 version > for > > >> this > > >> > > test? > > >> > > Indeed, I want to focus on Storm 1.2.0 "core" + Storm Kafka Client > > >> 1.2.0 > > >> > > > > >> > > Best regards, > > >> > > Alexandre Vermeerbergen > > >> > > > > >> > > > > >> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00 Jungtaek Lim < > > >> kabh...@gmail.com>: > > >> > > > > >> > > > Alexandre, > > >> > > > > > >> > > > https://www.dropbox.com/s/mg2gnunk24oesyc/apache-storm- > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0 > > >> > > > > > >> > > > Above link is custom binary distribution of current 1.x-branch > > >> > (SNAPSHOT > > >> > > of > > >> > > > 1.2.0). Could you run the test in your environment first and let > > us > > >> > know > > >> > > > about the result? Regardless of including metrics V2 to 1.2.0 or > > >> not, > > >> > > your > > >> > > > test report should be valuable for us. > > >> > > > > > >> > > > And please refer my analysis of current metrics: > > >> > > > https://cwiki.apache.org/confluence/display/STORM/ > > >> > > Limitations+of+current+ > > >> > > > metrics+feature > > >> > > > to > > >> > > > see why we want to move toward to Metrics V2. Current PR on > > Metrics > > >> V2 > > >> > is > > >> > > > for initial state and it could cover only some of issues in > list, > > >> but I > > >> > > > expect we will address another issues as well based on the PR. > > >> > > > > > >> > > > Thanks in advance! > > >> > > > Jungtaek Lim (HeartSaVioR) > > >> > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre Vermeerbergen < > > >> > > > avermeerber...@gmail.com>님이 > > >> > > > 작성: > > >> > > > > > >> > > > > Hello Hugo, > > >> > > > > > > >> > > > > As I already posted, I was getting ready to test the upcoming > > RC, > > >> so > > >> > > > yeah : > > >> > > > > as soon as I have some binaries for testing, I can run any > > Storm > > >> > 1.2.0 > > >> > > > > preview on my pre-production (well stressed) environment. Is > > there > > >> > some > > >> > > > URL > > >> > > > > from where I can download such "snapshot" binaries? > > >> > > > > > > >> > > > > > > >> > > > > Best regards, > > >> > > > > Alexandre Vermeerbergen > > >> > > > > > > >> > > > > PS: By the way, I have no interest in the metrics-related > > >> > enhancements > > >> > > > you > > >> > > > > mentionned - I can understand it is of interest for people > aware > > >> of > > >> > it > > >> > > > and > > >> > > > > who are expecting "something" related to it - but I would be > sad > > >> to > > >> > > see a > > >> > > > > postponing of the imminent Storm 1.2.0 release for something > we > > do > > >> > not > > >> > > > use > > >> > > > > at all. > > >> > > > > PPS: I would love to learn more about these "metrics", maybe > > it's > > >> > > > something > > >> > > > > worth which never quite understood... any link with clear > > >> explanation > > >> > > of > > >> > > > > this feature? > > >> > > > > > > >> > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro < > > >> > hlo...@hortonworks.com > > >> > > >: > > >> > > > > > > >> > > > > > I am also in agreement that we should not delay the 1.2.0 > > >> release > > >> > for > > >> > > > too > > >> > > > > > long, but in order to release it a few things need to be > kept > > in > > >> > > mind: > > >> > > > > > > > >> > > > > > - If we want to avoid releasing 1.3.0 soon or at all, that > > >> means > > >> > > that > > >> > > > we > > >> > > > > > should include in 1.2.0 as many important features (e.g. > > >> metricsV2) > > >> > > as > > >> > > > > > possible within a reasonable time frame. Also to avoid as > much > > >> back > > >> > > > > porting > > >> > > > > > as possible one should really get all features that we > foresee > > >> we > > >> > > want > > >> > > > to > > >> > > > > > maintain in 1.2.0 right away, and then simply maintain them > > >> there. > > >> > > > > However > > >> > > > > > it’s a fact of life that if a bug is found and it is a > > blocker, > > >> it > > >> > > will > > >> > > > > > have to be back ported. > > >> > > > > > > > >> > > > > > - STORM-2153 details that the new metrics requirements were > > >> driven > > >> > by > > >> > > > the > > >> > > > > > users. Therefore if this feature is highly important and > > sought > > >> > > after, > > >> > > > it > > >> > > > > > may not make much sense to release 1.2.0 without it. If we > do > > >> > release > > >> > > > > 1.2.0 > > >> > > > > > without the metrics, does it mean that metrics will go only > in > > >> > 2.0? I > > >> > > > > would > > >> > > > > > conjecture that most production deployments will take a > while > > to > > >> > > > upgrade > > >> > > > > to > > >> > > > > > 2.0 even after it is released. That means that they will > still > > >> be > > >> > > > running > > >> > > > > > Storm without the benefit of the new metrics. > > >> > > > > > > > >> > > > > > - Several fixes have got in storm-kafka-client. There have > > >> been a > > >> > > lot > > >> > > > of > > >> > > > > > changes and I wonder to which degree they have been system > > >> tested > > >> > in > > >> > > > > > addition the existing unit tests > > >> > > > > > > > >> > > > > > @Alexandre, since you are using storm-kafka-client and have > > >> filed > > >> > > some > > >> > > > > > bugs and driven some feature requests, I would like to ask > if > > >> you > > >> > > could > > >> > > > > > help us within what is reasonably possible for you with the > > >> > > following: > > >> > > > > > > > >> > > > > > - system test the latest storm-kafka-client changes in your > > >> > > > > > test/pre-production environment > > >> > > > > > - provide some information on about your setup, > > >> > > > > > - how you are using storm-kafka-client > > >> > > > > > - how it is performing > > >> > > > > > - Kafka brokers, number of topics/partitions, storm > > >> > > parallelism, > > >> > > > > and > > >> > > > > > some info about your network > > >> > > > > > - if it is reasonable to do so share some of your tests such > > >> that > > >> > we > > >> > > > can > > >> > > > > > also test it at our end. > > >> > > > > > - Tell us specific things that you would like us to test > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Hugo > > >> > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde Døssing < > > >> > > > > stigdoess...@gmail.com> > > >> > > > > > wrote: > > >> > > > > > > > > >> > > > > > > I agree with Jungtaek, if metrics v2 can go in very soon > it > > >> > should > > >> > > go > > >> > > > > in, > > >> > > > > > > otherwise I'd rather release 1.2.0 now and work on getting > > >> 2.0.0 > > >> > > > ready > > >> > > > > > for > > >> > > > > > > release. > > >> > > > > > > > > >> > > > > > > 1.x and master have drifted pretty far apart, and it's > > >> causing a > > >> > > lot > > >> > > > of > > >> > > > > > > porting work at this point. 1.x still has a lot of Clojure > > >> code, > > >> > > > > > storm-core > > >> > > > > > > hasn't been split yet in that branch and it's also > targeting > > >> JDK > > >> > > 1.7. > > >> > > > > > It's > > >> > > > > > > seems rare at this point that a PR cherry picks cleanly > onto > > >> 1.x > > >> > > from > > >> > > > > > > master. > > >> > > > > > > > > >> > > > > > > I've linked the list of issues that are only fixed in > 2.0.0 > > >> and > > >> > not > > >> > > > > 1.x, > > >> > > > > > to > > >> > > > > > > illustrate how far ahead 2.0.0 is. > > >> > > > > > > > > >> > > > > > > https://pste.eu/p/9CJT.html > > >> > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer < > ai...@hortonworks.com > > >: > > >> > > > > > > > > >> > > > > > >> Hi Taylor, > > >> > > > > > >> > > >> > > > > > >> Is it https://github.com/apache/storm/pull/2203 ? > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> I think it would be great to get this in 1.2 release. Can > > we > > >> try > > >> > > to > > >> > > > > > >> address the issues in a week or so and get this in? > > >> > > > > > >> > > >> > > > > > >> Thanks, > > >> > > > > > >> Arun > > >> > > > > > >> > > >> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" < > ptgo...@gmail.com > > > > > >> > > wrote: > > >> > > > > > >> > > >> > > > > > >>> The original idea for the 1.2 release was that would > > >> introduce > > >> > > the > > >> > > > > > >> metrics v2 work (and there is additional work required > > there, > > >> > so I > > >> > > > > > >> understand a desire not to delay the release). Do we want > > to > > >> > stick > > >> > > > > with > > >> > > > > > >> that, or deviate? If the latter, would we do a 1.3 > release > > >> for > > >> > > > > metrics? > > >> > > > > > >>> > > >> > > > > > >>> As far as a 1.1.2 release I’m fine with releasing that > at > > >> any > > >> > > time. > > >> > > > > > >>> > > >> > > > > > >>> -Taylor > > >> > > > > > >>> > > >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek Lim < > > >> kabh...@gmail.com> > > >> > > > > wrote: > > >> > > > > > >>>> > > >> > > > > > >>>> I think we could start release phase for both 1.1.2 and > > >> 1.2.0 > > >> > > when > > >> > > > > > >>>> https://github.com/apache/storm/pull/2423 will be > > merged. > > >> > > > > > >>>> > > >> > > > > > >>>> Thanks, > > >> > > > > > >>>> Jungtaek Lim (HeartSaVioR) > > >> > > > > > >>>> > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre Vermeerbergen < > > >> > > > > > >> avermeerber...@gmail.com>님이 > > >> > > > > > >>>> 작성: > > >> > > > > > >>>> > > >> > > > > > >>>>> Hello, > > >> > > > > > >>>>> > > >> > > > > > >>>>> I'd love to see a Storm 1.2.0 released: it's a perfect > > >> > schedule > > >> > > > for > > >> > > > > > me > > >> > > > > > >> for > > >> > > > > > >>>>> test any Release Candidate that might be available if > it > > >> > > happens > > >> > > > > > soon. > > >> > > > > > >>>>> > > >> > > > > > >>>>> Best regards, > > >> > > > > > >>>>> Alexandre > > >> > > > > > >>>>> > > >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan < > > >> ar...@apache.org>: > > >> > > > > > >>>>> > > >> > > > > > >>>>>> Hi, > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> Looks like we are only waiting on > > >> > https://issues.apache.org/ > > >> > > > > > >>>>>> jira/browse/STORM-2546 . > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> Are there any other issues which are blockers for > Storm > > >> > 1.2.0? > > >> > > > > Would > > >> > > > > > >> be > > >> > > > > > >>>>>> great to see the 1.2.0 release out soon as it has a > lot > > >> of > > >> > > > > critical > > >> > > > > > >>>>> fixes. > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> Thanks, > > >> > > > > > >>>>>> Arun > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> > > >> > > > > > >>>>>> > > >> > > > > > >>>>> > > >> > > > > > >>> > > >> > > > > > >> > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
