Alexandre, There's so much pain on using relocated artifact in other module in a project. You need to refer the class as relocated name, and your IDE (at least IntelliJ) complains and the compilation in IDE will not succeed. (The build via Maven will succeed though.) There was same issue on 2.0.0 as well, and for now we don't shade anything in 2.0.0. We need to discuss which things to shade before discussing about releasing, indeed.
And normally we have been allowing adding dependency as minor release. I understand someone could say that it is breaking dependency, and should lead another relocation as well. But my 2 cents the requirement is too restrictive, and may end up with Guava-like versioning. (releasing major versions sooner...) I think relocating is not an ideal solution, and eventually we may have nice solution (like classloader isolation) and then we could sort out all the related issues. Before that, I could not block adding dependencies in minor release if it is necessary. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 11월 19일 (일) 오후 8:02, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성: > Alexandre, > > Thanks for trying. I think this time it's not a problem with > storm-kafka-client. It has a dependency on commons-lang 2.5, which is > declared in the storm-kafka-client pom. We usually recommend that people > use Maven/Gradle/Ivy or similar systems to ensure that they get all > dependencies when they build their topologies. In such a case Maven (or > similar) would see that your topology depends on storm-kafka-client, and > also download and include commons-lang in the topology jar. When you put > together dependencies manually you need to include commons-lang manually as > well. > > You can get the commons-lang 2.5 jar here > > http://central.maven.org/maven2/commons-lang/commons-lang/2.5/commons-lang-2.5.jar > > 2017-11-19 11:49 GMT+01:00 Alexandre Vermeerbergen < > avermeerber...@gmail.com > >: > > > Hello Stig, > > > > Than you very much for you quick answer & fix. > > Unfortunaly, I still have topologies failing to start but with a > different > > exception this time: > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > org/apache/commons/lang/StringUtils > > at > > org.apache.storm.kafka.spout.NamedTopicFilter.getTopicsString( > > NamedTopicFilter.java:66) > > at > > org.apache.storm.kafka.spout.ManualPartitionSubscription.getTopicsString( > > ManualPartitionSubscription.java:70) > > at > > com.acme.storm.evaluator.spout.BasicKafkaSpout.<init>( > > BasicKafkaSpout.java:59) > > at com.acme.storm.util.StormUtils.getKafkaSpout( > > StormUtils.java:888) > > at > > com.acme.storm.evaluator.SLAEventsInterceptorTopology.main( > > SLAEventsInterceptorTopology.java:120) > > Caused by: java.lang.ClassNotFoundException: > > org.apache.commons.lang.StringUtils > > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338) > > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > > > > Looks like you introduced a dependency on commons lang that wasn't part > of > > Storm, isn't it ? > > > > Keep faith, next time will be better :) > > > > Best regards, > > Alexandre > > > > 2017-11-19 10:44 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: > > > > > I've put up a fix here https://github.com/apache/storm/pull/2426. > > There's > > > an updated storm-kafka-client jar at > > > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_ > > > v9fd/view?usp=sharing > > > if you'd like to try it out. > > > > > > 2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com > >: > > > > > > > Oops. The exception is not intentional, it's a bug. In 1.2.0 we check > > for > > > > the "enable.auto.commit" key in the kafkaConsumerProp map, and if it > is > > > set > > > > we warn in the log that it shouldn't be, because users should use the > > > > KafkaSpoutConfig.Builder.setProcessingGuarantee method instead. When > > the > > > > property is set we try to set the processing guarantee to match the > > > > pre-1.2.0 behavior, but I made a mistake and assumed the property key > > is > > > a > > > > boolean when it might actually be either a boolean or a string. I'll > > put > > > up > > > > a fix ASAP. > > > > > > > > 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00 Alexandre Vermeerbergen > > < > > > > avermeerber...@gmail.com>: > > > > > > > >> Hello Stig, > > > >> > > > >> Here's my first feedback on this Storm 1.2.0 preview on my > Supervision > > > >> system based on Storm : I have 11 topologies KO (not even able to > > > start), > > > >> and 4 topologies which seem to be unaffected. > > > >> > > > >> Details: > > > >> - I used the binaries posted on dropbox by Jungteak and your binary > > for > > > >> storm-kafka-client > > > >> - Rebuilt all our topologies using these storm-core.jar & > > > >> storm-kafka-client.jar from these storm-1.2.0-snapshot (and I got > > > >> absolutely no error at build time). > > > >> Meaning that I have new "big jar" files including this newer > > > >> storm-kafka-client.jar (except for 1 topology which depends on > another > > > >> build system) > > > >> - The 11 topologies which fail to start show up the following trace > > when > > > >> they are submitted: > > > >> > > > >> Running: java -client -Ddaemon.name= -Dstorm.options= > > > >> -Dstorm.home=/usr/local/Storm/storm-stable > > > >> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs > > > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib > > > >> -Dstorm.conf.file= -cp > > > >> /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-stabl > > > >> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/local/ > > > >> Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-stable/ > > > >> conf:/usr/local/Storm/storm-stable/bin > > > >> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar > > > >> -Dstorm.dependency.jars= > > > >> -Dstorm.dependency.artifacts={} > > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology > > > >> slaEventsInterceptor > > > >> Latest > > > >> 1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology - Topology : > > > >> 'slaEventsInterceptor', KafkaOffsetTimeStrategy = 'Latest' > > > >> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read > > > >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties > > > file: > > > >> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties > > (No > > > >> such file or directory) > > > >> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set > > > >> enable.auto.commit manually. Instead use > > > >> KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be > treated > > > as > > > >> an > > > >> error in the next major release. For now the spout will be > configured > > to > > > >> behave like it would have in pre-1.2.0 releases. > > > >> Exception in thread "main" java.lang.ClassCastException: > > > java.lang.String > > > >> cannot be cast to java.lang.Boolean > > > >> at > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.setAutoCommitM > > > >> ode(KafkaSpoutConfig.java:721) > > > >> at > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(KafkaSp > > > >> outConfig.java:97) > > > >> at > > > >> org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.build( > > > >> KafkaSpoutConfig.java:671) > > > >> at > > > >> com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(S > > > >> LAEventsInterceptorTopology.java:118) > > > >> > > > >> The line part of our application which triggered above stack trace > is > > > the > > > >> following one: > > > >> > > > >> KafkaSpoutConfig<String, Event> spoutConfigForMainTopic = > > > >> KafkaSpoutConfig > > > >> .builder(elasticKafkaBrokers, KafkaTopics.MAIN) > > > >> .setValue(EventKafkaDeserializer.class) > > > >> .setGroupId(consumerId + "_" + KafkaTopics.MAIN) > > > >> .setFirstPollOffsetStrategy(strategy) > > > >> .setProp(kafkaConsumerProp) > > > >> .setRecordTranslator( > > > >> new SLAEventsInterceptorKafkaRecor > > > dTranslator()) > > > >> .build(); > > > >> > > > >> I understand that there's a WARN message, but if it's a warning, > then > > > why > > > >> do we exit with such a brutal exception? > > > >> > > > >> Best regards, > > > >> Alexandre Vermeerbergen > > > >> > > > >> > > > >> > > > >> > > > >> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing < > stigdoess...@gmail.com > > >: > > > >> > > > >> > Alexandre, > > > >> > > > > >> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client jar here > > > >> > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_ > > > >> > v9fd/view?usp=sharing. > > > >> > You can probably keep the 1.1.0 versions of storm-hbase and > > > storm-hdfs, > > > >> > since the core Storm API hasn't changed. > > > >> > > > > >> > If you want to build the jars yourself, clone the repository at > > > >> > https://github.com/apache/storm and check out the 1.x-branch > > branch. > > > >> You > > > >> > can build the entire project by running "mvn clean install > > > -DskipTests" > > > >> in > > > >> > the root. The individual module jars will be available in the > target > > > >> > directories for each module, e.g. storm-kafka-client.jar will be > in > > > >> > external/storm-kafka-client/target. > > > >> > > > > >> > If you need to build the regular Storm distribution (the tarball > > > >> Jungtaek > > > >> > sent you), you can go to storm-dist/binary and run "mvn clean > > install > > > >> > -Dgpg.skip". The tar/zip will be in storm-dist/binary/target. > > > >> > > > > >> > Further reference in case you need it > > > >> > https://github.com/apache/storm/blob/master/DEVELOPER. > > > >> > md#build-the-code-and-run-the-tests > > > >> > . > > > >> > > > > >> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen < > > > >> > avermeerber...@gmail.com > > > >> > >: > > > >> > > > > >> > > Hello Jungtaek, > > > >> > > > > > >> > > Thanks for the link to 1.2.0 preview binaries. > > > >> > > However, since we heavily depend on Storm Kafka Client, would > you > > > >> please > > > >> > > either add it or remind me how to build the jars of this > external > > > >> lib, at > > > >> > > same "snapshot" version as the rest ? > > > >> > > On a side note, some of our topologies also rely on storm hbase > & > > > >> storm > > > >> > > hdfs : does it matters if these later ones stay in 1.1.0 version > > for > > > >> this > > > >> > > test? > > > >> > > Indeed, I want to focus on Storm 1.2.0 "core" + Storm Kafka > Client > > > >> 1.2.0 > > > >> > > > > > >> > > Best regards, > > > >> > > Alexandre Vermeerbergen > > > >> > > > > > >> > > > > > >> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00 Jungtaek Lim < > > > >> kabh...@gmail.com>: > > > >> > > > > > >> > > > Alexandre, > > > >> > > > > > > >> > > > https://www.dropbox.com/s/mg2gnunk24oesyc/apache-storm- > > > >> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0 > > > >> > > > > > > >> > > > Above link is custom binary distribution of current 1.x-branch > > > >> > (SNAPSHOT > > > >> > > of > > > >> > > > 1.2.0). Could you run the test in your environment first and > let > > > us > > > >> > know > > > >> > > > about the result? Regardless of including metrics V2 to 1.2.0 > or > > > >> not, > > > >> > > your > > > >> > > > test report should be valuable for us. > > > >> > > > > > > >> > > > And please refer my analysis of current metrics: > > > >> > > > https://cwiki.apache.org/confluence/display/STORM/ > > > >> > > Limitations+of+current+ > > > >> > > > metrics+feature > > > >> > > > to > > > >> > > > see why we want to move toward to Metrics V2. Current PR on > > > Metrics > > > >> V2 > > > >> > is > > > >> > > > for initial state and it could cover only some of issues in > > list, > > > >> but I > > > >> > > > expect we will address another issues as well based on the PR. > > > >> > > > > > > >> > > > Thanks in advance! > > > >> > > > Jungtaek Lim (HeartSaVioR) > > > >> > > > > > > >> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre Vermeerbergen < > > > >> > > > avermeerber...@gmail.com>님이 > > > >> > > > 작성: > > > >> > > > > > > >> > > > > Hello Hugo, > > > >> > > > > > > > >> > > > > As I already posted, I was getting ready to test the > upcoming > > > RC, > > > >> so > > > >> > > > yeah : > > > >> > > > > as soon as I have some binaries for testing, I can run any > > > Storm > > > >> > 1.2.0 > > > >> > > > > preview on my pre-production (well stressed) environment. Is > > > there > > > >> > some > > > >> > > > URL > > > >> > > > > from where I can download such "snapshot" binaries? > > > >> > > > > > > > >> > > > > > > > >> > > > > Best regards, > > > >> > > > > Alexandre Vermeerbergen > > > >> > > > > > > > >> > > > > PS: By the way, I have no interest in the metrics-related > > > >> > enhancements > > > >> > > > you > > > >> > > > > mentionned - I can understand it is of interest for people > > aware > > > >> of > > > >> > it > > > >> > > > and > > > >> > > > > who are expecting "something" related to it - but I would be > > sad > > > >> to > > > >> > > see a > > > >> > > > > postponing of the imminent Storm 1.2.0 release for something > > we > > > do > > > >> > not > > > >> > > > use > > > >> > > > > at all. > > > >> > > > > PPS: I would love to learn more about these "metrics", maybe > > > it's > > > >> > > > something > > > >> > > > > worth which never quite understood... any link with clear > > > >> explanation > > > >> > > of > > > >> > > > > this feature? > > > >> > > > > > > > >> > > > > > > > >> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro < > > > >> > hlo...@hortonworks.com > > > >> > > >: > > > >> > > > > > > > >> > > > > > I am also in agreement that we should not delay the 1.2.0 > > > >> release > > > >> > for > > > >> > > > too > > > >> > > > > > long, but in order to release it a few things need to be > > kept > > > in > > > >> > > mind: > > > >> > > > > > > > > >> > > > > > - If we want to avoid releasing 1.3.0 soon or at all, > that > > > >> means > > > >> > > that > > > >> > > > we > > > >> > > > > > should include in 1.2.0 as many important features (e.g. > > > >> metricsV2) > > > >> > > as > > > >> > > > > > possible within a reasonable time frame. Also to avoid as > > much > > > >> back > > > >> > > > > porting > > > >> > > > > > as possible one should really get all features that we > > foresee > > > >> we > > > >> > > want > > > >> > > > to > > > >> > > > > > maintain in 1.2.0 right away, and then simply maintain > them > > > >> there. > > > >> > > > > However > > > >> > > > > > it’s a fact of life that if a bug is found and it is a > > > blocker, > > > >> it > > > >> > > will > > > >> > > > > > have to be back ported. > > > >> > > > > > > > > >> > > > > > - STORM-2153 details that the new metrics requirements > were > > > >> driven > > > >> > by > > > >> > > > the > > > >> > > > > > users. Therefore if this feature is highly important and > > > sought > > > >> > > after, > > > >> > > > it > > > >> > > > > > may not make much sense to release 1.2.0 without it. If we > > do > > > >> > release > > > >> > > > > 1.2.0 > > > >> > > > > > without the metrics, does it mean that metrics will go > only > > in > > > >> > 2.0? I > > > >> > > > > would > > > >> > > > > > conjecture that most production deployments will take a > > while > > > to > > > >> > > > upgrade > > > >> > > > > to > > > >> > > > > > 2.0 even after it is released. That means that they will > > still > > > >> be > > > >> > > > running > > > >> > > > > > Storm without the benefit of the new metrics. > > > >> > > > > > > > > >> > > > > > - Several fixes have got in storm-kafka-client. There > have > > > >> been a > > > >> > > lot > > > >> > > > of > > > >> > > > > > changes and I wonder to which degree they have been system > > > >> tested > > > >> > in > > > >> > > > > > addition the existing unit tests > > > >> > > > > > > > > >> > > > > > @Alexandre, since you are using storm-kafka-client and > have > > > >> filed > > > >> > > some > > > >> > > > > > bugs and driven some feature requests, I would like to ask > > if > > > >> you > > > >> > > could > > > >> > > > > > help us within what is reasonably possible for you with > the > > > >> > > following: > > > >> > > > > > > > > >> > > > > > - system test the latest storm-kafka-client changes in > your > > > >> > > > > > test/pre-production environment > > > >> > > > > > - provide some information on about your setup, > > > >> > > > > > - how you are using storm-kafka-client > > > >> > > > > > - how it is performing > > > >> > > > > > - Kafka brokers, number of topics/partitions, storm > > > >> > > parallelism, > > > >> > > > > and > > > >> > > > > > some info about your network > > > >> > > > > > - if it is reasonable to do so share some of your tests > such > > > >> that > > > >> > we > > > >> > > > can > > > >> > > > > > also test it at our end. > > > >> > > > > > - Tell us specific things that you would like us to test > > > >> > > > > > > > > >> > > > > > Thanks, > > > >> > > > > > Hugo > > > >> > > > > > > > > >> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde Døssing < > > > >> > > > > stigdoess...@gmail.com> > > > >> > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > I agree with Jungtaek, if metrics v2 can go in very soon > > it > > > >> > should > > > >> > > go > > > >> > > > > in, > > > >> > > > > > > otherwise I'd rather release 1.2.0 now and work on > getting > > > >> 2.0.0 > > > >> > > > ready > > > >> > > > > > for > > > >> > > > > > > release. > > > >> > > > > > > > > > >> > > > > > > 1.x and master have drifted pretty far apart, and it's > > > >> causing a > > > >> > > lot > > > >> > > > of > > > >> > > > > > > porting work at this point. 1.x still has a lot of > Clojure > > > >> code, > > > >> > > > > > storm-core > > > >> > > > > > > hasn't been split yet in that branch and it's also > > targeting > > > >> JDK > > > >> > > 1.7. > > > >> > > > > > It's > > > >> > > > > > > seems rare at this point that a PR cherry picks cleanly > > onto > > > >> 1.x > > > >> > > from > > > >> > > > > > > master. > > > >> > > > > > > > > > >> > > > > > > I've linked the list of issues that are only fixed in > > 2.0.0 > > > >> and > > > >> > not > > > >> > > > > 1.x, > > > >> > > > > > to > > > >> > > > > > > illustrate how far ahead 2.0.0 is. > > > >> > > > > > > > > > >> > > > > > > https://pste.eu/p/9CJT.html > > > >> > > > > > > > > > >> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer < > > ai...@hortonworks.com > > > >: > > > >> > > > > > > > > > >> > > > > > >> Hi Taylor, > > > >> > > > > > >> > > > >> > > > > > >> Is it https://github.com/apache/storm/pull/2203 ? > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> I think it would be great to get this in 1.2 release. > Can > > > we > > > >> try > > > >> > > to > > > >> > > > > > >> address the issues in a week or so and get this in? > > > >> > > > > > >> > > > >> > > > > > >> Thanks, > > > >> > > > > > >> Arun > > > >> > > > > > >> > > > >> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" < > > ptgo...@gmail.com > > > > > > > >> > > wrote: > > > >> > > > > > >> > > > >> > > > > > >>> The original idea for the 1.2 release was that would > > > >> introduce > > > >> > > the > > > >> > > > > > >> metrics v2 work (and there is additional work required > > > there, > > > >> > so I > > > >> > > > > > >> understand a desire not to delay the release). Do we > want > > > to > > > >> > stick > > > >> > > > > with > > > >> > > > > > >> that, or deviate? If the latter, would we do a 1.3 > > release > > > >> for > > > >> > > > > metrics? > > > >> > > > > > >>> > > > >> > > > > > >>> As far as a 1.1.2 release I’m fine with releasing that > > at > > > >> any > > > >> > > time. > > > >> > > > > > >>> > > > >> > > > > > >>> -Taylor > > > >> > > > > > >>> > > > >> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek Lim < > > > >> kabh...@gmail.com> > > > >> > > > > wrote: > > > >> > > > > > >>>> > > > >> > > > > > >>>> I think we could start release phase for both 1.1.2 > and > > > >> 1.2.0 > > > >> > > when > > > >> > > > > > >>>> https://github.com/apache/storm/pull/2423 will be > > > merged. > > > >> > > > > > >>>> > > > >> > > > > > >>>> Thanks, > > > >> > > > > > >>>> Jungtaek Lim (HeartSaVioR) > > > >> > > > > > >>>> > > > >> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre Vermeerbergen < > > > >> > > > > > >> avermeerber...@gmail.com>님이 > > > >> > > > > > >>>> 작성: > > > >> > > > > > >>>> > > > >> > > > > > >>>>> Hello, > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> I'd love to see a Storm 1.2.0 released: it's a > perfect > > > >> > schedule > > > >> > > > for > > > >> > > > > > me > > > >> > > > > > >> for > > > >> > > > > > >>>>> test any Release Candidate that might be available > if > > it > > > >> > > happens > > > >> > > > > > soon. > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> Best regards, > > > >> > > > > > >>>>> Alexandre > > > >> > > > > > >>>>> > > > >> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan < > > > >> ar...@apache.org>: > > > >> > > > > > >>>>> > > > >> > > > > > >>>>>> Hi, > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Looks like we are only waiting on > > > >> > https://issues.apache.org/ > > > >> > > > > > >>>>>> jira/browse/STORM-2546 . > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Are there any other issues which are blockers for > > Storm > > > >> > 1.2.0? > > > >> > > > > Would > > > >> > > > > > >> be > > > >> > > > > > >>>>>> great to see the 1.2.0 release out soon as it has a > > lot > > > >> of > > > >> > > > > critical > > > >> > > > > > >>>>> fixes. > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> Thanks, > > > >> > > > > > >>>>>> Arun > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>>> > > > >> > > > > > >>>>> > > > >> > > > > > >>> > > > >> > > > > > >> > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > > > > > >
