I've put up a fix here https://github.com/apache/storm/pull/2426. There's
an updated storm-kafka-client jar at
https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_v9fd/view?usp=sharing
if you'd like to try it out.
2017-11-19 10:17 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> Oops. The exception is not intentional, it's a bug. In 1.2.0 we check for
> the "enable.auto.commit" key in the kafkaConsumerProp map, and if it is set
> we warn in the log that it shouldn't be, because users should use the
> KafkaSpoutConfig.Builder.setProcessingGuarantee method instead. When the
> property is set we try to set the processing guarantee to match the
> pre-1.2.0 behavior, but I made a mistake and assumed the property key is a
> boolean when it might actually be either a boolean or a string. I'll put up
> a fix ASAP.
>
> 2017-11-19 <20%2017%2011%2019> 9:55 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com>:
>
>> Hello Stig,
>>
>> Here's my first feedback on this Storm 1.2.0 preview on my Supervision
>> system based on Storm : I have 11 topologies KO (not even able to start),
>> and 4 topologies which seem to be unaffected.
>>
>> Details:
>> - I used the binaries posted on dropbox by Jungteak and your binary for
>> storm-kafka-client
>> - Rebuilt all our topologies using these storm-core.jar &
>> storm-kafka-client.jar from these storm-1.2.0-snapshot (and I got
>> absolutely no error at build time).
>> Meaning that I have new "big jar" files including this newer
>> storm-kafka-client.jar (except for 1 topology which depends on another
>> build system)
>> - The 11 topologies which fail to start show up the following trace when
>> they are submitted:
>>
>> Running: java -client -Ddaemon.name= -Dstorm.options=
>> -Dstorm.home=/usr/local/Storm/storm-stable
>> -Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
>> -Dstorm.conf.file= -cp
>> /usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-stabl
>> e/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/local/
>> Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-stable/
>> conf:/usr/local/Storm/storm-stable/bin
>> -Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar
>> -Dstorm.dependency.jars=
>> -Dstorm.dependency.artifacts={}
>> com.acme.storm.evaluator.SLAEventsInterceptorTopology
>> slaEventsInterceptor
>> Latest
>> 1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology - Topology :
>> 'slaEventsInterceptor', KafkaOffsetTimeStrategy = 'Latest'
>> 1026 [main] WARN c.a.s.u.StormUtils - Couldn't read
>> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties file:
>> /usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties (No
>> such file or directory)
>> 1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
>> enable.auto.commit manually. Instead use
>> KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated as
>> an
>> error in the next major release. For now the spout will be configured to
>> behave like it would have in pre-1.2.0 releases.
>> Exception in thread "main" java.lang.ClassCastException: java.lang.String
>> cannot be cast to java.lang.Boolean
>> at
>> org.apache.storm.kafka.spout.KafkaSpoutConfig.setAutoCommitM
>> ode(KafkaSpoutConfig.java:721)
>> at
>> org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(KafkaSp
>> outConfig.java:97)
>> at
>> org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.build(
>> KafkaSpoutConfig.java:671)
>> at
>> com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(S
>> LAEventsInterceptorTopology.java:118)
>>
>> The line part of our application which triggered above stack trace is the
>> following one:
>>
>> KafkaSpoutConfig<String, Event> spoutConfigForMainTopic =
>> KafkaSpoutConfig
>> .builder(elasticKafkaBrokers, KafkaTopics.MAIN)
>> .setValue(EventKafkaDeserializer.class)
>> .setGroupId(consumerId + "_" + KafkaTopics.MAIN)
>> .setFirstPollOffsetStrategy(strategy)
>> .setProp(kafkaConsumerProp)
>> .setRecordTranslator(
>> new SLAEventsInterceptorKafkaRecordTranslator())
>> .build();
>>
>> I understand that there's a WARN message, but if it's a warning, then why
>> do we exit with such a brutal exception?
>>
>> Best regards,
>> Alexandre Vermeerbergen
>>
>>
>>
>>
>> 2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
>>
>> > Alexandre,
>> >
>> > I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client jar here
>> > https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_
>> > v9fd/view?usp=sharing.
>> > You can probably keep the 1.1.0 versions of storm-hbase and storm-hdfs,
>> > since the core Storm API hasn't changed.
>> >
>> > If you want to build the jars yourself, clone the repository at
>> > https://github.com/apache/storm and check out the 1.x-branch branch.
>> You
>> > can build the entire project by running "mvn clean install -DskipTests"
>> in
>> > the root. The individual module jars will be available in the target
>> > directories for each module, e.g. storm-kafka-client.jar will be in
>> > external/storm-kafka-client/target.
>> >
>> > If you need to build the regular Storm distribution (the tarball
>> Jungtaek
>> > sent you), you can go to storm-dist/binary and run "mvn clean install
>> > -Dgpg.skip". The tar/zip will be in storm-dist/binary/target.
>> >
>> > Further reference in case you need it
>> > https://github.com/apache/storm/blob/master/DEVELOPER.
>> > md#build-the-code-and-run-the-tests
>> > .
>> >
>> > 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen <
>> > avermeerber...@gmail.com
>> > >:
>> >
>> > > Hello Jungtaek,
>> > >
>> > > Thanks for the link to 1.2.0 preview binaries.
>> > > However, since we heavily depend on Storm Kafka Client, would you
>> please
>> > > either add it or remind me how to build the jars of this external
>> lib, at
>> > > same "snapshot" version as the rest ?
>> > > On a side note, some of our topologies also rely on storm hbase &
>> storm
>> > > hdfs : does it matters if these later ones stay in 1.1.0 version for
>> this
>> > > test?
>> > > Indeed, I want to focus on Storm 1.2.0 "core" + Storm Kafka Client
>> 1.2.0
>> > >
>> > > Best regards,
>> > > Alexandre Vermeerbergen
>> > >
>> > >
>> > > 2017-11-18 <20%2017%2011%2018> 7:41 GMT+01:00 Jungtaek Lim <
>> kabh...@gmail.com>:
>> > >
>> > > > Alexandre,
>> > > >
>> > > > https://www.dropbox.com/s/mg2gnunk24oesyc/apache-storm-
>> > > > 1.2.0-SNAPSHOT.tar.gz?dl=0
>> > > >
>> > > > Above link is custom binary distribution of current 1.x-branch
>> > (SNAPSHOT
>> > > of
>> > > > 1.2.0). Could you run the test in your environment first and let us
>> > know
>> > > > about the result? Regardless of including metrics V2 to 1.2.0 or
>> not,
>> > > your
>> > > > test report should be valuable for us.
>> > > >
>> > > > And please refer my analysis of current metrics:
>> > > > https://cwiki.apache.org/confluence/display/STORM/
>> > > Limitations+of+current+
>> > > > metrics+feature
>> > > > to
>> > > > see why we want to move toward to Metrics V2. Current PR on Metrics
>> V2
>> > is
>> > > > for initial state and it could cover only some of issues in list,
>> but I
>> > > > expect we will address another issues as well based on the PR.
>> > > >
>> > > > Thanks in advance!
>> > > > Jungtaek Lim (HeartSaVioR)
>> > > >
>> > > > 2017년 11월 18일 (토) 오전 6:40, Alexandre Vermeerbergen <
>> > > > avermeerber...@gmail.com>님이
>> > > > 작성:
>> > > >
>> > > > > Hello Hugo,
>> > > > >
>> > > > > As I already posted, I was getting ready to test the upcoming RC,
>> so
>> > > > yeah :
>> > > > > as soon as I have some binaries for testing, I can run any Storm
>> > 1.2.0
>> > > > > preview on my pre-production (well stressed) environment. Is there
>> > some
>> > > > URL
>> > > > > from where I can download such "snapshot" binaries?
>> > > > >
>> > > > >
>> > > > > Best regards,
>> > > > > Alexandre Vermeerbergen
>> > > > >
>> > > > > PS: By the way, I have no interest in the metrics-related
>> > enhancements
>> > > > you
>> > > > > mentionned - I can understand it is of interest for people aware
>> of
>> > it
>> > > > and
>> > > > > who are expecting "something" related to it - but I would be sad
>> to
>> > > see a
>> > > > > postponing of the imminent Storm 1.2.0 release for something we do
>> > not
>> > > > use
>> > > > > at all.
>> > > > > PPS: I would love to learn more about these "metrics", maybe it's
>> > > > something
>> > > > > worth which never quite understood... any link with clear
>> explanation
>> > > of
>> > > > > this feature?
>> > > > >
>> > > > >
>> > > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro <
>> > hlo...@hortonworks.com
>> > > >:
>> > > > >
>> > > > > > I am also in agreement that we should not delay the 1.2.0
>> release
>> > for
>> > > > too
>> > > > > > long, but in order to release it a few things need to be kept in
>> > > mind:
>> > > > > >
>> > > > > > - If we want to avoid releasing 1.3.0 soon or at all, that
>> means
>> > > that
>> > > > we
>> > > > > > should include in 1.2.0 as many important features (e.g.
>> metricsV2)
>> > > as
>> > > > > > possible within a reasonable time frame. Also to avoid as much
>> back
>> > > > > porting
>> > > > > > as possible one should really get all features that we foresee
>> we
>> > > want
>> > > > to
>> > > > > > maintain in 1.2.0 right away, and then simply maintain them
>> there.
>> > > > > However
>> > > > > > it’s a fact of life that if a bug is found and it is a blocker,
>> it
>> > > will
>> > > > > > have to be back ported.
>> > > > > >
>> > > > > > - STORM-2153 details that the new metrics requirements were
>> driven
>> > by
>> > > > the
>> > > > > > users. Therefore if this feature is highly important and sought
>> > > after,
>> > > > it
>> > > > > > may not make much sense to release 1.2.0 without it. If we do
>> > release
>> > > > > 1.2.0
>> > > > > > without the metrics, does it mean that metrics will go only in
>> > 2.0? I
>> > > > > would
>> > > > > > conjecture that most production deployments will take a while to
>> > > > upgrade
>> > > > > to
>> > > > > > 2.0 even after it is released. That means that they will still
>> be
>> > > > running
>> > > > > > Storm without the benefit of the new metrics.
>> > > > > >
>> > > > > > - Several fixes have got in storm-kafka-client. There have
>> been a
>> > > lot
>> > > > of
>> > > > > > changes and I wonder to which degree they have been system
>> tested
>> > in
>> > > > > > addition the existing unit tests
>> > > > > >
>> > > > > > @Alexandre, since you are using storm-kafka-client and have
>> filed
>> > > some
>> > > > > > bugs and driven some feature requests, I would like to ask if
>> you
>> > > could
>> > > > > > help us within what is reasonably possible for you with the
>> > > following:
>> > > > > >
>> > > > > > - system test the latest storm-kafka-client changes in your
>> > > > > > test/pre-production environment
>> > > > > > - provide some information on about your setup,
>> > > > > > - how you are using storm-kafka-client
>> > > > > > - how it is performing
>> > > > > > - Kafka brokers, number of topics/partitions, storm
>> > > parallelism,
>> > > > > and
>> > > > > > some info about your network
>> > > > > > - if it is reasonable to do so share some of your tests such
>> that
>> > we
>> > > > can
>> > > > > > also test it at our end.
>> > > > > > - Tell us specific things that you would like us to test
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Hugo
>> > > > > >
>> > > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde Døssing <
>> > > > > stigdoess...@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > I agree with Jungtaek, if metrics v2 can go in very soon it
>> > should
>> > > go
>> > > > > in,
>> > > > > > > otherwise I'd rather release 1.2.0 now and work on getting
>> 2.0.0
>> > > > ready
>> > > > > > for
>> > > > > > > release.
>> > > > > > >
>> > > > > > > 1.x and master have drifted pretty far apart, and it's
>> causing a
>> > > lot
>> > > > of
>> > > > > > > porting work at this point. 1.x still has a lot of Clojure
>> code,
>> > > > > > storm-core
>> > > > > > > hasn't been split yet in that branch and it's also targeting
>> JDK
>> > > 1.7.
>> > > > > > It's
>> > > > > > > seems rare at this point that a PR cherry picks cleanly onto
>> 1.x
>> > > from
>> > > > > > > master.
>> > > > > > >
>> > > > > > > I've linked the list of issues that are only fixed in 2.0.0
>> and
>> > not
>> > > > > 1.x,
>> > > > > > to
>> > > > > > > illustrate how far ahead 2.0.0 is.
>> > > > > > >
>> > > > > > > https://pste.eu/p/9CJT.html
>> > > > > > >
>> > > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer <ai...@hortonworks.com>:
>> > > > > > >
>> > > > > > >> Hi Taylor,
>> > > > > > >>
>> > > > > > >> Is it https://github.com/apache/storm/pull/2203 ?
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> I think it would be great to get this in 1.2 release. Can we
>> try
>> > > to
>> > > > > > >> address the issues in a week or so and get this in?
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >> Arun
>> > > > > > >>
>> > > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" <ptgo...@gmail.com>
>> > > wrote:
>> > > > > > >>
>> > > > > > >>> The original idea for the 1.2 release was that would
>> introduce
>> > > the
>> > > > > > >> metrics v2 work (and there is additional work required there,
>> > so I
>> > > > > > >> understand a desire not to delay the release). Do we want to
>> > stick
>> > > > > with
>> > > > > > >> that, or deviate? If the latter, would we do a 1.3 release
>> for
>> > > > > metrics?
>> > > > > > >>>
>> > > > > > >>> As far as a 1.1.2 release I’m fine with releasing that at
>> any
>> > > time.
>> > > > > > >>>
>> > > > > > >>> -Taylor
>> > > > > > >>>
>> > > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek Lim <
>> kabh...@gmail.com>
>> > > > > wrote:
>> > > > > > >>>>
>> > > > > > >>>> I think we could start release phase for both 1.1.2 and
>> 1.2.0
>> > > when
>> > > > > > >>>> https://github.com/apache/storm/pull/2423 will be merged.
>> > > > > > >>>>
>> > > > > > >>>> Thanks,
>> > > > > > >>>> Jungtaek Lim (HeartSaVioR)
>> > > > > > >>>>
>> > > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre Vermeerbergen <
>> > > > > > >> avermeerber...@gmail.com>님이
>> > > > > > >>>> 작성:
>> > > > > > >>>>
>> > > > > > >>>>> Hello,
>> > > > > > >>>>>
>> > > > > > >>>>> I'd love to see a Storm 1.2.0 released: it's a perfect
>> > schedule
>> > > > for
>> > > > > > me
>> > > > > > >> for
>> > > > > > >>>>> test any Release Candidate that might be available if it
>> > > happens
>> > > > > > soon.
>> > > > > > >>>>>
>> > > > > > >>>>> Best regards,
>> > > > > > >>>>> Alexandre
>> > > > > > >>>>>
>> > > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan <
>> ar...@apache.org>:
>> > > > > > >>>>>
>> > > > > > >>>>>> Hi,
>> > > > > > >>>>>>
>> > > > > > >>>>>> Looks like we are only waiting on
>> > https://issues.apache.org/
>> > > > > > >>>>>> jira/browse/STORM-2546 .
>> > > > > > >>>>>>
>> > > > > > >>>>>> Are there any other issues which are blockers for Storm
>> > 1.2.0?
>> > > > > Would
>> > > > > > >> be
>> > > > > > >>>>>> great to see the 1.2.0 release out soon as it has a lot
>> of
>> > > > > critical
>> > > > > > >>>>> fixes.
>> > > > > > >>>>>>
>> > > > > > >>>>>> Thanks,
>> > > > > > >>>>>> Arun
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
