Just want to add a quick note as well that you should use the latest storm-kafka-client. The intent is that you add a dependency both to storm-kafka-client, and org.apache.kafka:kafka-clients in your project. This will let you control the Kafka client version. See https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41 for example.
2018-08-21 4:46 GMT+02:00 Erik Weathers <[email protected]>: > Thanks for providing all those details. > > > Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6. > > So, I'm wondering is there a low-influence idea (without changing > version of storm-core 1.0.3) for storm job to consume message from kafka > 1.1. Thanks a lot! > > I'm going to assume that you care about the version of storm-core that is > in the cluster (Nimbus, Supervisor, etc.), and that you are fine with > having storm-core v1.0.6 in your topology. > If so you can try doing something I have done before. I used the maven > shade plugin to: > > (1) build my topology's uber jar with a newer version of storm-core shaded > into my uber jar. > (2) relocated the org.apache.storm.utils class to avoid any conflicts. > (3) did some exclusions to avoid complaints about not including storm-core > in the topology uber jar. > > I am unsure if the latter 2 steps were fully necessary. But my general > strategy was just to include whatever bits my topology needed that weren't > being found in the version of the storm-core jar in the runtime environment > (Nimbus, Supervisor), and to work around any errors that were thrown. > > Here's the related parts of the maven pom file that I had for my earlier > experiment: > > - https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661 > > Note that I *was* able to build a storm topology with storm-core v1.1 > shaded in and then run it on a v1.0.3 storm cluster using this technique, > so you can try to do the similar exercise with your versions of storm-core. > I'm guessing this is considered a bad practice though, so looking into > getting the cluster upgraded to v1.0.6 should be pursued instead of leaving > the topology in this weird state for a long time. Also, if you do follow > this route then you should *definitely* mention this oddity if you ask > future questions of these storm email lists about problems with the > topology. > > - Erik > > P.S., not that it negatively impacted me here, but for future reference > having technical text (that exception you included) in an *image* is a > bad practice for 2 reasons: > (1) It is not accessibility friendly for vision impaired folks. They may > need to use screen-reading software that may lack the ability to translate > the image to text. > (2) the content is not indexed for searching the email threads. > > On Mon, Aug 20, 2018 at 5:22 PM stephen <[email protected]> wrote: > >> Sorry Erik, >> I'll give the details. The versions I am currently using are as follows： >> >> storm-core: 1.0.3 >> storm-kafka: 1.0.3 >> kafka: 0.8.1.1 >> >> For some reasons, I need to consuming from kafka 1.1. So, I tried to use >> the storm-kafka-client (1.0.3), but I found the version of kafka-clients is >> 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of >> kafka-client because of the cost of Message Conversion. If a consumer >> client is on a version before 0.10.0.0, the kafka broker will convert >> messages from the 0.10.0 format to an earlier format before sending the >> response to the consumer on an older version, that will result in >> additional overhead of performance and may cause an OOM in some cases. >> >> I found the version of kafka-client in storm-kafka-client 1.0.6 is >> 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception >> as follow: >> >> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6. >> >> So, I'm wondering is there a low-influence idea (without changing version >> of storm-core 1.0.3) for storm job to consume message from kafka 1.1. >> Thanks a lot! >> >> Best regards, >> Kun >> >> On 20 Aug 2018, at 11:45, Erik Weathers <[email protected]> wrote: >> >> hey Kun, >> >> I'm unclear which version of storm-kafka and storm-core you are currently >> using, please explain in more detail what you are trying to accomplish. >> You'll need to change from using storm-kafka to storm-kafka-client, but >> again, please explain the versions of everything you are using. >> >> Thanks! >> >> - Erik >> >> On Sun, Aug 19, 2018 at 7:33 PM stephen <[email protected]> wrote: >> >>> Dear all, >>> >>> I have a question about upgrade kafka from 0.8.X to 1.1 using storm-core >>> 1.0.3. Are there some feasible methods for me without changing the version >>> of storm-core? >>> >>> Best regards, >>> Kun >>> >>> >>
