stephen, we migrated latest storm-kafka-client to Storm 1.0.6, so I guess there should be no problem on Storm 1.0.6 regarding any issues including Time.nanoTime() you've observed or so.
I'd suggest upgrading Storm to 1.0.6 which upgrade would be expected to be done smoothly, but actually you should be better to upgrade to latest version since Apache Storm community no longer maintains 1.0.x version line actively. Thanks, Jungtaek Lim (HeartSaVioR) 2018년 8월 21일 (화) 오후 2:44, Stig Rohde Døssing <[email protected]>님이 작성: > Just want to add a quick note as well that you should use the latest > storm-kafka-client. The intent is that you add a dependency both to > storm-kafka-client, and org.apache.kafka:kafka-clients in your project. > This will let you control the Kafka client version. See > https://github.com/apache/storm/blob/master/examples/storm-kafka-client-examples/pom.xml#L41 > for example. > > 2018-08-21 4:46 GMT+02:00 Erik Weathers <[email protected]>: > >> Thanks for providing all those details. >> >> > Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6. >> > So, I'm wondering is there a low-influence idea (without changing >> version of storm-core 1.0.3) for storm job to consume message from kafka >> 1.1. Thanks a lot! >> >> I'm going to assume that you care about the version of storm-core that is >> in the cluster (Nimbus, Supervisor, etc.), and that you are fine with >> having storm-core v1.0.6 in your topology. >> If so you can try doing something I have done before. I used the maven >> shade plugin to: >> >> (1) build my topology's uber jar with a newer version of storm-core >> shaded into my uber jar. >> (2) relocated the org.apache.storm.utils class to avoid any conflicts. >> (3) did some exclusions to avoid complaints about not including >> storm-core in the topology uber jar. >> >> I am unsure if the latter 2 steps were fully necessary. But my general >> strategy was just to include whatever bits my topology needed that weren't >> being found in the version of the storm-core jar in the runtime environment >> (Nimbus, Supervisor), and to work around any errors that were thrown. >> >> Here's the related parts of the maven pom file that I had for my earlier >> experiment: >> >> - https://gist.github.com/erikdw/3500c74b23ed2b5e2e808b50c9b39661 >> >> Note that I *was* able to build a storm topology with storm-core v1.1 >> shaded in and then run it on a v1.0.3 storm cluster using this technique, >> so you can try to do the similar exercise with your versions of storm-core. >> I'm guessing this is considered a bad practice though, so looking into >> getting the cluster upgraded to v1.0.6 should be pursued instead of leaving >> the topology in this weird state for a long time. Also, if you do follow >> this route then you should *definitely* mention this oddity if you ask >> future questions of these storm email lists about problems with the >> topology. >> >> - Erik >> >> P.S., not that it negatively impacted me here, but for future reference >> having technical text (that exception you included) in an *image* is a >> bad practice for 2 reasons: >> (1) It is not accessibility friendly for vision impaired folks. They may >> need to use screen-reading software that may lack the ability to translate >> the image to text. >> (2) the content is not indexed for searching the email threads. >> >> On Mon, Aug 20, 2018 at 5:22 PM stephen <[email protected]> wrote: >> >>> Sorry Erik, >>> I'll give the details. The versions I am currently using are as follows： >>> >>> storm-core: 1.0.3 >>> storm-kafka: 1.0.3 >>> kafka: 0.8.1.1 >>> >>> For some reasons, I need to consuming from kafka 1.1. So, I tried to use >>> the storm-kafka-client (1.0.3), but I found the version of kafka-clients is >>> 0.9.0.1 in this storm-kafka-client. I want to use >= 0.10.0 version of >>> kafka-client because of the cost of Message Conversion. If a consumer >>> client is on a version before 0.10.0.0, the kafka broker will convert >>> messages from the 0.10.0 format to an earlier format before sending the >>> response to the consumer on an older version, that will result in >>> additional overhead of performance and may cause an OOM in some cases. >>> >>> I found the version of kafka-client in storm-kafka-client 1.0.6 is >>> 0.10.1. So, I used it running with storm-core 1.0.3, and found an Exception >>> as follow: >>> >>> Time.nanoTime is not a method in storm-core 1.0.3 until 1.0.6. >>> >>> So, I'm wondering is there a low-influence idea (without changing >>> version of storm-core 1.0.3) for storm job to consume message from kafka >>> 1.1. Thanks a lot! >>> >>> Best regards, >>> Kun >>> >>> On 20 Aug 2018, at 11:45, Erik Weathers <[email protected]> wrote: >>> >>> hey Kun, >>> >>> I'm unclear which version of storm-kafka and storm-core you are >>> currently using, please explain in more detail what you are trying to >>> accomplish. >>> You'll need to change from using storm-kafka to storm-kafka-client, but >>> again, please explain the versions of everything you are using. >>> >>> Thanks! >>> >>> - Erik >>> >>> On Sun, Aug 19, 2018 at 7:33 PM stephen <[email protected]> wrote: >>> >>>> Dear all, >>>> >>>> I have a question about upgrade kafka from 0.8.X to 1.1 using >>>> storm-core 1.0.3. Are there some feasible methods for me without changing >>>> the version of storm-core? >>>> >>>> Best regards, >>>> Kun >>>> >>>> >>> >
