Thanks Stig, So after some digging, I realized we are really migrating from the kafka trident emitter in storm-kafka, to the trident emitter in storm-kafka-client.
As far as I can see, the offset information is still stored in zk, and the offset info for storm-kafka is ( https://github.com/apache/storm/blob/master/external/storm-kafka/src/jvm/org/apache/storm/kafka/trident/TridentKafkaEmitter.java#L140 ) However this seems quite different from storm-kafka-client, which uses https://github.com/apache/storm/blob/master/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/trident/KafkaTridentSpoutBatchMetadata.java#L56 I'm not sure under which zknode this information is stored - and if the zknode itself is different between the two implementations. Looks like I need a tool to copy the stored values in zk from old storm-kafka to storm-kafka-client? Another option I suppose is to: - stop topic producers - run the old code until it drains all topics - start new code with FirstPollOffsetStrategy.LATEST Although this seems risky. Thanks! - Nasron On Thu, Dec 21, 2017 at 4:23 PM, Stig Rohde Døssing <[email protected]> wrote: > Hi Nasron, > > I don't believe there's currently a tool to help you migrate. We did it > manually by writing a small utility that looked up the commit offsets in > Storm's Zookeeper, opened a KafkaConsumer with the new consumer group id > and committed the offsets for the appropriate partitions. We stopped our > topologies, used this utility and redeployed with the new spout. > > Assuming there isn't already a tool for migration floating around > somewhere, I think we could probably build some migration support into the > storm-kafka-client spout. If the path to the old offsets in Storm's > Zookeeper is given, we might be able to extract them and start up the new > spout from there. > > 2017-12-19 21:59 GMT+01:00 Nasron Cheong <[email protected]>: > >> Hi, >> >> I'm trying to determine steps for migration to the storm-kafka-client in >> order to use the new kafka client. >> >> It's not quite clear to me how offsets are migrated - is there a specific >> set of steps to ensure offsets are moved from the ZK based offsets into the >> kafka based offsets? >> >> Or is the original configuration respected, and storm-kafka-client can >> mostly be a drop in replacement? >> >> I want to avoid having spouts reset to the beginning of topics after >> deployment, due to this change. >> >> Thanks. >> >> - Nasron >> > >
