Thank you for the suggestion. In our situation we can stop all the Kafka producers before migration so the migration seems to be simpler. The way that we came up is:
- Stop all Kafka producers - Wait for Storm Trident to finish consuming remain messages - Shutdown Storm topology - Install Kafka, Storm, Zookeeper on new data center - Manually create currtx znode of Storm's coordinator and set its value to the value that we have in old data center - Deploy Storm topology to new data center and start consuming from the end of topic I tested and it seems to work. On Tue, Aug 11, 2015 at 11:59 PM, Kishore Senji <[email protected]> wrote: > I haven't tried it myself, but I guess you would just need to sync the > sub-tree for the topology that you are migrating from /transactional/<id> > on the old cluster to the new cluster. /transactional is the default value > of transactional.zookeeper.root unless you have modified. > > Similar is the case for KafkaSpout (non-trident) where the data is stored > in /<root>/<spout-id>/ > > On Tue, Aug 11, 2015 at 5:50 PM Binh Nguyen Van <[email protected]> > wrote: > >> Hi, >> >> I am working on a project that is using Storm Trident for streaming >> processing and I am >> using opaque transaction trident Kafka spout to read data from Kafka, >> process it and >> save result to database. >> >> The system is running for a while and we have more than one topologies >> running. Now >> we decided to slowly move our applications to new data center, we will >> have new Kafka, >> ZooKeeper, Storm and database cluster on new data center while still >> remain the old >> Kafka, ZooKeeper, Storm and database cluster for non-migrated topologies >> to use. >> >> I wonder if any of you have done this before and could you please share >> your experience? >> How do you migrate transactional state that Storm keeps in ZK? We can >> copy snapshot >> files and recover from there when we migrate the first topology but we >> can't do it when >> we migrate other topologies since at that time the two ZK cluster are >> diverse. >> >> Thanks >> -Binh >> >
