You can, but that could potentially cause a major bottleneck and you would need at least 6 nodes to handle 1/2 of them dying at once. With zookeeper, adding more nodes can cause slow performance. I’d love to hear someone tell me I’m wrong and how to better manage it :)
I’ve actually been working on writing our storm (non-trident) offsets from kafka to a separate zookeeper cluster from the one kafka uses to take the load off of our primary zookeeper cluster. > On Aug 28, 2015, at 1:41 PM, Susheel Kumar Gadalay <[email protected]> > wrote: > > Can't you have zookeeper spawning data centers if there is connectivity. > > Enlarge zookeeper to more than 3 nodes and extra nodes from other data > centers and > also minimum replication to be set. > > > On 8/28/15, Rick Mangi <[email protected] <mailto:[email protected]>> wrote: >> As long as you are replicating kafka and your offset checkpoints in the >> secondary datacenter you should be able to do this. The offsets into kafka >> from your spout tell you where you are in your processing (as long as your >> topology is acking). >> >> That said, if those offsets are in zookeeper you will need to find a way to >> replicate those to your failover datacenter. >> >>> On Aug 27, 2015, at 3:08 PM, nitin sharma <[email protected]> >>> wrote: >>> >>> hi Andrew, >>> >>> our setup includes Kafka --> Storm --> Cassandra .. >>> >>> we want to have a backup cluster for Storm, so that in case of any outage >>> the backup server (topology) picks up from where Active server topology >>> stopped and process the events and sends them to Cassandra. >>> >>> >>> Regards, >>> Nitin Kumar Sharma. >>> >>> >>> On Thu, Aug 27, 2015 at 3:04 PM, Andrew Xor <[email protected] >>> <mailto:[email protected] <mailto:[email protected]>>> >>> wrote: >>> I can try and give you some ideas, but start by first telling us how >>> you've setup your infrastructure a bit so we can better help you... >>> >>> Kindly yours, >>> >>> Andrew Grammenos >>> >>> -- PGP PKey -- >>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt >>> <https://www.dropbox.com/s/2kcxe59zsi9nrdt/pgpsig.txt>> >>> https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt >>> <https://www.dropbox.com/s/yxvycjvlsc111bh/pgpsig.txt> >>> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt >>> <https://www.dropbox.com/s/ei2nqsen641daei/pgpsig.txt>> >>> >>> On Thu, Aug 27, 2015 at 9:28 PM, nitin sharma <[email protected] >>> <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>> >>> wrote: >>> Hi All, >>> >>> I have been asked how can we setup a failover datacenter for Storm, so >>> that if by any chance (lets say due to power outages) the complete Active >>> datacenter becomes offline then Standby Datacenter can become alive and >>> process the real time feeds. >>> >>> Does anyone have faced this kind of scenario? what are the best practice >>> to follow? >>> >>> Regards, >>> Nitin Kumar Sharma.
signature.asc
Description: Message signed with OpenPGP using GPGMail
