Hi all, I’m building a homogenous cluster database (think a simplified Riak, just for fun), and I’ve set up over TCP:
DEALER[server A, for A], DEALER[server B, for A], DEALER[server C, for A] <-> ROUTER[A] DEALER[server A, for B], DEALER[server B, for B], DEALER[server C, for B] <-> ROUTER[B] DEALER[server A, for C], DEALER[server B, for C], DEALER[server C, for C] <-> ROUTER[C] Hence, if you want to send a message to N random nodes and collect responses (as my database commands), you pick N random nodes from the member list (thoughtfully implemented at https://github.com/hashicorp/memberlist) and then you grab their DEALER (which on NodeJoin has connected to the appropriate ROUTER), send, and receive from it. However, profiling using timers shows a lot of time being spent in DEALER.RecvMessage(). If I run a thousand sends and recv them all, it takes 40 microseconds each, but when I run ApacheBench for a single request and reply on demand as it has to work, the average time to receive is over 180 microseconds. When you run a read operation, currently I query all known nodes for the key and do a read repair, so reads have a lot of dealer/router round trips involved, sometimes as many as 8 on a 3-node cluster. I can’t figure our where the bottleneck is, because in my experience, ØMQ sockets are very fast. Even sending each message to the ROUTER takes only 6 microseconds; processing it at the other end (i.e. a DB read) only takes 30 microseconds, so all up we should be waiting less than 40 microseconds for three of them to come back, not 500-600 on average as the timers report. I’m beginning to think it’s the Go scheduler, even. Take a look at my code, it’s at github.com/cormacrelf/mec-db (http://github.com/cormacrelf/mec-db) - the networking code resides in peers/peers.go (https://github.com/cormacrelf/mec-db/blob/master/peers/peers.go), the aforementioned replies are in store/store.go (https://github.com/cormacrelf/mec-db/blob/master/store/store.go) Listen(). -- Cormac Relf
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
