Vishal, Please see the freelance model; a broker design won't scale to 1M messages a second without significant extra work.
-Pieter On Fri, Dec 12, 2014 at 7:57 AM, Vishal Ahuja <[email protected]> wrote: > Andre, thank you for your help! I changed the client to send 10k requests at > once, and then send two requests for every reply that it receives. I chose > 10k because that allowed me to run up to ten clients (each sending 100k > messages) in parallel without any packet loss. Here is the client code: > > #include "../include/mdp.h" > > #include <time.h> > > int main (int argc, char *argv []) > > { > > int verbose = (argc > 1 && streq (argv [1], "-v")); > > mdp_client_t *session = mdp_client_new (argv[1], verbose); > > //mdp_client_t *session = mdp_client_new("tcp://localhost:5555", > verbose); > > int count1, count2; > > struct timeval start,end; > > gettimeofday(&start, NULL); > > for (count1 = 0; count1 < 10000; count1++) { > > zmsg_t *request = zmsg_new (); > > zmsg_pushstr (request, "Hello world"); > > mdp_client_send (session, "echo", &request); > > } > > printf("sent all\n"); > > for (count1 = 0; count1 < 45000; count1++) { > > zmsg_t *reply = mdp_client_recv (session,NULL,NULL); > > if (reply) > > { > > zmsg_destroy (&reply); > > zmsg_t *request = zmsg_new (); > > zmsg_pushstr (request, "Hello world"); > > mdp_client_send (session, "echo", &request); > > > request = zmsg_new (); > > zmsg_pushstr (request, "Hello world"); > > mdp_client_send (session, "echo", &request); > > } > > else > > break; // Interrupted by Ctrl-C > > } > > /* receiving the remaining 55k replies */ > > for(count1 = 45000; count1 < 100000; count1++) > > { > > zmsg_t *reply = mdp_client_recv (session,NULL,NULL); > > if (reply) > > { > > zmsg_destroy (&reply); > > } > > else > > break; > > > } > > gettimeofday(&end, NULL); > > long elapsed = (end.tv_sec - start.tv_sec) +((end.tv_usec - > start.tv_usec)/1000000); > > printf("time = %ld\n", elapsed); > > printf ("%d replies received\n", count1); > > mdp_client_destroy (&session); > > return 0; > > } > > > I ran the broker, worker, and the clients within the same machine. Here is > the recorded time: > > # of clients in parallel > > (each client sends 100k ) Time elapsed (seconds) > > 1 4 > > 2 9 > > 3 12 > > 4 16 > > 5 21 > > 10 43 > > > So for every 100k requests, the broker is taking about 4 seconds. Is this > the expected behavior? Am not sure how to achieve million messages per > second. Please advise. > > Sincerely, > > Vishal > > > > > On Thu, Dec 11, 2014 at 10:22 PM, André Caron <[email protected]> > wrote: >> >> You are observing lost replies because the broker is sending replies >> faster than the client is consuming them: each time the broker sends a >> reply, it is stored in an internal queue for that client. When the broker >> sends more replies than the client can consume, this internal queue fills up >> to tolerate shorts bursts (the client's RCVHWM and the broker's SNDHWM limit >> indirectly control this tolerance). However, if the broker's send rate is >> constantly faster than the client's recv rate, you need infinite memory to >> keep avoid losing replies. >> >> In this scenario, the clients are operating in request-reply, so if this >> happens, they are sending requests faster than they can consume the replies. >> Basically, your client is sending requests until both the client's reply >> queue and the broker's reply queue (for that client) are saturated and then, >> instead of reading replies, it continues sending requests. This doesn't >> make any sense in any real world scenario. The "real" problem here is that >> the test your are performing is not realistic. >> >> Note that it would be different if it was pub-sub instead of req-rep >> because the client would not be in control of the broker's "reply" rate. >> >> To fix this to match a real-world scenario, you should modify your client >> program such that it has at most N outstanding requests, where N is >> guaranteed to fit in the cilent's reply queue and broker's reply queue: >> - initially send N requests; >> - each time you get a reply, send a new request. >> >> If you do this and the client's RCVHWM and broker's SNDHWM are >> sufficiently high to hold a total of N replies, then you will go down to 0 >> lost replies -- unless you hit other problems like disconnections, of >> course. >> >> Instead of doing that, you could technically increase the server's SNDHWM >> to hold 100,000 replies per client (assuming the broker has enough memory), >> but the problem same problem would creep up again if you increase the number >> of outstanding requests to more than 100,000. >> >> Cheers, >> >> André >> >> On Thu, Dec 11, 2014 at 8:05 AM, Vishal Ahuja <[email protected]> wrote: >>> >>> Sure, but when I have 3 parallel clients (300k messages), I am observing >>> packet loss. >>> >>> Sincerely, >>> >>> Vishal >>> >>> On Thu, Dec 11, 2014 at 5:43 PM, André Caron <[email protected]> >>> wrote: >>>> >>>> Throughput is bound by multiple factors. In your case, my guess is that >>>> your bottleneck is the throughput of a single TCP socket on your system. I >>>> doubt that you can reach millions of messages per second with only 3 >>>> clients. If you add more clients, you should see the server's throughput >>>> increase well over 30k. >>>> >>>> Cheers, >>>> >>>> André >>>> >>>> On Thu, Dec 11, 2014 at 4:40 AM, Vishal Ahuja <[email protected]> wrote: >>>>> >>>>> I removed the printfs, and increased the TCP send and receive buffers >>>>> to the max possible. When I have two clients (running in parallel) sending >>>>> 100k messages, there is no packet loss. But when I increase the number of >>>>> clients to three, then each client receives around 93k packets. Also, the >>>>> majordomo specification (http://rfc.zeromq.org/spec:7) says that the >>>>> broker >>>>> should be able to switch millions of messages per second, whereas I >>>>> seeing a >>>>> throughput of around 30k messages per second. Can someone tell how to >>>>> improve the throughput? >>>>> >>>>> Sincerely, >>>>> >>>>> Vishal >>>>> >>>>> On Thu, Dec 11, 2014 at 10:37 AM, Andre Caron <[email protected]> >>>>> wrote: >>>>>> >>>>>> The broker uses a router socket, which silently drops outbound >>>>>> messages when the sndhwm is reached. If your client is reading more >>>>>> slowly >>>>>> than the broker is writing, you'll lose messages. >>>>>> >>>>>> In your case, the slow part is writing to the terminal. Try removing >>>>>> the printf in the recv loop and you should lose fewer messages. >>>>>> >>>>>> André >>>>>> >>>>>> > On Dec 10, 2014, at 9:40 PM, Vishal Ahuja <[email protected]> wrote: >>>>>> > >>>>>> > While reading the zeromq guide, I came across client code >>>>>> > (pertaining to the majordomo pattern) which sends 100k requests in a >>>>>> > loop, >>>>>> > and then receives the reply in a second loop. >>>>>> > >>>>>> > #include "../include/mdp.h" >>>>>> > #include <time.h> >>>>>> > >>>>>> > >>>>>> > int main (int argc, char *argv []) >>>>>> > { >>>>>> > int verbose = (argc > 1 && streq (argv [1], "-v")); >>>>>> > mdp_client_t *session = mdp_client_new >>>>>> > ("tcp://localhost:5555", verbose); >>>>>> > int count; >>>>>> > for (count = 0; count < 100000; count++) { >>>>>> > zmsg_t *request = zmsg_new (); >>>>>> > zmsg_pushstr (request, "Hello world"); >>>>>> > mdp_client_send (session, "echo", &request); >>>>>> > } >>>>>> > printf("sent all\n"); >>>>>> > >>>>>> > for (count = 0; count < 100000; count++) { >>>>>> > zmsg_t *reply = mdp_client_recv (session,NULL,NULL); >>>>>> > if (reply) >>>>>> > zmsg_destroy (&reply); >>>>>> > else >>>>>> > break; // Interrupted by Ctrl-C >>>>>> > printf("reply received:%d\n", count); >>>>>> > } >>>>>> > printf ("%d replies received\n", count); >>>>>> > mdp_client_destroy (&session); >>>>>> > return 0; >>>>>> > } >>>>>> > >>>>>> > I have added a counter to count the number of replies that the >>>>>> > worker (test_worker.c) sends to the broker, and another counter in >>>>>> > mdp_broker.c to count the number of replies the broker sends to a >>>>>> > client. >>>>>> > Both of these count up to 100k, but the client is receiving only >>>>>> > around 37k >>>>>> > replies. >>>>>> > >>>>>> > If the number of client requests is set to around 40k, then it >>>>>> > receives all the replies. Can someone please tell me why packets are >>>>>> > lost >>>>>> > when the client sends more than 40k asynchronous requests? >>>>>> > >>>>>> > I tried setting the HWM to 100k for the broker socket, but the >>>>>> > problem persists: >>>>>> > >>>>>> > static broker_t * >>>>>> > s_broker_new (int verbose) >>>>>> > { >>>>>> > broker_t *self = (broker_t *) zmalloc (sizeof (broker_t)); >>>>>> > int64_t hwm = 100000; >>>>>> > // Initialize broker state >>>>>> > self->ctx = zctx_new (); >>>>>> > self->socket = zsocket_new (self->ctx, ZMQ_ROUTER); >>>>>> > zmq_setsockopt(self->socket, ZMQ_SNDHWM, &hwm, sizeof(hwm)); >>>>>> > >>>>>> > zmq_setsockopt(self->socket, ZMQ_RCVHWM, &hwm, sizeof(hwm)); >>>>>> > self->verbose = verbose; >>>>>> > self->services = zhash_new (); >>>>>> > self->workers = zhash_new (); >>>>>> > self->waiting = zlist_new (); >>>>>> > self->heartbeat_at = zclock_time () + HEARTBEAT_INTERVAL; >>>>>> > return self; >>>>>> > } >>>>>> > >>>>>> > Vishal >>>>>> > _______________________________________________ >>>>>> > zeromq-dev mailing list >>>>>> > [email protected] >>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>>> _______________________________________________ >>>>>> zeromq-dev mailing list >>>>>> [email protected] >>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> zeromq-dev mailing list >>>>> [email protected] >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> zeromq-dev mailing list >>>> [email protected] >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>>> >>> >>> >>> _______________________________________________ >>> zeromq-dev mailing list >>> [email protected] >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >>> >> >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> > > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
