the question is, whether using a dedicated TCP channel per request a good idea or is there a better way to do this kind of comm. with zeromq. it does work, but i am not sure that it is the correct way to do it. assuming there is such a thing. i was hoping to be able to reuse or share the TCP sockets between multiple requests...
as for your suggestion about malamute. I'll take a look at it. Thanks, Koren On Mon, Jul 20, 2015, 00:21 Kenneth Adam Miller <[email protected]> wrote: > I don't have any problem or question explicitly stated, or even that there > is some needed improvement that you don't have, so it's hard to know how to > be helpful. The one thing I would say that I have experience with and that > I could also help with is the idea of your external service managing > requests from potentially N different users. Eventually, N will grow to be > too large to be effectively handled by one machine, so you'll need to load > balance. That's what the malamute project is about, so if you want if you > want to address the idea of scalability there, I can help you. > > On Sun, Jul 19, 2015 at 5:12 PM, Koren Shoval <[email protected]> wrote: > >> Hi, >> >> I have a scenario that I'd like to get some feedback on. Though I've read >> a large part of the guide and played around, I'm still quite a notice with >> ZMQ. >> >> My use case is this: (kind of long, sorry -- not sure how to explain >> better) >> >> a user talks to an 'external' service and asks to get data. >> the 'external' service talks to a 'data' service and asks for data (not >> using zmq) >> >> i'm attempting to leverage zmq to do fast and stable transmission of data >> between 'external' and multiple 'data' services. >> >> there can and will be up to 2000 clients talking to the 'external' >> service and each asking for data. >> each client request will go to a different 'data' service and it will >> serve the request (that part is not done in zmq at the moment) >> >> the 'external' service does not need to send anything to the 'data' >> service. >> >> what i've done so far, is use a dealer at each point in a p2p kind of >> way, where i open a tcp dealr socket in the 'external' and (not using zmq) >> tell the 'data' service to send data to the 'external' service's zmq tcp >> socket. i do that with another zmq dealer socket in the 'data' service side. >> >> so the bottom line is that for each client --> external service <-- data >> service >> i open a two way dealer connection in tcp. >> >> no router and no other sockets are used. >> i feel like i'm wasting tcp resources, too many ports and too many active >> connections. >> >> i thought about using a router or doing something like major domo, but it >> seems to be too costly to send an identity on each message block. (data >> returned to each client request is avg. 50MB) >> and i used 64KB buffers and sent binary messages. >> >> also, to do what with major domo. i'd have to kind of create a unique >> identity to each client request. to i'd have 2000 services. and all just >> for p2p behavior. >> >> thought about router tcp with inproc dealers to reuse a single tcp socket >> for all communications between an 'external' service and a single 'data' >> service. but i could not get that to work, since i can't share the state >> between threads and each dealer works with one client request. it was >> complicated to get working. >> >> i also tried to abuse pub/sub and treat each client request as a 'topic'. >> but i feel like multicast is a waste of network resource here, since there >> will only be 1:1 subscription to broadcast here. >> >> any help will be appreciated. >> >> Thanks, >> Koren >> >> _______________________________________________ >> zeromq-dev mailing list >> [email protected] >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev >> >> > _______________________________________________ > zeromq-dev mailing list > [email protected] > http://lists.zeromq.org/mailman/listinfo/zeromq-dev >
_______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
