Re: [zeromq-dev] Should 0MQ be used and what socket types to use?
On Jan 25, 2013, at 11:58 PM, Niels Berglund niels.it.bergl...@gmail.com wrote: Hi all! I am working for a company whose business is similar to online financial trading, where the similarities are the requirement for low latency and high throughput. The systems are situated in data centers across the world, and clients connect over the internet to an individual system. A system consists of a software router which routes requests from clients to back-end services, and responses to the clients from the services. The services may also send unsolicited messages to the clients. There are a limited number of back-end services, but there could theoretically be an infinite number of concurrent users. Obviously there are limitations how many clients can connect to one specific software router, but there would then be multiple machines with software routers for the clients to connect to. We are now looking at the future architecture, and part of that is communication mechanisms. The underlying OS for this is Windows and at the moment we are using Microsoft's socket library. Our socket code are functional, but not pretty and I have a feeling that if we were to change to something like 0MQ, etc., it would be beneficial for us. So my questions are: 1. Would an architecture like the above be feasible using 0MQ? 2. What 0MQ socket types would be best for an architecture like this? My thoughts are that clients would connect to the software router using a DEALER socket (seeing that we want full asynch, and that the client could send multiple requests without requiring a reply in between). The client-facing socket in the software router would be a ROUTER socket, but I am not sure if the client service facing socket in the software router should be a ROUTER or DEALER? As for the services, they would connect to the software router using a DEALER socket. 1. Yes, this architecture is very feasible. I have personally created a very similar architecture for my own trading applications. 2. I recommend using DEALER/ROUTER exclusively for any bidirectional communications. Those socket types can be viewed as the foundation for any protocol (that is, you can create any of the other patterns such as push/pull or pub/sub using just dealer/router). You will want heartbeating to detect dead (or slow) clients and services, so again dealer/router is the correct choice. I have found the other socket types to primarily be useful for intra-thread communication since they are (from the programmer perspective) uni-directional. My system runs in production on the Windows platform. I have also deployed some services to linux but Windows is the primary target platform. If you have other specific questions, feel free to ask. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Issues with large numbers of clients
You are connecting two peers using the same identity. This is illegal and will result in unspecified behavior. Unless you have a strong reason for it, don't set socket identities at all. Can you make a test case that doesn't set socket identities? -Pieter On Sat, Jan 26, 2013 at 12:02 AM, Will Moss wm...@bu.mp wrote: Hi Peter, I have looked through the internals a bit, but certainly don't understand them entirely. That said, I don't think it's possible for it to clean up on disconnect--if it did it would not be providing the abstraction surrounding client reconnections that it promises. I made a little test program to prove this to myself: https://gist.github.com/662192fac6fb9d77fc2b Let me know if I can provide any more assistance with the epoll bug. I'd love to get it fixed because the behaviour when this occurs is to have the process cpu spike up to 100% and start processing substantially fewer requests that it should (resulting in us having to restart the process and lose data in flight). Thanks, Will On Wed, Jan 23, 2013 at 12:16 PM, Pieter Hintjens p...@imatix.com wrote: On Wed, Jan 23, 2013 at 8:01 PM, Will Moss wm...@bu.mp wrote: I can certainly make a test case for the socket leakage, but as I said before, I thought this was a known issue, so I'm a little confused. The issue the Guide talks about is application code that tracks peers. I've not tested, and it's not documented, how the ROUTER socket manages its internal tables. I'd expect the router socket to clean up its internal tables when clients disconnect. So if it's not doing this, we need a test case. I doubt I'm going to be able to come up with a test case that can get a socket into EPOLLERR or EPOLLHUP, but I think it's reasonably easy to see how this can happen from the code. That may be enough to fix the problem. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Zeromq messages getting dropped
Hi, My guess was it might have an issue on libzmq (zeromq c library) when large messages were heavily sent. Thanks Min On Jan 26, 2013, at 4:01 PM, Ritesh Adval riteshad...@gaikai.com wrote: Hi Min, Thanks for the update.Just to confirm, Are you saying that this issue is on zeromq c library or jzmq c wrapper? Just an update that when I replaced DEALER socket which connects to ROUTER socket of broker with REQ socket and replaced DEALER socket which connects to DEALER socket of broker with REP socket, then I do not see message loss when doing the same test. (REQ socket does send and then recv and REP does opposite recv and send) -Ritesh Sent from my iPhone. On Jan 25, 2013, at 8:42 PM, Min mini...@gmail.com wrote: I was able to reproduce the issue on jzmq even on zeromq 3.2.2. What I discovered is about last 30K bytes of 45K message was not sometimes delivered to in-router on raw close. I didn't build equivalent C code, as jzmq is a thin wrapper of native C library it could have the same problem. But I didn't find a clear solution yet. Thanks Min On Thu, Jan 24, 2013 at 6:39 AM, Ritesh Adval riteshad...@gaikai.com wrote: Hello, I have created a bug for this issue with instructions and java test case. Its at https://zeromq.jira.com/browse/LIBZMQ-497 Thanks Ritesh On Tue, Jan 22, 2013 at 6:30 PM, Ritesh Adval riteshad...@gaikai.com wrote: Thanks Min, I will create a bug with instruction and unit test. I was also experimenting with Java only version of zeromq (https://github.com/zeromq/jeromq). When running same test it does not drop message but has some other issue. -Ritesh On Mon, Jan 21, 2013 at 11:53 PM, Min mini...@gmail.com wrote: Ritesh, If you can reproduce the problem, Java code should be fine. Community could look into it. Thanks Min 2013년 1월 17일 목요일에 Ritesh Adval님이 작성: Hi Charles, I have test program in JAVA, I am not a C programmer so i will probably take me time to reproduce this in C. Can someone first take a look at my JAVA program to see if I am not doing anything stupid. Should I create bug and attach Java maven project? Its very easy to run it, all you need is zeromq 2.2.0 installed and jzmq built and installed by building jzmq (https://github.com/zeromq/jzmq). I can add instructions to the bug report. Once confirmed that program looks right I can try to create a C version of the test but will take me some time. let me know. Thanks Ritesh On Wed, Jan 16, 2013 at 10:55 PM, Charles Remes li...@chuckremes.com wrote: On Jan 16, 2013, at 4:08 PM, Ritesh Adval riteshad...@gaikai.com wrote: Hi Charles, Yes I close the socket in my thread after sending 100 messages, and I expect that LINGER will make sure messages are sent to the other end, I expected that context termination will block and make sure any pending messages are sent, but thats not happening. context termination returns quickly. Just now tried again in my unit test by setting LINGER to Integer.MAX_VALUE explicitly in all my sockets and ran the test again and it did fail with messages getting dropped. The interesting thing is only the 100th message (The last one) from some of my concurrent threads are getting dropped. Time to show someone the code. That's the easiest way to figure it out. If you can reproduce this in C, that will get a lot more attention. Here's how to open an issue: http://www.zeromq.org/docs:issue-tracking cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] jzmq is built without ZThread.
Oh, There're a missing entry. I've sent a pull request, https://github.com/zeromq/jzmq/pull/153 Thanks Min On Jan 14, 2013, at 9:12 PM, crocket crockabisc...@gmail.com wrote: On Windows, I built it with Visual Studio 2010 Express. On Linux, I built it with ./configure; make On Mon, Jan 14, 2013 at 9:11 PM, crocket crockabisc...@gmail.com wrote: I built jzmq on windows and linux, and both builds produced zmq.jar wich lacked ZThread.class in it. When I build zmq.jar with mvn package or mvn jar:jar, zmq.jar has ZThread.class. Is it a bug? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] A discussion about consolidating jeromq namespace with jzmq
Hello, I want to listen opinions from community about consolidating JeroMQ namespace with jzmq. When we use JeroMQ, it starts the following import statement import org.jeromq.ZMQ; import org.jeromq.ZMQ.Socket; ... For jzmq import org.zeromq.ZMQ; import org.zeromq.ZMQ.Socket; ... Other than namespace, class and method signature are compatible with jzmq. I hope JeroMQ to be inter-changible with jzmq just by changing the jar file. If we think current jzmq signature as an Java API, we could use different carrier based on their purpose without changing the source code. At the future, another implementation, say JavaMQ, will appear and we could also it by dropping another jar. Then we could extract general Java interfaces. Please let me know any opinion about renaming JeroMQ namespace from 'org.jeromq' to 'org.zeromq' Thanks Min ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] A discussion about consolidating jeromq namespace with jzmq
Hello, I want to listen opinions from community about consolidating JeroMQ namespace with jzmq. I hope JeroMQ to be inter-changible with jzmq just by changing the jar file. It seems like this would potentially 1) increase test coverage for both projects, 2) simplify preparing minimal test cases reproducing some problems. Both of these result in better code quality. So, +1. Regards, Ivan ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Started finding answers on my own, so the latest version in Github for PUB mechanism does know who's subscribed to what and it further looks like it only sends matching messages to subscribed SUB client's. https://github.com/zeromq/libzmq/blob/master/src/xpub.cpp#L121 Also in sub there's an additional sanity check to ensure that the client is actually subscribed to messages it receives, very nice. Also got to say it's pretty clever how 0MQ builds up complexity by re-using it's own messaging infrastructure for (un)subscriptions. -- Forwarded message -- From: David J W zeromq-dev-subscr...@ominian.net Date: Sat, Jan 26, 2013 at 12:56 PM Subject: Greetings questions about inproc PUB/SUB inside Twisted. To: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
You will save a ton of time if you read the guide (zguide.zeromq.org) and the FAQ. The guide may appear quite long, but for what you intend to accomplish you could probably stop after chapter 5. I'll give you a head start on a few things. 1. inproc 0mq supports 4 transport types. inproc is for communicating between sockets *within the same* process. It essentially does some pointer trickery so it's very fast. It does *not* work between processes. ipc is for inter-process communication. Most OSes support IPC via local unix domain sockets. 0mq is no different and just relies on the OS mechanism. tcp is for communication between processes on the same or different boxes. All messages go up and down the entire TCP stack unless the OS has optimizations for localhost communication. epgm is for multicast. It cannot be used for talking to processes on the same box. Luckily 0mq lets a single socket bind (and/or connect) to multiple transports at the same time so each limitation isn't a big deal. 2. Subscription forwarding As you have discovered, the 3.2+ releases of libzmq make the PUB socket intelligent about where to send data. It can save quite a bit of bandwidth if you are on a constrained network. 3. Understanding 0mq We encourage and invite you to read the source, but the project (and community) prides itself on having some of the best documentation around (on par with commercial products). You should not need to read the source to understand the library. Please join us on irc.freenode.net in #zeromq or post your questions here. Be warned that one of the first questions asked is did you read the guide? You'll get less help if you answer no because most of us old timers are tired of answering novice questions that are clearly answered in the excellent docs (guide, man pages, faq). Welcome! cr On Jan 26, 2013, at 1:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev
Re: [zeromq-dev] jzmq is built without ZThread.
I believe java files other than ZThread.java are missing in ms visual studio project and Makefile. On Sun, Jan 27, 2013 at 1:03 AM, Yu Dongmin mini...@gmail.com wrote: Oh, There're a missing entry. I've sent a pull request, https://github.com/zeromq/jzmq/pull/153 Thanks Min On Jan 14, 2013, at 9:12 PM, crocket crockabisc...@gmail.com wrote: On Windows, I built it with Visual Studio 2010 Express. On Linux, I built it with ./configure; make On Mon, Jan 14, 2013 at 9:11 PM, crocket crockabisc...@gmail.com wrote: I built jzmq on windows and linux, and both builds produced zmq.jar wich lacked ZThread.class in it. When I build zmq.jar with mvn package or mvn jar:jar, zmq.jar has ZThread.class. Is it a bug? ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Issues with large numbers of clients
Pieter, I'm using the same client identity to demonstrate the point that the ROUTER socket cannot possibly be removing the routing information when the client disconnects, since when it reconnects later, the data is still buffered and ready to be sent out. I don't know how I would make a test case that demonstrated the internal workings of the ROUTER socket when there are no methods exposed to show the size of that table or anything else like that. If you just had a process that connected and disconnected from another process like the server process I wrote, I'm sure you would see the memory increase (as well as the cpu load, as it has to scan a larger and larger table) and not drop back down. Will On Sat, Jan 26, 2013 at 6:40 AM, Pieter Hintjens p...@imatix.com wrote: You are connecting two peers using the same identity. This is illegal and will result in unspecified behavior. Unless you have a strong reason for it, don't set socket identities at all. Can you make a test case that doesn't set socket identities? -Pieter On Sat, Jan 26, 2013 at 12:02 AM, Will Moss wm...@bu.mp wrote: Hi Peter, I have looked through the internals a bit, but certainly don't understand them entirely. That said, I don't think it's possible for it to clean up on disconnect--if it did it would not be providing the abstraction surrounding client reconnections that it promises. I made a little test program to prove this to myself: https://gist.github.com/662192fac6fb9d77fc2b Let me know if I can provide any more assistance with the epoll bug. I'd love to get it fixed because the behaviour when this occurs is to have the process cpu spike up to 100% and start processing substantially fewer requests that it should (resulting in us having to restart the process and lose data in flight). Thanks, Will On Wed, Jan 23, 2013 at 12:16 PM, Pieter Hintjens p...@imatix.com wrote: On Wed, Jan 23, 2013 at 8:01 PM, Will Moss wm...@bu.mp wrote: I can certainly make a test case for the socket leakage, but as I said before, I thought this was a known issue, so I'm a little confused. The issue the Guide talks about is application code that tracks peers. I've not tested, and it's not documented, how the ROUTER socket manages its internal tables. I'd expect the router socket to clean up its internal tables when clients disconnect. So if it's not doing this, we need a test case. I doubt I'm going to be able to come up with a test case that can get a socket into EPOLLERR or EPOLLHUP, but I think it's reasonably easy to see how this can happen from the code. That may be enough to fix the problem. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious? On Sat, Jan 26, 2013 at 9:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
On Jan 26, 2013, at 8:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious….? You joined a mailing list to ask this question? No, you aren't serious. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
No I have been a member and enjoy reading about 0mq. I am considering its use On Sat, Jan 26, 2013 at 10:17 PM, Charles Remes li...@chuckremes.comwrote: On Jan 26, 2013, at 8:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious….? You joined a mailing list to ask this question? No, you aren't serious. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
0mq makes some things practical or more efficient: - Messaging as a general form of development. Consider inproc workers. - A semi-standard way of interfacing between apps. Mongrel2 would be a lot less interesting if it required a broker, IMO. - MxN or mesh routings are relatively painless to set up and make for better sleep at night. The other messaging solutions have their advantages, too. Nothing wrong with many tools. On Saturday, January 26, 2013 09:57:39 PM Matthew Kaufman wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious? On Sat, Jan 26, 2013 at 9:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
