Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
// why can't developers just use standard messaging solutions like rabbit or apache mq // You CAN. On 01/27/2013 11:40 AM, Justin Karneges wrote: 0mq makes some things practical or more efficient: - Messaging as a general form of development. Consider inproc workers. - A semi-standard way of interfacing between apps. Mongrel2 would be a lot less interesting if it required a broker, IMO. - MxN or mesh routings are relatively painless to set up and make for better sleep at night. The other messaging solutions have their advantages, too. Nothing wrong with many tools. On Saturday, January 26, 2013 09:57:39 PM Matthew Kaufman wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious? On Sat, Jan 26, 2013 at 9:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Started finding answers on my own, so the latest version in Github for PUB mechanism does know who's subscribed to what and it further looks like it only sends matching messages to subscribed SUB client's. https://github.com/zeromq/libzmq/blob/master/src/xpub.cpp#L121 Also in sub there's an additional sanity check to ensure that the client is actually subscribed to messages it receives, very nice. Also got to say it's pretty clever how 0MQ builds up complexity by re-using it's own messaging infrastructure for (un)subscriptions. -- Forwarded message -- From: David J W zeromq-dev-subscr...@ominian.net Date: Sat, Jan 26, 2013 at 12:56 PM Subject: Greetings questions about inproc PUB/SUB inside Twisted. To: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
You will save a ton of time if you read the guide (zguide.zeromq.org) and the FAQ. The guide may appear quite long, but for what you intend to accomplish you could probably stop after chapter 5. I'll give you a head start on a few things. 1. inproc 0mq supports 4 transport types. inproc is for communicating between sockets *within the same* process. It essentially does some pointer trickery so it's very fast. It does *not* work between processes. ipc is for inter-process communication. Most OSes support IPC via local unix domain sockets. 0mq is no different and just relies on the OS mechanism. tcp is for communication between processes on the same or different boxes. All messages go up and down the entire TCP stack unless the OS has optimizations for localhost communication. epgm is for multicast. It cannot be used for talking to processes on the same box. Luckily 0mq lets a single socket bind (and/or connect) to multiple transports at the same time so each limitation isn't a big deal. 2. Subscription forwarding As you have discovered, the 3.2+ releases of libzmq make the PUB socket intelligent about where to send data. It can save quite a bit of bandwidth if you are on a constrained network. 3. Understanding 0mq We encourage and invite you to read the source, but the project (and community) prides itself on having some of the best documentation around (on par with commercial products). You should not need to read the source to understand the library. Please join us on irc.freenode.net in #zeromq or post your questions here. Be warned that one of the first questions asked is did you read the guide? You'll get less help if you answer no because most of us old timers are tired of answering novice questions that are clearly answered in the excellent docs (guide, man pages, faq). Welcome! cr On Jan 26, 2013, at 1:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious? On Sat, Jan 26, 2013 at 9:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
On Jan 26, 2013, at 8:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious….? You joined a mailing list to ask this question? No, you aren't serious. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
No I have been a member and enjoy reading about 0mq. I am considering its use On Sat, Jan 26, 2013 at 10:17 PM, Charles Remes li...@chuckremes.comwrote: On Jan 26, 2013, at 8:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious….? You joined a mailing list to ask this question? No, you aren't serious. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Greetings questions about inproc PUB/SUB inside Twisted.
0mq makes some things practical or more efficient: - Messaging as a general form of development. Consider inproc workers. - A semi-standard way of interfacing between apps. Mongrel2 would be a lot less interesting if it required a broker, IMO. - MxN or mesh routings are relatively painless to set up and make for better sleep at night. The other messaging solutions have their advantages, too. Nothing wrong with many tools. On Saturday, January 26, 2013 09:57:39 PM Matthew Kaufman wrote: Regarding 0mq why can't developers just use standard messaging solutions like rabbit or apache mq what is so cool about 0mq? and yes i am being fully serious? On Sat, Jan 26, 2013 at 9:57 PM, Matthew Kaufman mkfmn...@gmail.com wrote: Ugh Zed Shaw is annoying lol. Hi Zed On Sat, Jan 26, 2013 at 2:56 PM, David J W zeromq-dev-subscr...@ominian.net wrote: Hello, My name is David W, I am a professional code monkey that was first introduced to 0MQ via Zed Shaw's talk at PyCon ( believe it was 2010 ) but only now have gotten the chance to sit down and start learning the library. I am on a sabbatical until ideal after PyCon since I got laid off and taking the time out to hit a bucket list ( including learning 0MQ ). On that note, a few years back as a re-invent the wheel project to learn Twisted and COMET I started writing a web MUD engine and centered the architecture around two message pipelines: User action's were locked stepped ( User A moved left, tell server, wait for it to say yes/no), broadcast to other User's that User A moved left, broadcast to all of User A's group they moved left. NPC's were just headless User's driven by a behavior time/tick subprocess that hooked upto the same pipelines.I set that project aside because I realized I needed a message queue of some sort and really didn't want to setup Rabbit or anything super industrial. Now along arrives 0MQ and since this is a personal project the priority is more about understanding how 0MQ works then accomplishing the actual project. In the above example I can imagine using 0mq's inproc socket's where client's are SUB types ( subscribe to map/domain, subscribe to group chat ) and their is a master process that has a router socket for incoming work and a pub socket for products [ User A in map 1 moved left] ). So here's my questions: For PUB/SUB the impression is that the actual queue sit's on the client socket. PUB pushes a message to all client's [ regardless of setsockopt(zmq.SUBSCRIBE ) ] and the act of reading the socket filter's/clears the queue down to what the client is subscribed to. Is this correct or is the subscription more intelligent ( PUB keep's a subscription roster, see's no one is subscribed, drop's the message OR client receives a message, isn't subscribed so it drops the message ). Has anyone had any experience running multiple SUB based client's inside of one process and are their any severe consequences. I imagine a SUB socket is going to instantiate the needed structures to hold a queue, the actual socket, and other house keeping structures but so far small tests (1-10 sockets) hasn't show much memory use. Additionally, if I do get past digging through 0MQ's mechanics, I was thinking it would be best to spin off the PUB side to it's only process. Which leads me to wonder if 0MQ inproc PUB/SUB actually relies on some clever memory mapping. eg Push a message on an inproc PUB socket which goes to a shared/mutex locked list and client's just read from this one list. Apologies if some of these questions seem naive, I haven't gotten the chance to read 0mq's C source code yet. Thanks, Dave ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
