Re: [zeromq-dev] Notes from a hackathon
I agree that the best place for session management is at the transport. However implementing the session management at the server socket level should be pretty simple, even as POC. Only issue is how to make sure user is not sending the same messages as the socket sending, but we might ignore the problem at the beginning. Do you think it worth a try or to leave it to the transport? On Tue, Feb 3, 2015 at 3:06 PM, Pieter Hintjens p...@imatix.com wrote: On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote: Pieter - regarding the blog post, I'm not sure about setting the routing id on the socket Indeed, that isn't atomic with sending and so can't work with threadsafe sockets. I've changed the article. I think we can either get the routing id as a message property, and/or add a reply to message semantic that does this implicitly. The second is easier for simple servers, the first is better for realistic servers. Since it only applies to server sockets, I think we would benefit from a zserver_reply () method (or similar). We might also think of moving the picture send/recv API into libzmq at some point since it's the immediate replacement for framing. I also have a question regarding the session management (really liked it), but how would know it is a socket level message and not a application level? Also do you think it should be enabled by default or only if an option is set? To make session management work we'd have to implement it using ZMTP control commands, invisibly to the application. Same as heartbeating at that level. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
I'm not sure if I like having picture as a libzmq level thing. It really only makes sense for C language bindings in my book. most other bindings use their own serialization mechanisms. On Tue, Feb 3, 2015 at 2:06 PM, Pieter Hintjens p...@imatix.com wrote: On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote: Pieter - regarding the blog post, I'm not sure about setting the routing id on the socket Indeed, that isn't atomic with sending and so can't work with threadsafe sockets. I've changed the article. I think we can either get the routing id as a message property, and/or add a reply to message semantic that does this implicitly. The second is easier for simple servers, the first is better for realistic servers. Since it only applies to server sockets, I think we would benefit from a zserver_reply () method (or similar). We might also think of moving the picture send/recv API into libzmq at some point since it's the immediate replacement for framing. I also have a question regarding the session management (really liked it), but how would know it is a socket level message and not a application level? Also do you think it should be enabled by default or only if an option is set? To make session management work we'd have to implement it using ZMTP control commands, invisibly to the application. Same as heartbeating at that level. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] zmqstream with req socket only sends the first message
Thanks a lot for your answer, now it works like a charm. I couldn't find documentation about this, do you think it's a good idea to add a paragraph to zmqstream.send docstring (or anywhere else) about threaded use? If you think it's a good idea, I have a suggestion based on the info you gave me. It's in the patch below, but feel free to modify in the way you think it is more appropriate. -- From cf9418dc14b2dc2e1aca5be1b79a105982dcd877 Mon Sep 17 00:00:00 2001 From: drebs dr...@riseup.net Date: Tue, 3 Feb 2015 12:50:59 -0200 Subject: [PATCH] Add doc for threaded zmqstream.send. --- zmq/eventloop/zmqstream.py |8 1 file changed, 8 insertions(+) diff --git a/zmq/eventloop/zmqstream.py b/zmq/eventloop/zmqstream.py index 86a97e4..59149ec 100644 --- a/zmq/eventloop/zmqstream.py +++ b/zmq/eventloop/zmqstream.py @@ -242,6 +242,14 @@ class ZMQStream(object): def send(self, msg, flags=0, copy=True, track=False, callback=None): Send a message, optionally also register a new callback for sends. See zmq.socket.send for details. + +When using threads, calling send on the zmqstream from another thread +doesn’t properly tell the IOLoop thread that there’s an event to +process. This could cause small delays if the IOLoop is already +processing lots of events, but it can also cause the message to never +be sent if the zmq socket is the only one it’s handling. In this case, +you should hand off the stream.send to the IOLoop’s thread via +IOLoop.add_callback. return self.send_multipart([msg], flags=flags, copy=copy, track=track, callback=callback) -- 1.7.10.4 -- MinRK codificou 9.4K bytes: Calling send on the zmqstream from another thread doesn’t properly tell the IOLoop thread that there’s an event to process. This could call small delays if the IOLoop is already processing lots of events, but it can cause the message to never send if the zmq socket is the only one it’s handling. You want your ZmqREQConnection.send to hand off the stream.send to the IOLoop’s thread via IOLoop.add_callback: def send(self, *args, **kwargs): print(Sending message to backend: (%s, %s) % (args, kwargs)) self._ioloop.add_callback(lambda : self._stream.send(*args, **kwargs)) -MinRK On Mon, Feb 2, 2015 at 12:58 PM, drebs [1]dr...@riseup.net wrote: Hello, zmq community. I’m trying to use a pyzmq’s zmqstream with a REQ socket to talk to a txzmq REP socket, but only the first message is ever sent and received back. REQ pyzmq endpoint code: [2]http://paste.debian.net/143615/ REP txzmq endpoint code: [3]http://paste.debian.net/143617/ I would expect that all the messages would be sent. Am I missing something here? If I let some requests accumulate, and then start the server, all the accumulated requests are replied, but the new ones are not sent. If I replace the REQ endpoint by a pure pyzmq version with no zmqstream or ioloop, it works fine ([4]http://paste.debian.net/143619/). Anyone can give some light on this? Thanks a lot! drebs. — -- zeromq-dev mailing list [5]zeromq-dev@lists.zeromq.org [6]http://lists.zeromq.org/mailman/listinfo/zeromq-dev References Visible links 1. http://mailto:dr...@riseup.net/ 2. http://paste.debian.net/143615/ 3. http://paste.debian.net/143617/ 4. http://paste.debian.net/143619/ 5. mailto:zeromq-dev@lists.zeromq.org 6. http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
Thanks for the questions. I'd try to flatter by saying how great these questions are, except I know you're all way too smart to fall for that. :) Let me answer a bunch of questions in one go: * Sorry for not posting this here. I've written a more detailed analysis of the motivation for this thread (and experiments): http://hintjens.com/blog:84. * The main goal with this is simplicity and accuracy. I'm tired of teaching people ZeroMQ constructs that are clearly over-designed and disconnected from reality in subtle ways. Each apology for friction (sorry, but don't use REQ-REP, they are kind of weird, sorry, IDENTITY is not really identity, sorry, message sometimes means frame and sometimes message) really signals costs up and down the stacks. * Sessions require protocol commands that are invisible to normal use. That means either protocol support (e.g. command frames) or some layering on top of ZeroMQ blobs. * Picture sending solves a cross-language problem, namely how to construct zproto-style commands without using dedicated codecs. While the API is C-ish, the functionality is not. I'd expect that to be wrapped in bindings. * Per-language serialization is an anti-pattern. It's common, and fine as such, yet ZeroMQ should IMO always strive to be cross-language. * REQ-REP between threads may make sense. We certainly use it (ZAP, for authentication). * Is this discussion C4 compatible? Yes, no, this is chatter, not a plan or roadmap. Until pull requests hit the road, it's all vapour. However the problems I restated in my blog article are real, to many people, and should be aired. * Multipart does easy serialization, yes. Not in C though. We've learned how to do variable list serialization with zproto-compatible picture sending in CZMQ. Is this a sufficient alternative? I think so. zsock_send() was essential to making actors cheap enough to use widely. egrep zsock_send *.c in malamute-core/src. * Locking on socketsL: we did in fact have a pull request for this once. It was... nasty. We reverted it. We breathed a sigh of relief. However looking back, the problem was valid and has never been solved since. * If you're doing multipart a lot in an application, you can replace that with zproto. You already have contracts then, why not make them explicit and documented? That's always a win in my experience. * Multi-hop request reply is a strange beast and I'm keen to understand those architectures that use it. My suspicion is that we've shifted our vision of what ZeroMQ is over the years. My own experience seems plausible, yet I distrust it. So, the stories in my article are please for falsification, not attempts to convince. * I think we have learned to not break existing code. It's going to be interesting to run multiple models at once. We do this in CZMQ and it works OK. There isn't much overhead. * Authentication and routing information should be (MUST be) two totally separate layers. CURVE certainly does not use the routing ID (even if it's still misnamed as IDENTITY). You can have the same client on N connections (obviously). You authenticate a client using a mechanism-specific key: PLAIN user id, or CURVE public key. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
* Sessions require protocol commands that are invisible to normal use. That means either protocol support (e.g. command frames) or some layering on top of ZeroMQ blobs. What would the argument be for not just using command frames for this? On Tue, Feb 3, 2015 at 4:04 PM, Pieter Hintjens p...@imatix.com wrote: Thanks for the questions. I'd try to flatter by saying how great these questions are, except I know you're all way too smart to fall for that. :) Let me answer a bunch of questions in one go: * Sorry for not posting this here. I've written a more detailed analysis of the motivation for this thread (and experiments): http://hintjens.com/blog:84. * The main goal with this is simplicity and accuracy. I'm tired of teaching people ZeroMQ constructs that are clearly over-designed and disconnected from reality in subtle ways. Each apology for friction (sorry, but don't use REQ-REP, they are kind of weird, sorry, IDENTITY is not really identity, sorry, message sometimes means frame and sometimes message) really signals costs up and down the stacks. * Sessions require protocol commands that are invisible to normal use. That means either protocol support (e.g. command frames) or some layering on top of ZeroMQ blobs. * Picture sending solves a cross-language problem, namely how to construct zproto-style commands without using dedicated codecs. While the API is C-ish, the functionality is not. I'd expect that to be wrapped in bindings. * Per-language serialization is an anti-pattern. It's common, and fine as such, yet ZeroMQ should IMO always strive to be cross-language. * REQ-REP between threads may make sense. We certainly use it (ZAP, for authentication). * Is this discussion C4 compatible? Yes, no, this is chatter, not a plan or roadmap. Until pull requests hit the road, it's all vapour. However the problems I restated in my blog article are real, to many people, and should be aired. * Multipart does easy serialization, yes. Not in C though. We've learned how to do variable list serialization with zproto-compatible picture sending in CZMQ. Is this a sufficient alternative? I think so. zsock_send() was essential to making actors cheap enough to use widely. egrep zsock_send *.c in malamute-core/src. * Locking on socketsL: we did in fact have a pull request for this once. It was... nasty. We reverted it. We breathed a sigh of relief. However looking back, the problem was valid and has never been solved since. * If you're doing multipart a lot in an application, you can replace that with zproto. You already have contracts then, why not make them explicit and documented? That's always a win in my experience. * Multi-hop request reply is a strange beast and I'm keen to understand those architectures that use it. My suspicion is that we've shifted our vision of what ZeroMQ is over the years. My own experience seems plausible, yet I distrust it. So, the stories in my article are please for falsification, not attempts to convince. * I think we have learned to not break existing code. It's going to be interesting to run multiple models at once. We do this in CZMQ and it works OK. There isn't much overhead. * Authentication and routing information should be (MUST be) two totally separate layers. CURVE certainly does not use the routing ID (even if it's still misnamed as IDENTITY). You can have the same client on N connections (obviously). You authenticate a client using a mechanism-specific key: PLAIN user id, or CURVE public key. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote: Pieter - regarding the blog post, I'm not sure about setting the routing id on the socket Indeed, that isn't atomic with sending and so can't work with threadsafe sockets. I've changed the article. I think we can either get the routing id as a message property, and/or add a reply to message semantic that does this implicitly. The second is easier for simple servers, the first is better for realistic servers. Since it only applies to server sockets, I think we would benefit from a zserver_reply () method (or similar). We might also think of moving the picture send/recv API into libzmq at some point since it's the immediate replacement for framing. I also have a question regarding the session management (really liked it), but how would know it is a socket level message and not a application level? Also do you think it should be enabled by default or only if an option is set? To make session management work we'd have to implement it using ZMTP control commands, invisibly to the application. Same as heartbeating at that level. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
Pieter - regarding the blog post, I'm not sure about setting the routing id on the socket, the problem is that it's not thread safe (2 calls to the socket) and could lead to complexity, we can use the thread local storage for the routing id but I think it is still complicated. Why not have it as part of the send method when sending messages, like: zsock_send(void *self, int routing_id, const char *picture, ...) when you don't need it pass zero or use another overload without the routing id. I also have a question regarding the session management (really liked it), but how would know it is a socket level message and not a application level? Also do you think it should be enabled by default or only if an option is set? On Tue, Feb 3, 2015 at 12:04 PM, Pieter Hintjens p...@imatix.com wrote: There has been a zmq_sendiov/recviov call for ages, undocumented and rarely used. This would indeed be a good alternative to multipart for the zerocopy use case. On Mon, Feb 2, 2015 at 10:59 PM, Maurice Barnum m...@yahoo-inc.com wrote: Is there any thought to adding a writev() analogue like zmq_msg_sendv()? One of the reasons I send send multiple frames is to avoid stitching together buffers just to call send, which something like writev() would address that use without the complexity of the socket buffering the parts. On Monday, February 2, 2015 8:36 AM, Joe McIlvain joe.eli@gmail.com wrote: In my opinion, maintaining some kind of multi-frame abstraction at the API level is in fact very useful to applications, even if they are sent atomically on the implementation level as you describe. For example, in CZMQ It's quite useful in program flow to be able to push or pop a single zframe onto or off of a zmsg, then send that zmsg as an entire message. Rarely do I find myself using zframe_send or zframe_recv with the send more and receive more flags in a CZMQ application. The CZMQ picture messages are nice, but they are not always the easiest API to integrate into an application. Having zmsg and zframe remain (or something like them), while removing zframe_send and zframe_recv could ease applications relying on multi-frame semantics into the transition. The implementation for zmsg would simply serialize the frames in some zmsg-determined way into a single-part zmq_msg to send and deserialize according to the same scheme to receive. This multi-frame serialization essentially be at the application level rather than at the ZMTP level. In short, I think a multi-frame abstraction is important for applications, but if we remove the ability to send one frame at a time at the fundamental level, this abstraction can remain at a higher level (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with the proposed changes. On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote: So the new sockets are in the github master, you can take a loot at the test to see how to use the new routing id field: https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp Few of the reasons I didn't like multi frames: * I tried in the past to make both zeromq and NetMQ thread safe, think of sending from multiple threads or receiving from multiple threads. we cannot do that with Multipart. * There are a lot of good transport that already implement message framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own framing it was not easy to add them. * Multipart, router usage (routing id frame) and not being thread safe make the learning curve of zeromq hard to beginners. Without three of them zeromq become much simpler. * At least with NetMQ single part message is much faster than multipart. * New stacks, multipart is complicated to implement, with the new API it will much more easier to implement new stacks (like JSMQ) or any native stack. Pieter I'm looking forward to see how you expose the routing id in the czmq API. I also like the czmq API for sending mutlipart messages (the picture feature) so maybe we can use that to generate single frame which is also compatible with zproto. About the implementation, none of new sockets support any option now. server behave like mandatory router, so when router is not reachable or highwater mark is reached an error will be returned. As ZMTP 3.1 is still in raw status, what do you think of removing the multipart from it? maybe the 3.1 will only support the new socket types. Anyway I really excited about this change. On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com wrote: What we really want IMO is per-peer metadata, and an API to get/set this. Using messages is a hack. Currently working on that :) Having two layers that both carry per-message data is... wrong IMO. Protocols supporting 'out of band' data aren't exactly uncommon.
Re: [zeromq-dev] Notes from a hackathon
Hello everyone, I have read this thread and the blog post from Pieter. I am looking forward to some of those things, but I have some reserve about other points. I'll try to explain what I think, and why. I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client to set routing information is a bit counterintuitive. Also there is risk for impersonation. If a peer disconnect, an other one could steal it's identity. I believe that relying on the socket identity to identify peer is not a robust enough approach, even when using CURVE. Storing the peer routing id in the message's properties seems like a good idea. I think thread-safe socket would be cool. As in great to have in some particular situation, but most of the case it shouldn't be needed. Aiming for a cleaner and simpler ZMQ internal code is good. We must be careful however that this will not make the end-user's life harder. I believe I see 2 points that would make my life harder as a ZMQ user today if we had what we is discussed here: + The removal (or depreciation) of the REQ-REP socket. While fragile over the network, I find them very useful for sending synchronous messages between threads. + Removal / depreciation of multipart messages. Read below. The next paragraph is about multipart messages. Let me be honest and state that I have no idea of the real performance penalty induced by multipart messages. What I want to defend here is their usefulness. I *really* like multipart messages. To be honest, this is one of the many features that made me try ZMQ. This is also the feature that helps me most: I my current project, I mainly pass messages around between thread, so I have less use for other awesome features like auto reconnect or security. IMO framing is very good because it allows you to send (and receive) various kind of data *easily*. I like doing this kind of stuff: zmqpp::message msg; msg HELLO WORLD 42 13; s.send(msg); And on the receiving end: std::string cmd; int value_1, value_2; msg cmd value_1 value_2; Sure, there is no verification of the message content, the application would even crash if a message was badly formatted. However, this is not a concern when messages are flowing on inproc:// : I know that I send and I know what to receive. I believe that the picture, which is great in CZMQ, is not really appropriate for C++ and maybe other language. My point is (for c++) that stream operator are a more apprioriate / natural way of doing what the picture does. ZProto is mentioned as a possible replacement to framing. Disclaimer: I've taken a look at zproto, but I have never used it. I think it's a great project with some strong advantages and real world use case. However, it comes at the cost of more complexity. This is what I meant when talking about making the end-user's life harder. Do I want to use zproto to design complex, client-server protocol? Yes, I probably should give it a try. Do I really want to use zproto when sending 3-4 frames on an inproc socket? I'd rather avoid it and keep it as simple as possible. In my opinion, maintaining some kind of multi-frame abstraction at the API level is in fact very useful to applications, even if they are sent atomically on the implementation level as you describe. I totally agree with Joe on this point. One last thing about multithreaded socket. Maybe this is completely stupid as I know little about libzmq's internal, but wouldn't it be possible to make explicit thread safe socket by providing a zmq_socket_lock()/unlock() function that would allow ZMQ internal to serialize multiple frames properly ? Sure, this is a not as classy as properly thread-safe socket, but would work in most case: grab the socket from shared memory, lock it, write/receive, unlock it. On Tue, Feb 3, 2015 at 3:52 PM, Luna Duclos luna.duc...@palmstonegames.com wrote: I'm not sure if I like having picture as a libzmq level thing. It really only makes sense for C language bindings in my book. most other bindings use their own serialization mechanisms. On Tue, Feb 3, 2015 at 2:06 PM, Pieter Hintjens p...@imatix.com wrote: On Tue, Feb 3, 2015 at 1:01 PM, Doron Somech somdo...@gmail.com wrote: Pieter - regarding the blog post, I'm not sure about setting the routing id on the socket Indeed, that isn't atomic with sending and so can't work with threadsafe sockets. I've changed the article. I think we can either get the routing id as a message property, and/or add a reply to message semantic that does this implicitly. The second is easier for simple servers, the first is better for realistic servers. Since it only applies to server sockets, I think we would benefit from a zserver_reply () method (or similar). We might also think of moving the picture send/recv API into libzmq at some point since it's the immediate replacement for framing. I also have a question regarding the session management (really liked it), but how
[zeromq-dev] meaning of ZMQ_EVENT_CONNECTED
Hi, I'm dealing with the pub-sub slow joiner problem. As I understand it, the problem arises, in part, because connections are asynchronous and zmq_connect() returns before the connection is actually established. I hooked up a monitor and wait until the monitor indicates the socket is in the CONNECTED state before publishing. However, even with the socket in the CONNECTED state, the first few messages are usually lost. Does ZMQ_EVENT_CONNECTED actually mean that the connection is fully established and ready for pubs? Or something lesser? Thanks, Scott P.S. Sorry if I'm just re-asking this question: http://grokbase.com/t/zeromq/zeromq-dev/151enr4dae/zmq-event-connected-and- router-socket. Not sure if pub-sub is sufficiently different than router-router change the meaning of ZMQ_EVENT_CONNECTED. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
I happily use multipart rather a lot, to the point that I recommend that pyzmq users exclusively use send/recv_multipart, and never single-frame send/recv. I think it enables a really nice API. I find it very useful to be able to send a large buffer plus a header of message information without having to copy both blocks into a new buffer before handing off to zeromq. I find this particularly valuable in Python, where building that new buffer can be quite expensive. In Python, and I suspect other higher level wrappers, where most messages are bytes or buffer objects (not wrapped in Message APIs), moving the routing information to message properties is not really an improvement, because I'm not aware of any pyzmq users accessing message properties, so the standard way to use pyzmq is to discard the `zmq_msg_t` object and only deal with bytes. But that's more of an API question for pyzmq than libzmq. As far as I can tell, I also use the multihop case that Pieter's blog post suggests does not seem to happen in practice every day, so maybe my use of zeromq is not typical. From the never break things stance of libzmq and the discussion above, it seems that there's no plan for multipart to ever be removed. Deprecated even seems like too strong a word for a useful feature that will continue to be supported indefinitely. Or is anyone proposing the eventual removal of multipart? -MinRK On Tue, Feb 3, 2015 at 9:37 AM, Arnaud Kapp kapp.a...@gmail.com wrote: Hello everyone, I have read this thread and the blog post from Pieter. I am looking forward to some of those things, but I have some reserve about other points. I'll try to explain what I think, and why. I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client to set routing information is a bit counterintuitive. Also there is risk for impersonation. If a peer disconnect, an other one could steal it's identity. I believe that relying on the socket identity to identify peer is not a robust enough approach, even when using CURVE. Storing the peer routing id in the message's properties seems like a good idea. I think thread-safe socket would be cool. As in great to have in some particular situation, but most of the case it shouldn't be needed. Aiming for a cleaner and simpler ZMQ internal code is good. We must be careful however that this will not make the end-user's life harder. I believe I see 2 points that would make my life harder as a ZMQ user today if we had what we is discussed here: + The removal (or depreciation) of the REQ-REP socket. While fragile over the network, I find them very useful for sending synchronous messages between threads. + Removal / depreciation of multipart messages. Read below. The next paragraph is about multipart messages. Let me be honest and state that I have no idea of the real performance penalty induced by multipart messages. What I want to defend here is their usefulness. I *really* like multipart messages. To be honest, this is one of the many features that made me try ZMQ. This is also the feature that helps me most: I my current project, I mainly pass messages around between thread, so I have less use for other awesome features like auto reconnect or security. IMO framing is very good because it allows you to send (and receive) various kind of data *easily*. I like doing this kind of stuff: zmqpp::message msg; msg HELLO WORLD 42 13; s.send(msg); And on the receiving end: std::string cmd; int value_1, value_2; msg cmd value_1 value_2; Sure, there is no verification of the message content, the application would even crash if a message was badly formatted. However, this is not a concern when messages are flowing on inproc:// : I know that I send and I know what to receive. I believe that the picture, which is great in CZMQ, is not really appropriate for C++ and maybe other language. My point is (for c++) that stream operator are a more apprioriate / natural way of doing what the picture does. ZProto is mentioned as a possible replacement to framing. Disclaimer: I've taken a look at zproto, but I have never used it. I think it's a great project with some strong advantages and real world use case. However, it comes at the cost of more complexity. This is what I meant when talking about making the end-user's life harder. Do I want to use zproto to design complex, client-server protocol? Yes, I probably should give it a try. Do I really want to use zproto when sending 3-4 frames on an inproc socket? I'd rather avoid it and keep it as simple as possible. In my opinion, maintaining some kind of multi-frame abstraction at the API level is in fact very useful to applications, even if they are sent atomically on the implementation level as you describe. I totally agree with Joe on this point. One last thing about multithreaded socket. Maybe this is completely stupid as I know little about libzmq's internal,
[zeromq-dev] XPUB connect socket doesn't receive unsubscribe when peer disconnects
Seems to only happen if the XPUB connects rather than binds. Is this intended behavior? The problem is that if I restart the process on the other side (with a SUB socket), then XPUB never clears the subscription, and so the publisher could accumulate lots of old subscriptions over time. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] meaning of ZMQ_EVENT_CONNECTED
Hello, I believe it's not enough. Well, your tests seem to prove it's not enough. My guess would be that the socket is indeed connected, but the subscriptions have not yet been sent (or acknowledged by the publisher). I may be wrong here, I don't know much about the ZMTP protocol. What you could do this (please wait for more experienced ZMQer's to confirm or not): Add a router socket on the publisher's side. Connect to this second with a dealer from the subscriber's side. Send some kind of PING message from the publisher (like every second, or w/e suits you). Connect your subscriber. When your subscriber receive a PING message, then send an I AM READY message to the router socket, using your dealer socket. When receiving the I AM READY message on the router socket, your publisher can start sending real message to the subscriber. This looks a bit daunting. Depending on what you're doing, it might be preferable to simply add a sleep() before sending the first message on the publisher. On Tue, Feb 3, 2015 at 8:35 PM, McMillan, Scott A scott.a.mcmil...@intel.com wrote: Hi, I'm dealing with the pub-sub slow joiner problem. As I understand it, the problem arises, in part, because connections are asynchronous and zmq_connect() returns before the connection is actually established. I hooked up a monitor and wait until the monitor indicates the socket is in the CONNECTED state before publishing. However, even with the socket in the CONNECTED state, the first few messages are usually lost. Does ZMQ_EVENT_CONNECTED actually mean that the connection is fully established and ready for pubs? Or something lesser? Thanks, Scott P.S. Sorry if I'm just re-asking this question: http://grokbase.com/t/zeromq/zeromq-dev/151enr4dae/zmq-event-connected-and- router-socket. Not sure if pub-sub is sufficiently different than router-router change the meaning of ZMQ_EVENT_CONNECTED. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- Kapp Arnaud - Xaqq ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
AK So *maybe* a documentation effort is what is needed to better AK explain those thing. The guide explains things in great detail, and an engaging manner. How do you improve on it, aside from removing the things that need explaining? -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
On Tue, Feb 3, 2015 at 11:27 PM, Thomas Rodgers rodg...@twrodgers.com wrote: * Sessions require protocol commands that are invisible to normal use. That means either protocol support (e.g. command frames) or some layering on top of ZeroMQ blobs. For sure, command frames are designed for this and that's what we'd use if we implemented sessions in libzmq. If we implemented them above libzmq, e.g. in CZMQ/zserver, zclient then we'd need some other layering. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
I like the idea of using ZAP as a test case. It's using multipart now and that does indeed work well. On Wed, Feb 4, 2015 at 12:35 AM, Gregg Irwin gr...@pointillistic.com wrote: AK So *maybe* a documentation effort is what is needed to better AK explain those thing. The guide explains things in great detail, and an engaging manner. How do you improve on it, aside from removing the things that need explaining? -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
M Perhaps it is because I spend my days in a higher level language M like Python, but zproto is not an attractive option. Same here. I will read in detail about it shortly, but it may not make it into my toolbox as a multipart replacement. Multipart looked very cool when I found 0MQ, but I've ended up not using it much. I'm not doing high performance stuff though. Simplicity and ease of use are tops on my list. -- Gregg ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
@Doron: Yes I like multi part because its a very simple way build multiple-fields message. I once considered using protobuff + raw socket for a pet project of mine, and I ended using ZMQ framing because it was simpler. The problem if we pick an other serialization solution is that it needs to built-in libzmq. Per-binding serialization would be a disaster IMO. Arnaud - you can do that today, just use synchronization object, however this is not really work, what if you call receive and there is no message currently? so you block and the socket is still locked, maybe for long time. Now you want to send from another thread but socket is locked. Ah yes that's true, you'd have to use non-blocking read and such. I reckon this is clearly not as practical as a fully thread safe socket would be. Multi-part does easy serialization, yes. Not in C though. We've learned how to do variable list serialization with zproto-compatible picture sending in CZMQ. Is this a sufficient alternative? I think so. zsock_send() was essential to making actors cheap enough to use widely. egrep zsock_send *.c in malamute-core/src. I'd argue that using zmsg_addstr() / zmsg_add() / zmsg_pop() are fine for serialization. Agreed, when using the raw libzmq API this is not as nice. If you're doing multipart a lot in an application, you can replace that with zproto. You already have contracts then, why not make them explicit and documented? That's always a win in my experience. Yes I'm using multi-part quite a lot. I manually documented those contracts and they are enforced by assert() (for anything that does'nt come from the external world, obviously). I'd argue that using c++ stream operator is easier, but I'd really have to convert part of my code to zproto and give it an honest try. Picture sending solves a cross-language problem, namely how to construct zproto-style commands without using dedicated codecs. While the API is C-ish, the functionality is not. I'd expect that to be wrapped in bindings. Yes but I believe pictures are sometime not practical. (I think) they require that the picture be known when the message is being created. I have some code where I do the following (if it's that weird / bad please tell me): Some code create a message, push the identity frame and then some application-level header. Control flow is handed to some other function, along with a reference to the message. This function push a command frame and more data (type and how many are unkown to the previous piece of code). Control flow return to initial code, and send the message. On the other side of the socket, I do something similar: Pop application-level header. Pop the command frame. Based on those 2 piece of data, call a function (again, along with a reference to the message). Function will expect some kind of data layout, will extract it and perform work. In this scenario, the first part of the code doesn't know (or care, really) about the real content of the message. In case my explanation sucked, this is a bit similar to the ZAP rfc: the request contains some must be there frames, and then some zero or more opaque frames. In case of ZAP, those opaque frames only make sense to mechanism specific code. In my example, they only make sense to some specific message handling code. I do not really see how ZAP could be that simple and nice to work w/ if it wasn't for multi-part messages. The main goal with this is simplicity and accuracy. I'm tired of teaching people ZeroMQ constructs that are clearly over-designed and disconnected from reality in subtle ways. Each apology for friction (sorry, but don't use REQ-REP, they are kind of weird, sorry, IDENTITY is not really identity, sorry, message sometimes means frame and sometimes message) really signals costs up and down the stacks. This is a valid point. However, a good read through the guide really help understanding how things work. I wouldn't be that surprised if a lot of people having trouble understanding frames vs message or REQ-REP problem didn't really read the zguide. So *maybe* a documentation effort is what is needed to better explain those thing. I agree about the IDENTITY tho. I do not know, I don't have much experience teaching, and none teaching ZMQ. Anyway, no matter what it's going to be interesting. On Tue, Feb 3, 2015 at 11:04 PM, Pieter Hintjens p...@imatix.com wrote: Thanks for the questions. I'd try to flatter by saying how great these questions are, except I know you're all way too smart to fall for that. :) Let me answer a bunch of questions in one go: * Sorry for not posting this here. I've written a more detailed analysis of the motivation for this thread (and experiments): http://hintjens.com/blog:84. * The main goal with this is simplicity and accuracy. I'm tired of teaching people ZeroMQ constructs that are clearly over-designed and disconnected from reality in subtle ways. Each apology for friction (sorry,
Re: [zeromq-dev] zmqstream with req socket only sends the first message
On Tue, Feb 3, 2015 at 7:00 AM, drebs dr...@riseup.net wrote: Thanks a lot for your answer, now it works like a charm. I couldn't find documentation about this, do you think it's a good idea to add a paragraph to zmqstream.send docstring (or anywhere else) about threaded use? If you think it's a good idea, I have a suggestion based on the info you gave me. It's in the patch below, but feel free to modify in the way you think it is more appropriate. Since this is an issue generic to any use of tornado IOLoop, I would be a bit more brief, and link to the tornado docs http://tornado.readthedocs.org/en/latest/ioloop.html#tornado.ioloop.IOLoop.add_callback . -MinRK -- From cf9418dc14b2dc2e1aca5be1b79a105982dcd877 Mon Sep 17 00:00:00 2001 From: drebs dr...@riseup.net Date: Tue, 3 Feb 2015 12:50:59 -0200 Subject: [PATCH] Add doc for threaded zmqstream.send. --- zmq/eventloop/zmqstream.py |8 1 file changed, 8 insertions(+) diff --git a/zmq/eventloop/zmqstream.py b/zmq/eventloop/zmqstream.py index 86a97e4..59149ec 100644 --- a/zmq/eventloop/zmqstream.py +++ b/zmq/eventloop/zmqstream.py @@ -242,6 +242,14 @@ class ZMQStream(object): def send(self, msg, flags=0, copy=True, track=False, callback=None): Send a message, optionally also register a new callback for sends. See zmq.socket.send for details. + +When using threads, calling send on the zmqstream from another thread +doesn’t properly tell the IOLoop thread that there’s an event to +process. This could cause small delays if the IOLoop is already +processing lots of events, but it can also cause the message to never +be sent if the zmq socket is the only one it’s handling. In this case, +you should hand off the stream.send to the IOLoop’s thread via +IOLoop.add_callback. return self.send_multipart([msg], flags=flags, copy=copy, track=track, callback=callback) A -- 1.7.10.4 -- MinRK codificou 9.4K bytes: Calling send on the zmqstream from another thread doesn’t properly tell the IOLoop thread that there’s an event to process. This could call small delays if the IOLoop is already processing lots of events, but it can cause the message to never send if the zmq socket is the only one it’s handling. You want your ZmqREQConnection.send to hand off the stream.send to the IOLoop’s thread via IOLoop.add_callback: def send(self, *args, **kwargs): print(Sending message to backend: (%s, %s) % (args, kwargs)) self._ioloop.add_callback(lambda : self._stream.send(*args, **kwargs)) -MinRK On Mon, Feb 2, 2015 at 12:58 PM, drebs [1]dr...@riseup.net wrote: Hello, zmq community. I’m trying to use a pyzmq’s zmqstream with a REQ socket to talk to a txzmq REP socket, but only the first message is ever sent and received back. REQ pyzmq endpoint code: [2]http://paste.debian.net/143615/ REP txzmq endpoint code: [3]http://paste.debian.net/143617/ I would expect that all the messages would be sent. Am I missing something here? If I let some requests accumulate, and then start the server, all the accumulated requests are replied, but the new ones are not sent. If I replace the REQ endpoint by a pure pyzmq version with no zmqstream or ioloop, it works fine ([4]http://paste.debian.net/143619/). Anyone can give some light on this? Thanks a lot! drebs. — -- zeromq-dev mailing list [5]zeromq-dev@lists.zeromq.org [6]http://lists.zeromq.org/mailman/listinfo/zeromq-dev References Visible links 1. http://mailto:dr...@riseup.net/ 2. http://paste.debian.net/143615/ 3. http://paste.debian.net/143617/ 4. http://paste.debian.net/143619/ 5. mailto:zeromq-dev@lists.zeromq.org 6. http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
Hi, Please excuse for writing this on my phone I do not see any usecase nor detailed motivation what the problem is that causes this somewhat fundamental proposal. I think this approach is somewhat surprising and against the C4 thing. Not sure if this is a problem or not, but there is a lesson there somehow. Could you please (re-)publish the problem descriptions? Is the problem in czmq , netmq, go-bindings or really in libzmq? Apart from this 'style' related thing, i have a technical question: When a router (API4.x)received a curve-encrypted message, i thought that the identity field was the indented way to identify which of the many possible clients send the message. In fact I was thinking of signing the identity field using the client secret key to have a robuster way of stopping clients which woul like to steal an identity. So with an 'int' as ID how is it possible to distinguish client1 from client2 cryptografically? Or did I misunderstand how this should have been done in API4.x? And is this possible with api5.x? kind regards Frank -- unterwegs Am 02.02.2015 um 10:23 schrieb Pieter Hintjens p...@imatix.com: Hi folks, We had an interesting pre-FOSDEM hackathon on Thursday and Friday, in Brussels. One of the threads that came out, thanks mainly to Doron Somech (NetMQ) was a strategy for simplifying ZeroMQ. This started as a discussion of Nanomsg's dropping of multipart messages. Overall, multipart messages add a lot of complexity and confusion to the library and bindings. In case we forget: - frame vs, message vs. part - routing id frames - request-reply envelopes - router sockets - identities Multipart messages are the main reason we can't make threadsafe sockets. So, we're going to experiment with shifting ZeroMQ (libzmq, NetMQ, CZMQ, and other bindings) in this direction: - deprecate multipart messages from the API - when we need framing, we can use zproto - deprecate ROUTER and DEALER and slowly replace with SERVER / CLIENT sockets that refuse multipart data - deprecate REQ and REP - SERVER has get/set routing ID on message - routing ID is an integer - routing ID cannot be set by peer, so we deprecate ZMQ_IDENTITY - start to aim for threadsafe sockets - deprecate the ability to build request-reply chains ... This is not a roadmap, nor is this a proposal, it's just a realization by several of us that multipart messages create complexity, which we dislike, and which causes cost and irritation. Cheers Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev smime.p7s Description: S/MIME cryptographic signature ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
The single biggest problem I see people having in the JVM community is their lack of understanding of the concurrency model. I haven't had time to internalize the proposal just yet but if it helps people write better/simpler software, I'm all for it. -Trev On Tue, Feb 3, 2015 at 4:33 PM, frank sound...@gmx.net wrote: Hi, Please excuse for writing this on my phone I do not see any usecase nor detailed motivation what the problem is that causes this somewhat fundamental proposal. I think this approach is somewhat surprising and against the C4 thing. Not sure if this is a problem or not, but there is a lesson there somehow. Could you please (re-)publish the problem descriptions? Is the problem in czmq , netmq, go-bindings or really in libzmq? Apart from this 'style' related thing, i have a technical question: When a router (API4.x)received a curve-encrypted message, i thought that the identity field was the indented way to identify which of the many possible clients send the message. In fact I was thinking of signing the identity field using the client secret key to have a robuster way of stopping clients which woul like to steal an identity. So with an 'int' as ID how is it possible to distinguish client1 from client2 cryptografically? Or did I misunderstand how this should have been done in API4.x? And is this possible with api5.x? kind regards Frank -- unterwegs Am 02.02.2015 um 10:23 schrieb Pieter Hintjens p...@imatix.com: Hi folks, We had an interesting pre-FOSDEM hackathon on Thursday and Friday, in Brussels. One of the threads that came out, thanks mainly to Doron Somech (NetMQ) was a strategy for simplifying ZeroMQ. This started as a discussion of Nanomsg's dropping of multipart messages. Overall, multipart messages add a lot of complexity and confusion to the library and bindings. In case we forget: - frame vs, message vs. part - routing id frames - request-reply envelopes - router sockets - identities Multipart messages are the main reason we can't make threadsafe sockets. So, we're going to experiment with shifting ZeroMQ (libzmq, NetMQ, CZMQ, and other bindings) in this direction: - deprecate multipart messages from the API - when we need framing, we can use zproto - deprecate ROUTER and DEALER and slowly replace with SERVER / CLIENT sockets that refuse multipart data - deprecate REQ and REP - SERVER has get/set routing ID on message - routing ID is an integer - routing ID cannot be set by peer, so we deprecate ZMQ_IDENTITY - start to aim for threadsafe sockets - deprecate the ability to build request-reply chains ... This is not a roadmap, nor is this a proposal, it's just a realization by several of us that multipart messages create complexity, which we dislike, and which causes cost and irritation. Cheers Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Notes from a hackathon
Can I ask why people think thread safety around sockets is in any way important? or even beneficial at all if it impacts performance negatively? Most libraries I've found aren't thread safe because it's cheaper to be that way and let the application handle any concurrency in a sensible way for whatever they are trying to solve. On 2 February 2015 at 20:36, Thomas Rodgers rodg...@twrodgers.com wrote: My eventual suggestion was to look at using metadata for this, it is already there and is per-connection. But there are (currently) a few problems with using it this way and I wanted to think those through a bit before making the suggestion. I am not sure why file_desc was placed where it is. On Mon, Feb 2, 2015 at 2:25 PM, Doron Somech somdo...@gmail.com wrote: Currently the router_id is in the last 4 bytes of the 64 bytes message size, could this cause alignment issue? Regarding the idea to combine it with the file descriptor, I think it can cause portability issues in the future, but it might be platform specific. Why wasn't it originally in end of the union which can solve the alignment issue? Anyway in the future we might not need it, because we can query the socket with the routing id to get metadata. On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com wrote: Some observations on adding a discrete router_id field to the message - Depending on alignment, this reduces the size of a VSM message by 4 or 8 bytes. Does this field really need to be a member of the union? Currently msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values (even on Windows, where the FD is 64 bits, it will never give values 2^32). Could we not just pack these two together, and make use of the currently unused bits? It gets rid of the combinatorial expansion of adding fields to each member of the union (at least for this case). On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote: Brian - The issue is that with the new api and to enable thread safety the routing id is part of the message. So czmq new api should somehow expose routing id on the message and/or part of the send message. On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote: Doron - I did a test implementation around this idea in goczmq today and I'm liking the semantics. I wrote an interface that conforms to io.Reader and io.Writer where you pass a call a byte buffer and you get the message placed into it. When the socket is a router socket, it silently drops the ID frame under the hood, but sets a lastClientID in the socket struct, that if you need, you can get with a GetLastClientID call. For reference: https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note strings in go are just immutable byte arrays) https://github.com/zeromq/goczmq/blob/master/sock.go#L333 https://github.com/zeromq/goczmq/blob/master/sock.go#L366 So if the socket is a router, the Write call just transparently uses the lastClientID value. In a one to one request / reply, you then don't need to think about it at all, and if you need to do async work, you can get and set the id as needed. I'll just move this to use the new API later - just wanted to try it out and see how I liked it, and I say thumbs up to dropping frames on the new Server / Client sockets. On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote: So the new sockets are in the github master, you can take a loot at the test to see how to use the new routing id field: https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp Few of the reasons I didn't like multi frames: * I tried in the past to make both zeromq and NetMQ thread safe, think of sending from multiple threads or receiving from multiple threads. we cannot do that with Multipart. * There are a lot of good transport that already implement message framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own framing it was not easy to add them. * Multipart, router usage (routing id frame) and not being thread safe make the learning curve of zeromq hard to beginners. Without three of them zeromq become much simpler. * At least with NetMQ single part message is much faster than multipart. * New stacks, multipart is complicated to implement, with the new API it will much more easier to implement new stacks (like JSMQ) or any native stack. Pieter I'm looking forward to see how you expose the routing id in the czmq API. I also like the czmq API for sending mutlipart messages (the picture feature) so maybe we can use that to generate single frame which is also compatible with zproto. About the implementation, none of new sockets support any option now. server behave like mandatory router, so when router is not reachable or highwater mark is reached an error will be returned. As ZMTP 3.1 is still in raw status, what do you think of removing
Re: [zeromq-dev] Notes from a hackathon
I've written up my understanding of this thread: http://hintjens.com/blog:84 Thread safety would solve the perennial problem of sending data via ZeroMQ, from short-lived threads. Performance in many cases would go up, not down. We see way too many apps that create/connect a socket for every message they send. Apart from that I agree that thread safety is a weird thing and only worth doing in very specific cases (ZeroMQ contexts and sockets seem to be good candidates). On Tue, Feb 3, 2015 at 10:52 AM, Ben Gray b...@benjamg.com wrote: Can I ask why people think thread safety around sockets is in any way important? or even beneficial at all if it impacts performance negatively? Most libraries I've found aren't thread safe because it's cheaper to be that way and let the application handle any concurrency in a sensible way for whatever they are trying to solve. On 2 February 2015 at 20:36, Thomas Rodgers rodg...@twrodgers.com wrote: My eventual suggestion was to look at using metadata for this, it is already there and is per-connection. But there are (currently) a few problems with using it this way and I wanted to think those through a bit before making the suggestion. I am not sure why file_desc was placed where it is. On Mon, Feb 2, 2015 at 2:25 PM, Doron Somech somdo...@gmail.com wrote: Currently the router_id is in the last 4 bytes of the 64 bytes message size, could this cause alignment issue? Regarding the idea to combine it with the file descriptor, I think it can cause portability issues in the future, but it might be platform specific. Why wasn't it originally in end of the union which can solve the alignment issue? Anyway in the future we might not need it, because we can query the socket with the routing id to get metadata. On Mon, Feb 2, 2015 at 10:02 PM, Thomas Rodgers rodg...@twrodgers.com wrote: Some observations on adding a discrete router_id field to the message - Depending on alignment, this reduces the size of a VSM message by 4 or 8 bytes. Does this field really need to be a member of the union? Currently msg_t::file_desc is 64 bit to force alignment, but FDs are 32bit values (even on Windows, where the FD is 64 bits, it will never give values 2^32). Could we not just pack these two together, and make use of the currently unused bits? It gets rid of the combinatorial expansion of adding fields to each member of the union (at least for this case). On Mon, Feb 2, 2015 at 1:06 PM, Doron Somech somdo...@gmail.com wrote: Brian - The issue is that with the new api and to enable thread safety the routing id is part of the message. So czmq new api should somehow expose routing id on the message and/or part of the send message. On Feb 2, 2015 7:39 PM, Brian Knox bk...@digitalocean.com wrote: Doron - I did a test implementation around this idea in goczmq today and I'm liking the semantics. I wrote an interface that conforms to io.Reader and io.Writer where you pass a call a byte buffer and you get the message placed into it. When the socket is a router socket, it silently drops the ID frame under the hood, but sets a lastClientID in the socket struct, that if you need, you can get with a GetLastClientID call. For reference: https://github.com/zeromq/goczmq/blob/master/sock.go#L35 (note strings in go are just immutable byte arrays) https://github.com/zeromq/goczmq/blob/master/sock.go#L333 https://github.com/zeromq/goczmq/blob/master/sock.go#L366 So if the socket is a router, the Write call just transparently uses the lastClientID value. In a one to one request / reply, you then don't need to think about it at all, and if you need to do async work, you can get and set the id as needed. I'll just move this to use the new API later - just wanted to try it out and see how I liked it, and I say thumbs up to dropping frames on the new Server / Client sockets. On Mon, Feb 2, 2015 at 4:37 PM, Doron Somech somdo...@gmail.com wrote: So the new sockets are in the github master, you can take a loot at the test to see how to use the new routing id field: https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp Few of the reasons I didn't like multi frames: * I tried in the past to make both zeromq and NetMQ thread safe, think of sending from multiple threads or receiving from multiple threads. we cannot do that with Multipart. * There are a lot of good transport that already implement message framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own framing it was not easy to add them. * Multipart, router usage (routing id frame) and not being thread safe make the learning curve of zeromq hard to beginners. Without three of them zeromq become much simpler. * At least with NetMQ single part message is much faster than multipart. * New stacks, multipart is complicated to implement, with the new API it will much more easier to implement new
Re: [zeromq-dev] Notes from a hackathon
There has been a zmq_sendiov/recviov call for ages, undocumented and rarely used. This would indeed be a good alternative to multipart for the zerocopy use case. On Mon, Feb 2, 2015 at 10:59 PM, Maurice Barnum m...@yahoo-inc.com wrote: Is there any thought to adding a writev() analogue like zmq_msg_sendv()? One of the reasons I send send multiple frames is to avoid stitching together buffers just to call send, which something like writev() would address that use without the complexity of the socket buffering the parts. On Monday, February 2, 2015 8:36 AM, Joe McIlvain joe.eli@gmail.com wrote: In my opinion, maintaining some kind of multi-frame abstraction at the API level is in fact very useful to applications, even if they are sent atomically on the implementation level as you describe. For example, in CZMQ It's quite useful in program flow to be able to push or pop a single zframe onto or off of a zmsg, then send that zmsg as an entire message. Rarely do I find myself using zframe_send or zframe_recv with the send more and receive more flags in a CZMQ application. The CZMQ picture messages are nice, but they are not always the easiest API to integrate into an application. Having zmsg and zframe remain (or something like them), while removing zframe_send and zframe_recv could ease applications relying on multi-frame semantics into the transition. The implementation for zmsg would simply serialize the frames in some zmsg-determined way into a single-part zmq_msg to send and deserialize according to the same scheme to receive. This multi-frame serialization essentially be at the application level rather than at the ZMTP level. In short, I think a multi-frame abstraction is important for applications, but if we remove the ability to send one frame at a time at the fundamental level, this abstraction can remain at a higher level (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with the proposed changes. On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech somdo...@gmail.com wrote: So the new sockets are in the github master, you can take a loot at the test to see how to use the new routing id field: https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp Few of the reasons I didn't like multi frames: * I tried in the past to make both zeromq and NetMQ thread safe, think of sending from multiple threads or receiving from multiple threads. we cannot do that with Multipart. * There are a lot of good transport that already implement message framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required it is own framing it was not easy to add them. * Multipart, router usage (routing id frame) and not being thread safe make the learning curve of zeromq hard to beginners. Without three of them zeromq become much simpler. * At least with NetMQ single part message is much faster than multipart. * New stacks, multipart is complicated to implement, with the new API it will much more easier to implement new stacks (like JSMQ) or any native stack. Pieter I'm looking forward to see how you expose the routing id in the czmq API. I also like the czmq API for sending mutlipart messages (the picture feature) so maybe we can use that to generate single frame which is also compatible with zproto. About the implementation, none of new sockets support any option now. server behave like mandatory router, so when router is not reachable or highwater mark is reached an error will be returned. As ZMTP 3.1 is still in raw status, what do you think of removing the multipart from it? maybe the 3.1 will only support the new socket types. Anyway I really excited about this change. On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers rodg...@twrodgers.com wrote: What we really want IMO is per-peer metadata, and an API to get/set this. Using messages is a hack. Currently working on that :) Having two layers that both carry per-message data is... wrong IMO. Protocols supporting 'out of band' data aren't exactly uncommon. However the key thing is, what's the problem. Then we can discuss solutions to that. I don't have an immediate use-case justifying it. I noted it, mostly because it has come up a few times since I started paying attention. On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens p...@imatix.com wrote: It seems to me that in order to remove multi-part messages and introduce new socket types (e.g. SERVER/CLIENT) that would necessitate a revision of the wire protocol. If we are going to do that, it might be worth considering per-message metadata - We'd have to be very clear about the problem that per-message metadata is aiming for. There is an elegance to delivering blobs and nothing more. Metadata can be added on top using zproto. What we really want IMO is per-peer metadata, and an API to get/set this. Using messages is a hack. If we are sending/receiving data on
Re: [zeromq-dev] Notes from a hackathon
The thing is that you like multi-part because it simple serialization tool and not because mutli access to socket, I think that removing mutli-part doesn't necessary mean removing serialization solution. For java and .Net we can use fluent api for sending: Message.New().AddString(Hello).AddString(World).Send(socket); The advantage of this kind of API is that we can calculate the needed buffer in advance (only the send will serialize everything). Receiving can be very close to the current API, but you don't really have frames, you can only pop. Any serialization solution we will pick should in my opinion be compatible with ZProto. Arnaud - you can do that today, just use synchronization object, however this is not really work, what if you call receive and there is no message currently? so you block and the socket is still locked, maybe for long time. Now you want to send from another thread but socket is locked. MinRK - the new CLIENT and SERVER probably won't support multi-part (and maybe keep-alive and heartbeat). On Tue, Feb 3, 2015 at 10:00 PM, MinRK benjami...@gmail.com wrote: I happily use multipart rather a lot, to the point that I recommend that pyzmq users exclusively use send/recv_multipart, and never single-frame send/recv. I think it enables a really nice API. I find it very useful to be able to send a large buffer plus a header of message information without having to copy both blocks into a new buffer before handing off to zeromq. I find this particularly valuable in Python, where building that new buffer can be quite expensive. In Python, and I suspect other higher level wrappers, where most messages are bytes or buffer objects (not wrapped in Message APIs), moving the routing information to message properties is not really an improvement, because I'm not aware of any pyzmq users accessing message properties, so the standard way to use pyzmq is to discard the `zmq_msg_t` object and only deal with bytes. But that's more of an API question for pyzmq than libzmq. As far as I can tell, I also use the multihop case that Pieter's blog post suggests does not seem to happen in practice every day, so maybe my use of zeromq is not typical. From the never break things stance of libzmq and the discussion above, it seems that there's no plan for multipart to ever be removed. Deprecated even seems like too strong a word for a useful feature that will continue to be supported indefinitely. Or is anyone proposing the eventual removal of multipart? -MinRK On Tue, Feb 3, 2015 at 9:37 AM, Arnaud Kapp kapp.a...@gmail.com wrote: Hello everyone, I have read this thread and the blog post from Pieter. I am looking forward to some of those things, but I have some reserve about other points. I'll try to explain what I think, and why. I agree that ZMQ_IDENTITY is a bit confusing. Allowing remote client to set routing information is a bit counterintuitive. Also there is risk for impersonation. If a peer disconnect, an other one could steal it's identity. I believe that relying on the socket identity to identify peer is not a robust enough approach, even when using CURVE. Storing the peer routing id in the message's properties seems like a good idea. I think thread-safe socket would be cool. As in great to have in some particular situation, but most of the case it shouldn't be needed. Aiming for a cleaner and simpler ZMQ internal code is good. We must be careful however that this will not make the end-user's life harder. I believe I see 2 points that would make my life harder as a ZMQ user today if we had what we is discussed here: + The removal (or depreciation) of the REQ-REP socket. While fragile over the network, I find them very useful for sending synchronous messages between threads. + Removal / depreciation of multipart messages. Read below. The next paragraph is about multipart messages. Let me be honest and state that I have no idea of the real performance penalty induced by multipart messages. What I want to defend here is their usefulness. I *really* like multipart messages. To be honest, this is one of the many features that made me try ZMQ. This is also the feature that helps me most: I my current project, I mainly pass messages around between thread, so I have less use for other awesome features like auto reconnect or security. IMO framing is very good because it allows you to send (and receive) various kind of data *easily*. I like doing this kind of stuff: zmqpp::message msg; msg HELLO WORLD 42 13; s.send(msg); And on the receiving end: std::string cmd; int value_1, value_2; msg cmd value_1 value_2; Sure, there is no verification of the message content, the application would even crash if a message was badly formatted. However, this is not a concern when messages are flowing on inproc:// : I know that I send and I know what to receive. I believe that the picture, which is
