Re: [zeromq-dev] Idea draft : a module system on top of zyre/zbeacon.
On 12/13/2013 12:05 AM, crocket wrote: I'm not good with current module systems because they are difficult to use and maintain. Binding modules from other languages is difficult with FFI(Foreign language interface). Publishing namespace or answering namespace requests via zbeacon would be convenient. Each namespace has a list of methods or even variables. Namespace collisions can be detected via UDP discovery. Module inspection API might answer what serialization protocols they use and serve method signature requests. For now, that's all, and it seems simple and easy enough. Since I'll have been tied to a 9-6 job until Feb 2015 by some serious laws and commute takes 4 hours a day, I may not have time until then. So it's just an idea for now. Can you elaborate a bit more about what you mean? I'm not sure if I can follow you? It sounds like you are suggesting an RPC interface exchange? In regards to zbeacon/zyre I think the headers should be used to exchange capabilities. Arnaud -- w: http://www.sphaero.org t: http://twitter.com/sphaero g: http://github.com/sphaero i: freenode: sphaero_z25 ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Reconnect failed after iptables tricks
hi I was testing simple connection lose scenario on a simple DEALER-ROUTER architecture. D was a client on Windows7, R was a server on VirtualBox. What happened: I started them both, saw chatting between them in logs; after that I disabled a client ip on server host with iptables , ensured that there is no more chatting between peers; after that I had re-enabled IP address of the client on server host; then went to logs expecting to see that chatting began, but .. there was silence. The question is: is it eligible way to test reconnection? or this is a bug in ZMQ core? BR -artemv PS block script: iptables -I INPUT -s IP-of-client -j DROP unlock script: iptables -F ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Does ZMQ handle tcp-RST?
Hi Is there a way to tell ZMQ to handle tcp-RST? If remote peer abruptly goes down OS kernel sends RST to client. Right? So, can ZMQ handle that, and stop collecting messages inside in-mem queue for the remote peer which is down. Thanks in advance. BR -artemv ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] ZeroMQ IPC Windows
Hi, I'm doing some research at Instituto Superior Técnico concerning distributed systems and inter-process communication. Last days I was studying a little bit of ZeroMQ and now want to use it in my project since my goal is to improve performance between the communication of two different apps running on the same machine. I want to know if there is already someone implementing the *ØMQ local inter-process communication transport* using Windows Sockets and I would like to know more about this. As I said my focus is on performance and if needed I could start implementing this functionality. Meanwhile I will use tcp for this communication. Best Regards, Francisco Freire ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZeroMQ IPC Windows
I'm not using windows but if there is anything I could help a fellow colleague from Técnico, in Portuguese, just ask me ;) -- Bruno Rodrigues Sent from my iPhone No dia 13/12/2013, às 12:35, Francisco Freire francisco.m.fre...@gmail.com escreveu: Hi, I'm doing some research at Instituto Superior Técnico concerning distributed systems and inter-process communication. Last days I was studying a little bit of ZeroMQ and now want to use it in my project since my goal is to improve performance between the communication of two different apps running on the same machine. I want to know if there is already someone implementing the *ØMQ local inter-process communication transport* using Windows Sockets and I would like to know more about this. As I said my focus is on performance and if needed I could start implementing this functionality. Meanwhile I will use tcp for this communication. Best Regards, Francisco Freire ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Idea draft : a module system on top of zyre/zbeacon.
I didn't think of any term for my technology. I think it contains 1) a form of RPC interface exchange 2) methods grouped in namespaces. For now, the simplest approach I can think of is a ZeroMQ socket for each namespace. On Fri, Dec 13, 2013 at 8:15 PM, Arnaud Loonstra arn...@sphaero.org wrote: On 12/13/2013 12:05 AM, crocket wrote: I'm not good with current module systems because they are difficult to use and maintain. Binding modules from other languages is difficult with FFI(Foreign language interface). Publishing namespace or answering namespace requests via zbeacon would be convenient. Each namespace has a list of methods or even variables. Namespace collisions can be detected via UDP discovery. Module inspection API might answer what serialization protocols they use and serve method signature requests. For now, that's all, and it seems simple and easy enough. Since I'll have been tied to a 9-6 job until Feb 2015 by some serious laws and commute takes 4 hours a day, I may not have time until then. So it's just an idea for now. Can you elaborate a bit more about what you mean? I'm not sure if I can follow you? It sounds like you are suggesting an RPC interface exchange? In regards to zbeacon/zyre I think the headers should be used to exchange capabilities. Arnaud -- w: http://www.sphaero.org t: http://twitter.com/sphaero g: http://github.com/sphaero i: freenode: sphaero_z25 ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] NetMQ/JeroMQ following ZeroMQ changes
More detail now... :) We do not distinguish features from bugs except in one specific way, which is when backporting changes to stable versions. The change process is (a) identify problem, (b) make improvement. In a true sense, _every_ change can be viewed as fixing a bug, with before/after tests. Please keep this in mind and refrain from adding back notions of bug fix, feature, etc. to the issue tracker... Your real goal is interoperability, not feature completeness. You want a NetMQ app to talk to a libzmq app without concern. You may also want to implement the official API semantics just so NetMQ users can use the same reference manual. And then a higher level API that makes sense for .NET. For interop, my advice is to work by test cases and then cherry pick changes from libzmq only where they make sense. I suspect you will diverge and in time have a totally different code base. Which would be sensible. At all costs, avoid cargo cult patches, where you take code because you believe it will bring the gods back. Either a patch fixes a provable (testable) problem, it it is itself a problem. Lastly, final thing, any workflow or process you adopt for NetMQ should be invisible to libzmq. It has to be a pure hostile fork in that sense. This is to avoid global mutable state. You can figure out what I mean. Good luck. NetMQ is an awesome, fabulous project (as is JeroMQ). Pieter On Fri, Dec 13, 2013 at 4:39 PM, Pieter Hintjens piet...@gmail.com wrote: I'll respond in more detail later, when I can. We don't really do features. All patches should be solving a clear issue. What I'd do is aim for protocol interop and pull in changes which go that way. It is the only real criteria. Also, use test cases written in native libzmq to check netmq. On Dec 12, 2013 7:24 PM, Doron Somech somdo...@gmail.com wrote: Hi All, As a maintainer of NetMQ is pretty hard to follow the ZeroMQ changes. I'm trying to create a plan to catch up with the ZeroMQ features but it's hard to know which features belong to which version and which pull requests belong to each feature. Now that the ZeroMQ issue tracker is on github I want to suggest few ideas to make it easier to follow ZeroMQ changes: * Differentiate between feature and a bug (maybe using convention or label) * Relate each pull request to an issue (using a comment with # followed by the issue number and a convention in the Pull Request name). multiple pull requests can be related to an issue. (IMO if a pull request is fixing a bug of a feature that was not yet released the pull request need to be relate to the feature and not to open a bug) * Attach a feature to a version, maybe using milestone or labels. With the suggested changes it will be easier to follow ZeroMQ changes. I would be able to easily open issues in NetMQ tracker and link the original issue and pull requests to create a catch up plan for future releases. Let me know what you think about the suggested changes or your other ideas on how to catch with ZeroMQ changes. Regards, Doron ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] C# binding to zeromq (clrzmq project)
JeroMQ, NetMQ, and libzmq (with all its bindings) all speak the same protocol at least to ZMTP v2. If you want ZMTP v3 protocol features in NetMQ or JeroMQ you will have to wait, or add them, or pay someone to add them. On Thu, Dec 12, 2013 at 3:01 PM, Dmitriy Vsekhvalnov dvsekhval...@gmail.com wrote: Thanks Doron. How about NetMQ to JeroMQ? On Thu, Dec 12, 2013 at 4:05 PM, Doron Somech somdo...@gmail.com wrote: Hi Dmitriy, NetMQ can talk to ZeroMQ, just make sure you are using the latest version available on nuget (and leave the the Endian setting as Big). Regards, Doron On Wed, Dec 11, 2013 at 2:23 PM, Dmitriy Vsekhvalnov dvsekhval...@gmail.com wrote: Yeah, that is what is worrying me as well (no commits). Thanks, Matt ! On Wed, Dec 11, 2013 at 4:21 PM, Matt Connolly matt.conno...@me.com wrote: I haven’t had any problems. I recently switched from clrzmq to NetMQ and I’m talking to ruby processes using the rbczmq gem (which I chose because it was faster and more ruby-like in its implementation than the ruby-ffi zmq gem). I can’t speak for the maintainers of clrzmq. There have been no commits in over a year in here: https://github.com/zeromq/clrzmq -Matt On 11 Dec 2013, at 10:12 pm, Dmitriy Vsekhvalnov dvsekhval...@gmail.com wrote: Is it compatible with zeromq? I mean i can interop from C# side using netmq to ruby-ffi with native zeromq and vice versa? Officially it means native zeromq binding to CLR is dead? On Wed, Dec 11, 2013 at 3:58 PM, Matt Connolly matt.conno...@me.com wrote: NetMQ is a native C# implementation that appears more active than clrzmq: https://github.com/zeromq/netmq -Matt On 11 Dec 2013, at 9:39 pm, Dmitriy Vsekhvalnov dvsekhval...@gmail.com wrote: Hi All, trying to get an idea if clrzmq (https://github.com/zeromq/clrzmq) is supported or not anymore. It doesn't seem to support latest v4.x release. If not supported any alternates? Thanks. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] When unreliability is desired
We should ignore the encoder classes as they are specific to TCP and hacked for PGM (frame format was the same originally). Each protocol should have its own framing spec and implementation. On Thu, Dec 12, 2013 at 8:04 PM, Steven McCoy steven.mc...@miru.hk wrote: On 7 December 2013 00:34, lindl...@gmail.com wrote: I'll take a look at tipc. I've been using the pgm classes as a reference so far. They appear to use v1_encoder while tcp uses v2_encoder. If you can get a nice clean framework setup independent of the TCP/Unix socket setup I think many others and myself included would like to experiment adding many different types of protocol to 0mq. I have many on my wish list. -- Steve-o ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Does ZMQ handle tcp-RST?
On 12/13/2013 04:12 AM, artemv zmq wrote: Is there a way to tell ZMQ to handle tcp-RST? If remote peer abruptly goes down OS kernel sends RST to client. Right? So, can ZMQ handle that, and stop collecting messages inside in-mem queue for the remote peer which is down. If a TCP connection is disconnected (due to RST or otherwise), then a ZeroMQ bind socket will destroy its queue with that connection and stop queueing, and a connect socket will retain its queue but stop sending until the connection is restored. If you want to prevent queuing in all cases, set HWM to 0. Justin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Calling connect more than once on the same endpoint
I'll pedantically ask that we record problems rather than features. Problem: SUB accepts multiple connects to same PUB endpoint and result is nonsensical Solution: SUB socket should not allow multiple connects to same PUB endpoint. An issue will just rot, so you may want to find a way to send a patch to fix this problem. On Thu, Dec 12, 2013 at 8:09 PM, Trevor Bernard trevor.bern...@gmail.com wrote: Should I submit an issue for this feature? I would find it extremely useful. -Trev On Wed, Oct 16, 2013 at 11:39 AM, Pieter Hintjens p...@imatix.com wrote: Having said that, it is undocumented behavior, and there is no valid use for this in pub-sub (nor in dealer-router, nor in req-rep) and we might want to change the behavior. We do know what endpoints are already used, and could reject duplicate connects to the same endpoint. On Wed, Oct 16, 2013 at 4:26 PM, Pieter Hintjens p...@imatix.com wrote: Yes, this is expected behavior. Each connect is independent. In pub-sub this produces nonsense results but in other patterns it can be used to do things like prioritize one node over others (e.g. a PULL that connects twice to a PUSH). On Wed, Oct 16, 2013 at 4:17 PM, Trevor Bernard trevor.bern...@gmail.com wrote: I have a PUB/SUB topology and I accidently called connect twice to the same PUB endpoint and received duplicate messages. This holds true for N connects as well. Is this the correct behaviour? Simple test that recreates it: https://github.com/trevorbernard/double-trouble/blob/master/src/double_trouble/core.clj -Trev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev -- - Pieter Hintjens CEO of iMatix.com Founder of ZeroMQ community blog: http://hintjens.com -- - Pieter Hintjens CEO of iMatix.com Founder of ZeroMQ community blog: http://hintjens.com ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Does ZMQ handle tcp-RST?
On Fri, Dec 13, 2013 at 9:14 PM, Justin Karneges jus...@affinix.com wrote: If you want to prevent queuing in all cases, set HWM to 0. This will not actually prevent all queuing, just remove buffering in ZeroMQ. You'll still get buffering in TCP and on the network itself. If you want to remove all queuing completely, you have to switch to a synchronous REQ/REP model, which is nasty. Better, use a credit based flow control system to manage precisely the total amount of buffering. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Reconnect failed after iptables tricks
Is it possible that the interface is changed, so the bind is no longer active? On Fri, Dec 13, 2013 at 1:54 PM, artemv zmq artemv@gmail.com wrote: hi I was testing simple connection lose scenario on a simple DEALER-ROUTER architecture. D was a client on Windows7, R was a server on VirtualBox. What happened: I started them both, saw chatting between them in logs; after that I disabled a client ip on server host with iptables , ensured that there is no more chatting between peers; after that I had re-enabled IP address of the client on server host; then went to logs expecting to see that chatting began, but .. there was silence. The question is: is it eligible way to test reconnection? or this is a bug in ZMQ core? BR -artemv PS block script: iptables -I INPUT -s IP-of-client -j DROP unlock script: iptables -F ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] will receive fail if TCP goes down in REQ socket
Hi,
In most ZMQ sockets, the underlying TCP socket can change transparently.
Does that apply to REQ-REP sockets as well? Or will the receive call to ZMQ
socket fail?
sock = new REQ socket;
connect(sock);
while(1) {
request = receive(...);
...
send(response);
}
For example in the above code, let us say that the server with REP socket
on the other side crashed and restarted. What will happen?
Thanks,
Mohit.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Bidirectional communication
Hey everyone! I'm looking to have a server that can accept many client connections but once connected each client/server relationship is bidirectional. I know it's simple to do request/response but sometimes the server will be the one sending messages to the client first, or sometimes the client to the server first. The client will always initiate the connection to the server though. Does anyone know if there's an example that shows how to do this? I couldn't find one but some of my ZMQ terminology googling foo might have been off :) Thanks for the help in advance! Kelly ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Bidirectional communication
The client, being the one doing the connect, should also always send a first hello, and hence it’s a simple multi dealer (clients) - router (server) pattern. On Dec 13, 2013, at 21:39, Kelly Sommers kell.somm...@gmail.com wrote: Hey everyone! I'm looking to have a server that can accept many client connections but once connected each client/server relationship is bidirectional. I know it's simple to do request/response but sometimes the server will be the one sending messages to the client first, or sometimes the client to the server first. The client will always initiate the connection to the server though. Does anyone know if there's an example that shows how to do this? I couldn't find one but some of my ZMQ terminology googling foo might have been off :) Thanks for the help in advance! Kelly ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev signature.asc Description: Message signed with OpenPGP using GPGMail ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Bidirectional communication
If your server uses a ROUTER socket and your clients use DEALER sockets each can send messages to the other in any order. Once the clients send a message then the router can know the dealers identity to be able to send messages back. Best, Matt. Sent from my iPhone On 14 Dec 2013, at 7:39 am, Kelly Sommers kell.somm...@gmail.com wrote: Hey everyone! I'm looking to have a server that can accept many client connections but once connected each client/server relationship is bidirectional. I know it's simple to do request/response but sometimes the server will be the one sending messages to the client first, or sometimes the client to the server first. The client will always initiate the connection to the server though. Does anyone know if there's an example that shows how to do this? I couldn't find one but some of my ZMQ terminology googling foo might have been off :) Thanks for the help in advance! Kelly ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Not sure if ZeroMQ is right for me
Hi, I’m trying to determine if ZeroMQ is right for me. I have a local batch system of a few thousand nodes, several external batch systems distributed globally, and a server which distributes those jobs to their respective batch system. Upon startup and job completion, a job will notify the server that it has started/ended (traditionally done through email, although this has many problems, which is why we’re moving away from it). Typically jobs are staggered, but occasionally thousands of jobs all start up at once (still trying to explain to a coworker why querying oracle from 1100 hosts yesterday was bad idea) What I think I’d like to do is to have a system where the server and clients talk through a broker preferentially. There will be a main broker and a failover broker. When the server goes down, or unresponsive, the broker should deliver those to a persistent-queue client (similar to titanic service protocol?), and that persistent queue should keep those messages until the server is reconnected. When a client isn’t responsive (happens semi-frequently when you are dealing with thousands of batch machines), messages originating from the server should be delivered to the persistent queue, where it will retry at two intervals, after which the client is disconnected, server notified, and messages are discarded. (persistent queue will have write-back caching) For firewall reasons and other networking reasons, it may be necessary to have an intermediate broker (and possibly a persistent queue) at each of the external batch locations. Server implementation would probably use jeromq since the rest of the server is written in java, clients would likely use pyczmq. For message security, I was thinking I’d just pre-distribute a symmetric to each batch system that the jobs would use to encrypt their message (but leave any necessary metadata in the clear). I couldn’t really determine if there is a library/framework or something that already exists that would make this easy, although it seems like it shouldn’t be too hard with RabbitMQ. However, the support of zeromq and minimalism of the library is desirable to me and possible users. Even if I didn’t use RabbitMQ for the whole thing, I’d probably end up using it for the persistent queue at least. Should I even bother with ZeroMQ? Thanks, Brian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Bidirectional communication
You probably want a ROUTER + DEALER combination - the asyncsrv example in the guide should tell you what you need. http://zguide.zeromq.org/page:all#The-Asynchronous-Client-Server-Pattern On Fri, Dec 13, 2013 at 1:39 PM, Kelly Sommers kell.somm...@gmail.com wrote: Hey everyone! I'm looking to have a server that can accept many client connections but once connected each client/server relationship is bidirectional. I know it's simple to do request/response but sometimes the server will be the one sending messages to the client first, or sometimes the client to the server first. The client will always initiate the connection to the server though. Does anyone know if there's an example that shows how to do this? I couldn't find one but some of my ZMQ terminology googling foo might have been off :) Thanks for the help in advance! Kelly ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] will receive fail if TCP goes down in REQ socket
I believe the REQ will simply wait for the REP to come back up,
re-bind and send something.
On Fri, Dec 13, 2013 at 2:53 PM, Mohit Jaggi mohitja...@gmail.com wrote:
Hi,
In most ZMQ sockets, the underlying TCP socket can change transparently.
Does that apply to REQ-REP sockets as well? Or will the receive call to ZMQ
socket fail?
sock = new REQ socket;
connect(sock);
while(1) {
request = receive(...);
...
send(response);
}
For example in the above code, let us say that the server with REP socket on
the other side crashed and restarted. What will happen?
Thanks,
Mohit.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Not sure if ZeroMQ is right for me
It sounds like you would have good luck combining the two. RabbitMQ is best for the actual MQ part of the equation, storing messages on the brokers, while ZeroMQ is suited for passing the messages between the nodes - server and brokers, brokers and clients. Since you've looked at the Titanic pattern, just replace the Disk part with RabbitMQ and you should be on your way. (And just to be pedantic, if you do that I'd recommend flipping your terminology to fit the naming scheme: clients - workers, server - client) On Fri, Dec 13, 2013 at 4:14 PM, Van Klaveren, Brian N. b...@slac.stanford.edu wrote: Hi, I’m trying to determine if ZeroMQ is right for me. I have a local batch system of a few thousand nodes, several external batch systems distributed globally, and a server which distributes those jobs to their respective batch system. Upon startup and job completion, a job will notify the server that it has started/ended (traditionally done through email, although this has many problems, which is why we’re moving away from it). Typically jobs are staggered, but occasionally thousands of jobs all start up at once (still trying to explain to a coworker why querying oracle from 1100 hosts yesterday was bad idea) What I think I’d like to do is to have a system where the server and clients talk through a broker preferentially. There will be a main broker and a failover broker. When the server goes down, or unresponsive, the broker should deliver those to a persistent-queue client (similar to titanic service protocol?), and that persistent queue should keep those messages until the server is reconnected. When a client isn’t responsive (happens semi-frequently when you are dealing with thousands of batch machines), messages originating from the server should be delivered to the persistent queue, where it will retry at two intervals, after which the client is disconnected, server notified, and messages are discarded. (persistent queue will have write-back caching) For firewall reasons and other networking reasons, it may be necessary to have an intermediate broker (and possibly a persistent queue) at each of the external batch locations. Server implementation would probably use jeromq since the rest of the server is written in java, clients would likely use pyczmq. For message security, I was thinking I’d just pre-distribute a symmetric to each batch system that the jobs would use to encrypt their message (but leave any necessary metadata in the clear). I couldn’t really determine if there is a library/framework or something that already exists that would make this easy, although it seems like it shouldn’t be too hard with RabbitMQ. However, the support of zeromq and minimalism of the library is desirable to me and possible users. Even if I didn’t use RabbitMQ for the whole thing, I’d probably end up using it for the persistent queue at least. Should I even bother with ZeroMQ? Thanks, Brian ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] will receive fail if TCP goes down in REQ socket
ZeroMQ does not resend messages, so while the reconnect/queuing logic
will protect you to some degree, you still need to account for message loss.
If you're using REQ then you'll need to timeout the request, otherwise
if a request or response message is lost then you'll never be able to
make a request on the socket ever again. So don't just indefinitely
block on a send or receive. Further, ZeroMQ historically hasn't had a
way to get a REQ socket out of the waiting for response state in the
event of a timeout other than by closing the socket and starting over.
This means the REQ type is not really usable in production. Better to
use DEALER and format a REQ-compatible message yourself. REP does not
suffer from these problems, so you can keep on using that and have
DEALER talk to REP.
Note: it is possible that very recent versions of ZeroMQ allow REQ
sockets to revert state on error but I haven't been following this closely.
On 12/13/2013 04:17 PM, Sean Robertson wrote:
I believe the REQ will simply wait for the REP to come back up,
re-bind and send something.
On Fri, Dec 13, 2013 at 2:53 PM, Mohit Jaggi mohitja...@gmail.com wrote:
Hi,
In most ZMQ sockets, the underlying TCP socket can change transparently.
Does that apply to REQ-REP sockets as well? Or will the receive call to ZMQ
socket fail?
sock = new REQ socket;
connect(sock);
while(1) {
request = receive(...);
...
send(response);
}
For example in the above code, let us say that the server with REP socket on
the other side crashed and restarted. What will happen?
Thanks,
Mohit.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] will receive fail if TCP goes down in REQ socket
On 14 Dec 2013, at 11:46 am, Justin Karneges jus...@affinix.com wrote: ZeroMQ does not resend messages, so while the reconnect/queuing logic will protect you to some degree, you still need to account for message loss. If you're using REQ then you'll need to timeout the request, otherwise if a request or response message is lost then you'll never be able to make a request on the socket ever again. So don't just indefinitely block on a send or receive. Further, ZeroMQ historically hasn't had a way to get a REQ socket out of the waiting for response state in the event of a timeout other than by closing the socket and starting over. This means the REQ type is not really usable in production. Better to use DEALER and format a REQ-compatible message yourself. REP does not suffer from these problems, so you can keep on using that and have DEALER talk to REP. Surely REQ is still very useful in production: simply a client making a request. Connect, send request, wait for reply (or timeout) disconnect, job done. In the same way a HTTP connection would give you a single request and response. ? Of course the error handling on the client is to close the REQ socket and make a new one for a new request, but this is still fairly similar to a HTTP request. -Matt ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] will receive fail if TCP goes down in REQ socket
The ZMQ_REQ_RELAXED option on ZeroMQ v4, lets you resend requests.
On Sat, Dec 14, 2013 at 3:46 AM, Justin Karneges jus...@affinix.com wrote:
ZeroMQ does not resend messages, so while the reconnect/queuing logic
will protect you to some degree, you still need to account for message loss.
If you're using REQ then you'll need to timeout the request, otherwise
if a request or response message is lost then you'll never be able to
make a request on the socket ever again. So don't just indefinitely
block on a send or receive. Further, ZeroMQ historically hasn't had a
way to get a REQ socket out of the waiting for response state in the
event of a timeout other than by closing the socket and starting over.
This means the REQ type is not really usable in production. Better to
use DEALER and format a REQ-compatible message yourself. REP does not
suffer from these problems, so you can keep on using that and have
DEALER talk to REP.
Note: it is possible that very recent versions of ZeroMQ allow REQ
sockets to revert state on error but I haven't been following this closely.
On 12/13/2013 04:17 PM, Sean Robertson wrote:
I believe the REQ will simply wait for the REP to come back up,
re-bind and send something.
On Fri, Dec 13, 2013 at 2:53 PM, Mohit Jaggi mohitja...@gmail.com wrote:
Hi,
In most ZMQ sockets, the underlying TCP socket can change transparently.
Does that apply to REQ-REP sockets as well? Or will the receive call to ZMQ
socket fail?
sock = new REQ socket;
connect(sock);
while(1) {
request = receive(...);
...
send(response);
}
For example in the above code, let us say that the server with REP socket on
the other side crashed and restarted. What will happen?
Thanks,
Mohit.
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
