[zeromq-dev] Not sure if ZeroMQ is right for me

[Van Klaveren, Brian N.]

Hi,

I’m trying to determine if ZeroMQ is right for me. 

I have a local batch system of a few thousand nodes, several external batch 
systems distributed globally, and a server which distributes those jobs to 
their respective batch system. 

Upon startup and job completion, a job will notify the server that it has 
started/ended (traditionally done through email, although this has many 
problems, which is why we’re moving away from it).
Typically jobs are staggered, but occasionally thousands of jobs all start up 
at once (still trying to explain to a coworker why querying oracle from 1100 
hosts yesterday was bad idea)

What I think I’d like to do is to have a system where the server and clients 
talk through a broker preferentially.
There will be a main broker and a failover broker.
When the server goes down, or unresponsive, the broker should deliver those to 
a persistent-queue client (similar to titanic service protocol?), and that 
persistent queue should keep those messages until the server is reconnected.
When a client isn’t responsive (happens semi-frequently when you are dealing 
with thousands of batch machines), messages originating from the server should 
be delivered to the persistent queue, where it will retry at two intervals, 
after which the client is disconnected, server notified, and messages are 
discarded. (persistent queue will have write-back caching)

For firewall reasons and other networking reasons, it may be necessary to have 
an intermediate broker (and possibly a persistent queue) at each of the 
external batch locations.

Server implementation would probably use jeromq since the rest of the server is 
written in java, clients would likely use pyczmq.

For message security, I was thinking I’d just pre-distribute a symmetric to 
each batch system that the jobs would use to encrypt their message (but leave 
any necessary metadata in the clear).

I couldn’t really determine if there is a library/framework or something that 
already exists that would make this easy, although it seems like it shouldn’t 
be too hard with RabbitMQ. However, the support of zeromq and minimalism of the 
library is desirable to me and possible users. Even if I didn’t use RabbitMQ 
for the whole thing, I’d probably end up using it for the persistent queue at 
least. Should I even bother with ZeroMQ?

Thanks,
Brian

___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Re: [zeromq-dev] Not sure if ZeroMQ is right for me

[Sean Robertson]

It sounds like you would have good luck combining the two. RabbitMQ is
best for the actual MQ part of the equation, storing messages on the
brokers, while ZeroMQ is suited for passing the messages between the
nodes - server and brokers, brokers and clients. Since you've looked
at the Titanic pattern, just replace the Disk part with RabbitMQ
and you should be on your way. (And just to be pedantic, if you do
that I'd recommend flipping your terminology to fit the naming scheme:
clients - workers, server - client)

On Fri, Dec 13, 2013 at 4:14 PM, Van Klaveren, Brian N.
b...@slac.stanford.edu wrote:
 Hi,

 I’m trying to determine if ZeroMQ is right for me.

 I have a local batch system of a few thousand nodes, several external batch 
 systems distributed globally, and a server which distributes those jobs to 
 their respective batch system.

 Upon startup and job completion, a job will notify the server that it has 
 started/ended (traditionally done through email, although this has many 
 problems, which is why we’re moving away from it).
 Typically jobs are staggered, but occasionally thousands of jobs all start up 
 at once (still trying to explain to a coworker why querying oracle from 1100 
 hosts yesterday was bad idea)

 What I think I’d like to do is to have a system where the server and clients 
 talk through a broker preferentially.
 There will be a main broker and a failover broker.
 When the server goes down, or unresponsive, the broker should deliver those 
 to a persistent-queue client (similar to titanic service protocol?), and that 
 persistent queue should keep those messages until the server is reconnected.
 When a client isn’t responsive (happens semi-frequently when you are dealing 
 with thousands of batch machines), messages originating from the server 
 should be delivered to the persistent queue, where it will retry at two 
 intervals, after which the client is disconnected, server notified, and 
 messages are discarded. (persistent queue will have write-back caching)

 For firewall reasons and other networking reasons, it may be necessary to 
 have an intermediate broker (and possibly a persistent queue) at each of the 
 external batch locations.

 Server implementation would probably use jeromq since the rest of the server 
 is written in java, clients would likely use pyczmq.

 For message security, I was thinking I’d just pre-distribute a symmetric to 
 each batch system that the jobs would use to encrypt their message (but leave 
 any necessary metadata in the clear).

 I couldn’t really determine if there is a library/framework or something that 
 already exists that would make this easy, although it seems like it shouldn’t 
 be too hard with RabbitMQ. However, the support of zeromq and minimalism of 
 the library is desirable to me and possible users. Even if I didn’t use 
 RabbitMQ for the whole thing, I’d probably end up using it for the persistent 
 queue at least. Should I even bother with ZeroMQ?

 Thanks,
 Brian

 ___
 zeromq-dev mailing list
 zeromq-dev@lists.zeromq.org
 http://lists.zeromq.org/mailman/listinfo/zeromq-dev
___
zeromq-dev mailing list
zeromq-dev@lists.zeromq.org
http://lists.zeromq.org/mailman/listinfo/zeromq-dev