I am interested in local persistent queueing of messages in the case where
the remote server is unreachable.
The application I have in mind is a RADIUS server which forwards RADIUS
accounting packets to a central data warehouse for processing. However the
data warehouse and/or the intervening network connection may not be
available 100% of the time, so may need to be spooled for later delivery.:
Acct request zeromq
NAS -------------> RADIUS ----------------------> warehouse
<------------- server
Acct response ^
|
v
spool
Since the RADIUS server has sent an acknowledgement back to the NAS, the
data should be spooled to persistent storage. It doesn't need to be 100%
bomb-proof but in the normal case it should survive a process stop/start.
Another similar application might be forwarding syslogs over zeromq.
There's a tantalising hint under zmq_setsockopt(3) that ZMQ_SWAP might
perform some sort of disk offload, but it seems this might only be for
messages above the high water mark, and/or may or may not persist if the
client is terminated and restarted.
I tried it with the Ruby bindings, but that wasn't very productive:
s.setsockopt(SWAP, 1)
results in:
remote_swap.rb:32:in `setsockopt': Invalid argument (RuntimeError)
from remote_swap.rb:32
I also can't see anywhere in the libzmq C++ source where the swap option is
actually used (other than being parsed in options.cpp), but that's probably
just me being dense.
It seems to me that what I want is:
(1) some sort of local spooler/disk queue application which I can fire
the zeromq stream at. Is there existing code for this?
(2) some way of knowing that the warehouse has received a particular
message, so that I know it's safe to delete it from the disk spool. I'm not
worried about occasional duplicates.
http://www.zeromq.org/whitepapers:design-v05
talks about disconnection events and gap messages, but I can't find these
in the 2.0.6 code base.
$ grep -iR gap .
$
So perhaps I should implement some sort of reliable delivery protocol:
basically an ACK returned for each message after it has been received and
processed by the warehouse. (*)
Is there an existing 'best practice' for doing reliable delivery over
zeromq? I could, for example, use the REQ/REP pattern, but it looks like
it could be difficult to get decent throughput (AFAICS I would need
multiple threads at each end and multiple connected sockets). Or would it
be better to piggy-back something onto P2P, with a message stream in one
direction and an ACK stream in the other?
I'm sure this must be something which has been done many times before.
Regards,
Brian.
(*) A concrete example of what I'm thinking of is called "QoS 1" in MQTT:
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r0m0/index.jsp?topic=/com.ibm.etools.mft.doc/ac10850_.htm
http://www.mqtt.org/
_______________________________________________
zeromq-dev mailing list
[email protected]
http://lists.zeromq.org/mailman/listinfo/zeromq-dev
