Andrew,
In some other work some of I guys I work with have done have created a
low-latency scheduling mechanism via AMQP (Qpid) for the Condor Grid. It
sounds like you are trying to achieve something equivalent using a
different grid engine. If you like I can point you to doc and code for
that integration, either to use, hack, or to copy :-)
Carl.
Andrew Wright wrote:
Actually this was to integrate with Terracotta. TC needs some sort of
load balancer to get locality of reference (for performance) - qpid
would fit the bill here, via hierarchical topics. There'd only be a
single subscriber on a topic at a time. TC clients are notified as
nodes join/leave, so can take over the processing of a given topic if
the primary processor goes away.
As you say, Coherence has this sort of thing built in. Unfortunately
it also comes with a fairly hefty Oracle price tag...
Andrew
On 30 Jan 2009, at 23:14, Robert Greig wrote:
2009/1/30 Andrew Wright <[email protected]>:
Apologies, I've confused the options rather. We'd thought about
using the
dynamic federation features in M4 to get effective point-to-point links
between 'entry' nodes, and 'processing' nodes, in a distributed app.
The
idea was that each app node would subscribe to a subset of incoming
messages
(for load balancing), and the federation would take care of getting the
messages to the right place. Ideally there'd be no message loss in
the case
of app node failover, ie, we want to maintain the contents of an
incoming
work queue if the destination point of a dynamically federated queue
changed. I'd thought perhaps the atomic rebinding feature could be
useful
here.
And you are using Coherence as the grid, with the grid nodes as the
processing nodes? If so, I would be very tempted to use the
InvocableMap feature of Coherence and have the messages injected into
the grid using that method. That will guarantee that if a node goes
down as it is processing a message it is automatically handled by the
backup node.
For the injector nodes, I would using a number of nodes with a queue
to round-robin between the nodes.
I am not sure how the topic helps here - it will go to all subscribers
so unless the operation is idempotent you would have to handle which
subscriber is the "master"?
RG
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
