Clustering for high availability.

Initial cluster implementation will keep a common shared broker model synchronized across all nodes with cpg (Closed Process Group, part of openais.) CPG guarantees ordrer of delivery by using self-delivery. A node that sends a message should not take actions associated with that message until it is re-delivered via the CPG protocl. So the real processing of frames will be triggered not by a frame arriving via AMQP, but by that frame being re-delived via CPG multicast.

Interceptor chains:

Frame arrives on network.

• ConnectionFrameHandler

  See also:

  Frame dispatch refactoring proposal.

• ReplicateOutHandler: multicast frame to cluster

Frame arrives via multicast:

• MulticastFrameHandler: set channel/sendHandler on frame:
  • If (I am primary for frame) setSendHandler to normal chain

    See also:

    Frame dispatch refactoring proposal.

  • else setSendHandler(record session chain);
• RequestResponseHandler: update 0-9 request/response IDs.
• ProcessFrameHandler:

  See also:

  Frame dispatch refactoring proposal.

  The record session chain is:
• RequestResponseHandler:

  See also:

  Frame dispatch refactoring proposal.

• RecordSessionHandler: update local session to show that frame was sent.

The net result of the above is that all incoming AMQP frames are muticast to the cluster.Every cluster node processes every frame as normal except that non-primary nodes don't send any frames back to the client, they just record the fact that the primary node would have. The client sees a normal interaction, and every cluster member is able to take up the clients session.

Some details omitted: e.g. setting up correspondence between connections, channels and sessions. Multicast frames will be accompanied by a cluster-wide unique session identifier that nodes can map to channel numbers etc.

Note that although this is not our final goal for clustering, the final goal can be reached by implementing additional handlers (e.g. PointToPointBackupHandler) and adding logic to choose the right handlers based on configuration of primary, backup etc.

We will retain multicast backup for wiring, so the above will still apply to queue.declare et. al.

We will probably need a Cluster class for inter-node communication at some point.

---