Hi Alan, I've seen other replies to this but I had this in draft so will take it mail by mail..
The scenario you describe is different to my requirement - at least if I read you correctly. My requirement is a consumer initiated replay of messages from a queue where the messages are originating in any exchange. Tyically the consumers recovery is to "fill in a gap" of messages it may have missed as its avoiding XA and is optimistically writing to is store (RDBMS or whatever) before doing a 1 phase commit in the right order. Additionally, as its messaging and not a database, just if I've commited my messages is no reason not to want them again, at least not in my ideal solution :) Messaging is the recovery point. Replay would be onto a different queue to maintain ordering (but with dupes of course, idempotent consumers is my contract) whilst new messages would be getting delivered to the main queue (with a subset on the head being duplicated in the replay queue). The replayed messages would be drained by the client from the replay queue and then normal message consumption would resume on the main queue. >From my point of view, the contract is with the queue the consumer is reading from and not the originating exchange(s). I am most interested in this with a pubsub exchange so I think core broker support is needed to access the store and exchange chaining is not enough? There may be other messages on the queue from other exchanges and they also need replaying. The consumer is not interested in the upstream exchange. The consumer would always recover like this as its starting from some known stable point in history. There needs to be some "rendezvous" (old skool term I know) with a "replay complete" message on the replay queue in order to flip back to the main queue. A valid recovery point could also be a configuration defined "start of day" timestamp for the most erratic consumers. I'd like to use a hard messaging backbone as the ulimate point of recovery. I have bad consumers and want to rely on my messaging to fix my world. Please correct me if I am missing any AMQP behavour as I'm only causally aware of the spec from reading the docs and not working with it at this point and I wont "get it" until I work with it closer. More later, Many thanks all, Colin. > > On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote: > > Hi, > > > > A quick question... > > > > A common recovery scenario is requesting replay on a queue from a > > given, known message, either by its ID or by something in > the header. > > Its saves all that mucking about with XA and makes recovery > the normal > > way of system startup rather then some exceptional case. > > > > This means when a message has reached all its recipients it should > > never be marked for removal from the durable message store. > > > > As a user, I would also like to control when messages get > removed as > > part of my end of day or even weekly processs, probably based on > > header property selectors. > > > > I don't believe this is supported in AMQP in the wire protocol (nor > > perhaps should it be) - is it maybe something that could be > done via > > some standardised command messages to a "replay" exchange? > > > > Any thoughts on implementation difficulty? > > You could represent this in AMQP protocol using a custom > exchange type. > > A ReplayExchange stores all messages sent to it (until > cleaned up by some administrative function.) > > When binding a queue to a replay exchange you MAY specify an > argument "replay-from=N". A queue so bound will be > "pre-filled" with the range of message starting from the Nth > message up to the most recent message sent to the exchange. > Thereafter the exchange behaves like a fan-out exchange, each > new message being added to all the bound queues. > > That gives you replay entirely within the standard protocol, > using a standard extension point to add this as proprietary > broker functionality. > > Note that to combine replay with other exchange types > requires exchange chaining which has been discussed several > times but is not yet an agreed part of the protocol - this > may be another use case for that discussion > > Cheers, > Alan. > >
