Hi All, I'll add some comments/thoughts to the wiki page when possible.
In the meantime, I'm just wondering about the notion that replaying messages from a persistent store is likely to be the easy option when an RDBMS is knackered. Can't help wondering if we're making some assumptions about the manageability/reliability of BDB vs other vendor dbs ..... I can see how this would help, but I do think we need to consider the likely performance trade-off for persistent messaging very carefully in this solution. It's hard to make persistent messaging really quick anyway (or at least other messaging vendors find it hard :-) so we'll need to test comparatively so users would understand the 'cost' of replay-ability. Bfn, Marnie On 1/24/07, Carl Trieloff <[EMAIL PROTECTED]> wrote:
what is your apache cwiki user-id? Carl. Colin Crist wrote: > Sure, I'll put some notes together. > > I don't seem to have permissions to add any pages - can you give me access > or alternatively I can put the pages on my own confluence and get someone to > move them later. > > Colin. > > >> -----Original Message----- >> From: Carl Trieloff [mailto:[EMAIL PROTECTED] >> Sent: 24 January 2007 18:07 >> To: qpid-dev@incubator.apache.org >> Subject: Re: Message history and replay >> >> >> Colin, >> >> Would you be willing to create a page on the wiki for this >> cwiki.apache.org/qpid and lay out all the use cases there so >> we don't loose them. then we can add designs etc there also >> as we explore this topic. >> >> Carl. >> >> >> Colin Crist wrote: >> >>> I believe the customary apache response here is: >>> >>> +1 >>> >>> Colin. >>> >>> >>> >>>> -----Original Message----- >>>> From: John O'Hara [mailto:[EMAIL PROTECTED] >>>> Sent: 23 January 2007 16:56 >>>> To: qpid-dev@incubator.apache.org >>>> Subject: Re: Message history and replay >>>> >>>> Two options: >>>> 1) Make the 'rewind' accessible to the client on connect >>>> 2) Make it an administrative function (reset next message to X) >>>> >>>> Both are valid. >>>> Also retentions of several days should be valid subject to >>>> >> resource >> >>>> constraints. >>>> >>>> >>>> Also Queues should have attributes related to this: >>>> a) Archive Retention Period *0*, x seconds, y messages >>>> b) Rewindable Y / *N* >>>> c) Client Can Rewind Y / *N* >>>> d) Archive treatment - *Delete* or Copy-out >>>> e) Archive interval - *30s* (queue garbage collect period) >>>> f) Archive TTL-deleted messages - Y / *N* >>>> >>>> Defaults in *. >>>> >>>> Queues are smart resources. We need to spec out what they can do >>>> (and what >>>> *subset* of that we propose to standardise over on the AMQP WG). >>>> Here's a great place to put value added features. >>>> >>>> John >>>> >>>> On 23/01/07, Colin Crist <[EMAIL PROTECTED]> wrote: >>>> >>>> >>>>> I had thought a replay queue is where the messages would be >>>>> >>>>> >>>> put I'm no >>>> >>>> >>>>> longer convinced. >>>>> >>>>> In another mail John O'Hara uses the the reading email analogy. >>>>> Continuing this to its logical conclusion could mean that >>>>> >> replay is >> >>>>> actually an option when you open the queue for reading. It >>>>> >>>>> >>>> avoids the >>>> >>>> >>>>> client having to muck about with another queue. >>>>> >>>>> This is not the traditional view of a queue but is more >>>>> >>>>> >>>> general and I >>>> >>>> >>>>> like it. >>>>> >>>>> So, how about a queue contains all messages put on it >>>>> >> since it was >> >>>>> last "purged". When a queue is opened for reading, it can >>>>> >> be opened >> >>>>> with a selector and one part of that selector is whether >>>>> >> or not the >> >>>>> message has been commited by a consumer or not. >>>>> >>>>> A consumer could then always open the queue and request all >>>>> >>>>> >>>> messages, >>>> >>>> >>>>> committed or not, from a known point, e.g. a trade ID or >>>>> >>>>> >>>> message ID. >>>> >>>> >>>>> The key thing here I suppose is that this type of consumer >>>>> >>>>> >>>> is writing >>>> >>>> >>>>> the message in some other form into a database and so >>>>> >>>>> >>>> always knows the >>>> >>>> >>>>> last thing written. >>>>> A nice benefit is that if the database is really hosed you >>>>> >>>>> >>>> could get >>>> >>>> >>>>> replay from start of day. This would definitely help >>>>> >>>>> >>>> implement DR for >>>> >>>> >>>>> purely message based systems. >>>>> >>>>> So, can I specialise queues and layer this functionality on >>>>> >>>>> >>>> top of the >>>> >>>> >>>>> spec in the APID implementation with some kind of >>>>> >> provider specific >> >>>>> header property? >>>>> >>>>> Colin. >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Marnie McCormack [mailto:[EMAIL PROTECTED] >>>>>> Sent: 22 January 2007 19:54 >>>>>> To: qpid-dev@incubator.apache.org; [EMAIL PROTECTED] >>>>>> Subject: Re: Message history and replay >>>>>> >>>>>> I've been trying to imagine the scenarios in which a user >>>>>> >>>>>> >>>> will make >>>> >>>> >>>>>> use of a ReplayExchange. >>>>>> >>>>>> I thought that the most common usage would be the following: >>>>>> >>>>>> - the receiving application has lost some messages sent >>>>>> >> to it and >> >>>>>> needs them to be resent from a point in time or a message ID >>>>>> >>>>>> (The only other recovery scenario I thought likely was broker >>>>>> failure, which would be handled by restarted (with >>>>>> persistence) and the undelivered messages would already >>>>>> >> remain in >> >>>>>> the BDB store.) >>>>>> >>>>>> So, the bit I'm not sure about is this - you have no way >>>>>> >>>>>> >>>> of knowing >>>> >>>> >>>>>> in advance the point at which replay might be needed >>>>>> >> from (unless >> >>>>>> using client acks - which work with ordinary queues ?). >>>>>> >>>>>> Thus your recipient app starts up again (or recognises it >>>>>> >>>>>> >>>> has lost >>>> >>>> >>>>>> messages) and then wants to replay. So, the queue it is >>>>>> >>>>>> >>>> reading from >>>> >>>> >>>>>> is rebound to a replay exchange (with a marker >>>>>> ID) ? Does that mean that it is actually reading from a >>>>>> >> different >> >>>>>> queue than normal operation, or that the queue is bound to a >>>>>> different exchange type ? >>>>>> >>>>>> The logic behind this could be quite complex - in that >>>>>> >>>>>> >>>> the balance >>>> >>>> >>>>>> between normal operation and replay operation in subtle >>>>>> >> and would >> >>>>>> involve the receiving client understanding the AMQP behaviour at >>>>>> quite a low level ? >>>>>> >>>>>> Apologies if I've missed a simple flow for replay :-) >>>>>> >>>>>> Regards, >>>>>> Marnie >>>>>> >>>>>> On 1/22/07, Carl Trieloff <[EMAIL PROTECTED]> wrote: >>>>>> >>>>>> >>>>>>> This has been asked for before. I would interested to see if an >>>>>>> exchange could be used. I also think this might bring up >>>>>>> >>>>>>> >>>>>> the topic of >>>>>> >>>>>> >>>>>>> chained exchanges again. >>>>>>> >>>>>>> Colin, do you have any interest in creating a prototype >>>>>>> >>>>>>> >>>>>> ReplyExchange >>>>>> >>>>>> >>>>>>> to see how well it works? >>>>>>> Carl. >>>>>>> >>>>>>> Alan Conway wrote: >>>>>>> >>>>>>> >>>>>>>> On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> A quick question... >>>>>>>>> >>>>>>>>> A common recovery scenario is requesting replay on a >>>>>>>>> >>>>>>>>> >>>> queue from >>>> >>>> >>>>>>>>> a >>>>>>>>> >>>>>>>>> >>>>>>> given, >>>>>>> >>>>>>> >>>>>>>>> known message, either by its ID or by something in the >>>>>>>>> >>>>>>>>> >>>>>> header. Its >>>>>> >>>>>> >>>>>>> saves all >>>>>>> >>>>>>> >>>>>>>>> that mucking about with XA and makes recovery the >>>>>>>>> >>>>>>>>> >>>> normal way of >>>> >>>> >>>>>>>>> system startup rather then some exceptional case. >>>>>>>>> >>>>>>>>> This means when a message has reached all its recipients >>>>>>>>> >>>>>>>>> >>>>>> it should >>>>>> >>>>>> >>>>>>> never be >>>>>>> >>>>>>> >>>>>>>>> marked for removal from the durable message store. >>>>>>>>> >>>>>>>>> As a user, I would also like to control when messages >>>>>>>>> >>>>>>>>> >>>>>> get removed >>>>>> >>>>>> >>>>>>>>> as >>>>>>>>> >>>>>>>>> >>>>>>> part of >>>>>>> >>>>>>> >>>>>>>>> my end of day or even weekly processs, probably >>>>>>>>> >>>>>>>>> >>>> based on header >>>> >>>> >>>>>>> property >>>>>>> >>>>>>> >>>>>>>>> selectors. >>>>>>>>> >>>>>>>>> I don't believe this is supported in AMQP in the wire >>>>>>>>> >>>>>>>>> >>>>>> protocol (nor >>>>>> >>>>>> >>>>>>> perhaps >>>>>>> >>>>>>> >>>>>>>>> should it be) - is it maybe something that could be done >>>>>>>>> >>>>>>>>> >>>>>> via some >>>>>> >>>>>> >>>>>>>>> standardised command messages to a "replay" exchange? >>>>>>>>> >>>>>>>>> Any thoughts on implementation difficulty? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> You could represent this in AMQP protocol using a custom >>>>>>>> >>>>>>>> >>>>>> exchange type. >>>>>> >>>>>> >>>>>>>> A ReplayExchange stores all messages sent to it (until >>>>>>>> >>>>>>>> >>>>>> cleaned up by >>>>>> >>>>>> >>>>>>>> some administrative function.) >>>>>>>> >>>>>>>> When binding a queue to a replay exchange you MAY specify an >>>>>>>> argument "replay-from=N". A queue so bound will be >>>>>>>> >>>>>>>> >>>>>> "pre-filled" with >>>>>> >>>>>> >>>>>>>> the range of message starting from the Nth message up >>>>>>>> >> to the most >> >>>>>>>> recent message sent to the exchange. Thereafter the >>>>>>>> >>>>>>>> >>>>>> exchange behaves >>>>>> >>>>>> >>>>>>>> like a fan-out exchange, each new message being added to >>>>>>>> >>>>>>>> >>>>>> all the bound queues. >>>>>> >>>>>> >>>>>>>> That gives you replay entirely within the standard >>>>>>>> >>>>>>>> >>>>>> protocol, using a >>>>>> >>>>>> >>>>>>>> standard extension point to add this as proprietary broker >>>>>>>> functionality. >>>>>>>> >>>>>>>> Note that to combine replay with other exchange types >>>>>>>> >>>>>>>> >>>> requires >>>> >>>> >>>>>>>> exchange chaining which has been discussed several times >>>>>>>> >>>>>>>> >>>>>> but is not >>>>>> >>>>>> >>>>>>>> yet an agreed part of the protocol - this may be >>>>>>>> >> another use case >> >>>>>>>> for that discussion >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Alan. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >> > > >
