what is your apache cwiki user-id?
Carl.
Colin Crist wrote:
Sure, I'll put some notes together.
I don't seem to have permissions to add any pages - can you give me access
or alternatively I can put the pages on my own confluence and get someone to
move them later.
Colin.
-----Original Message-----
From: Carl Trieloff [mailto:[EMAIL PROTECTED]
Sent: 24 January 2007 18:07
To: qpid-dev@incubator.apache.org
Subject: Re: Message history and replay
Colin,
Would you be willing to create a page on the wiki for this
cwiki.apache.org/qpid and lay out all the use cases there so
we don't loose them. then we can add designs etc there also
as we explore this topic.
Carl.
Colin Crist wrote:
I believe the customary apache response here is:
+1
Colin.
-----Original Message-----
From: John O'Hara [mailto:[EMAIL PROTECTED]
Sent: 23 January 2007 16:56
To: qpid-dev@incubator.apache.org
Subject: Re: Message history and replay
Two options:
1) Make the 'rewind' accessible to the client on connect
2) Make it an administrative function (reset next message to X)
Both are valid.
Also retentions of several days should be valid subject to
resource
constraints.
Also Queues should have attributes related to this:
a) Archive Retention Period *0*, x seconds, y messages
b) Rewindable Y / *N*
c) Client Can Rewind Y / *N*
d) Archive treatment - *Delete* or Copy-out
e) Archive interval - *30s* (queue garbage collect period)
f) Archive TTL-deleted messages - Y / *N*
Defaults in *.
Queues are smart resources. We need to spec out what they can do
(and what
*subset* of that we propose to standardise over on the AMQP WG).
Here's a great place to put value added features.
John
On 23/01/07, Colin Crist <[EMAIL PROTECTED]> wrote:
I had thought a replay queue is where the messages would be
put I'm no
longer convinced.
In another mail John O'Hara uses the the reading email analogy.
Continuing this to its logical conclusion could mean that
replay is
actually an option when you open the queue for reading. It
avoids the
client having to muck about with another queue.
This is not the traditional view of a queue but is more
general and I
like it.
So, how about a queue contains all messages put on it
since it was
last "purged". When a queue is opened for reading, it can
be opened
with a selector and one part of that selector is whether
or not the
message has been commited by a consumer or not.
A consumer could then always open the queue and request all
messages,
committed or not, from a known point, e.g. a trade ID or
message ID.
The key thing here I suppose is that this type of consumer
is writing
the message in some other form into a database and so
always knows the
last thing written.
A nice benefit is that if the database is really hosed you
could get
replay from start of day. This would definitely help
implement DR for
purely message based systems.
So, can I specialise queues and layer this functionality on
top of the
spec in the APID implementation with some kind of
provider specific
header property?
Colin.
-----Original Message-----
From: Marnie McCormack [mailto:[EMAIL PROTECTED]
Sent: 22 January 2007 19:54
To: qpid-dev@incubator.apache.org; [EMAIL PROTECTED]
Subject: Re: Message history and replay
I've been trying to imagine the scenarios in which a user
will make
use of a ReplayExchange.
I thought that the most common usage would be the following:
- the receiving application has lost some messages sent
to it and
needs them to be resent from a point in time or a message ID
(The only other recovery scenario I thought likely was broker
failure, which would be handled by restarted (with
persistence) and the undelivered messages would already
remain in
the BDB store.)
So, the bit I'm not sure about is this - you have no way
of knowing
in advance the point at which replay might be needed
from (unless
using client acks - which work with ordinary queues ?).
Thus your recipient app starts up again (or recognises it
has lost
messages) and then wants to replay. So, the queue it is
reading from
is rebound to a replay exchange (with a marker
ID) ? Does that mean that it is actually reading from a
different
queue than normal operation, or that the queue is bound to a
different exchange type ?
The logic behind this could be quite complex - in that
the balance
between normal operation and replay operation in subtle
and would
involve the receiving client understanding the AMQP behaviour at
quite a low level ?
Apologies if I've missed a simple flow for replay :-)
Regards,
Marnie
On 1/22/07, Carl Trieloff <[EMAIL PROTECTED]> wrote:
This has been asked for before. I would interested to see if an
exchange could be used. I also think this might bring up
the topic of
chained exchanges again.
Colin, do you have any interest in creating a prototype
ReplyExchange
to see how well it works?
Carl.
Alan Conway wrote:
On Thu, 2007-01-18 at 15:52 +0000, Colin Crist wrote:
Hi,
A quick question...
A common recovery scenario is requesting replay on a
queue from
a
given,
known message, either by its ID or by something in the
header. Its
saves all
that mucking about with XA and makes recovery the
normal way of
system startup rather then some exceptional case.
This means when a message has reached all its recipients
it should
never be
marked for removal from the durable message store.
As a user, I would also like to control when messages
get removed
as
part of
my end of day or even weekly processs, probably
based on header
property
selectors.
I don't believe this is supported in AMQP in the wire
protocol (nor
perhaps
should it be) - is it maybe something that could be done
via some
standardised command messages to a "replay" exchange?
Any thoughts on implementation difficulty?
You could represent this in AMQP protocol using a custom
exchange type.
A ReplayExchange stores all messages sent to it (until
cleaned up by
some administrative function.)
When binding a queue to a replay exchange you MAY specify an
argument "replay-from=N". A queue so bound will be
"pre-filled" with
the range of message starting from the Nth message up
to the most
recent message sent to the exchange. Thereafter the
exchange behaves
like a fan-out exchange, each new message being added to
all the bound queues.
That gives you replay entirely within the standard
protocol, using a
standard extension point to add this as proprietary broker
functionality.
Note that to combine replay with other exchange types
requires
exchange chaining which has been discussed several times
but is not
yet an agreed part of the protocol - this may be
another use case
for that discussion
Cheers,
Alan.
