On 8/30/07, Goel, Tanmay <[EMAIL PROTECTED]> wrote: > > > > Dear Arnaud, All > > > > I have recently started looking into AMQP and came across many doubts. > Carl mentioned that you will be the best person to ask about transaction > management both for AMQP in general as a protocol and Qpid's > implementation of it. > > > > I have been having some doubts about reliability and transaction support > in AMQP. In general, how are transactions handled in AMQP for the > various use cases?
Tammy, my understanding is that AMQP provides reliability irrespective of whether you use transactions or not. Rafael is the expert on relaibility and he can correct me if I have made a mistake. As Rafael pointed out in a previous email, Transaction only cover the semantics of enque and deque of a messages and has no impact on reliability. For example, when the broker receives a persistent-message and then crashes > before it stores it on the external > DB, how will this affect the recovery? Also, this will make the > publisher to believe that the message wasn't received by the broker > since he will get a connection exception and will resend it. The broker should not acknowledge the message until it has stored the message (if persistant) in storage (and until it's backup(s) in the cluster have acked in the case of clustering) or until it gets consumed (which ever happens first). Only then should the broker ack the transfer. The cordination between the broker and it's backups is also a transaction to ensure consitency. If the publisher doesn't receive the ack before the broker crashes, it will resend the message when it connects to the broker (or it's backup). Alternatively, if the crash occurs after the message is put in > persistence or half way in the process. The client will behave in the > same way and resend the message after a broker restarts. Again the broker should only ack the message transfer after it puts in storage (and the clustering is done) or if it gets consumed (which ever happens first). You next questions is, if the broker crashes after doing all nesaccery and just before sending the ack. Again the operation of putting it into storage/clustering/acking should be atomic. If one step fails then the broker should rollback during recovery. So on restart the broker will erase the message from storage and since the publisher didn't get the ack it will resend it anyhow. But since the message has been put into persistence, the broker will send it > too after > it restarts. I mean, this is just one possible failure that occurred to > me and wasn't very clear from the spec. Another case would be, say for > pub-sub, when a persistent message is supposed to be published on 3 > queues, and the broker fails after putting in on 2. Again the broker should not ack until it enques the messages. The queuing stratergy should be atomic. Either it goes to all 3 or it doesn't go at all.(sort of a mini transaction). So when the broker recovers it will rollback the operation and it will remove the messages from the 2 queues. Could you explain a little about such cases (and others off the top of > your head) and help me understand how transactions are handled (what is > redone, what is undone, etc) and how does the broker internally assure > reliability and no data loss during the process starting from last > packet received via TCP, (maybe) put into persistence, routed by > exchange, queued, taken by consumer, ack received, message (and queue, > etc in case of temporary) deleted. (Am I missing some step here?) I > mean, as far as a transaction is concerned, failure can occur at any > step of the program/code for any type of operation, right? I assume the explanations given above answers these questions. Also, it seems that the boundaries of transactions is not very well > defined by the spec. For example, are the declare, property-change (say > setting of mode), delete, etc operations for queues, exchanges, bindings > all covered inside a transaction and deleted in case of an abort OR only > the messages and acknowledgements covered. And if these issues not very > clearly stated in the spec, won't this create a lot of uncertainty as > far as things like interoperability is concerned? Once again as Rafael mentioned, the spec is very clear. Transactions only cover the enqueue and dequeue of messages. Ex. If you start a transaction and create a queue, and you rollback, the queue will *not* be deleted. However if any messages that you published will *not* be enqueued if there is a rollback. Simillarly any messages that you consumed will *not* be dequeued if you rollback. These are some of my initial thoughts. I would be grateful if you could > explain these issues (unresolved problems, short-comings, my > misunderstanding, etc.). Hope this helps rajith. Thanks in advance. > > > > Regards, > > Tanmay > > > > > > > > ________________________________ > > From: Carl Trieloff [mailto:[EMAIL PROTECTED] > Sent: Tuesday, August 28, 2007 5:02 PM > To: Goel, Tanmay; Arnaud Simon > Subject: Re: [rabbitmq-discuss] Transactions in AMQP > > > > > > Tanmay, > > I have copied Arnaud, he is on leave till Monday (moving to Paris). So > you should be able correspond with him > next week, or set up a call with him. > > He is our in house TXN specialist, together with Mark Little. Between > Arnuad & Mark they have > written/ contributed to just about every txn spec in the industry, and > worked on most TP monitors. > > He should be able to answer any question you could dream of on txns. > > Regards > Carl. > > > > Goel, Tanmay wrote: > > Gotcha. > > > > No, I don't have any particular interest in UDP. It was just a "quick > question" to confirm, with you, what I had understood from the spec. > > > > Yes, please go ahead and put me in touch with Arnaud. It would be nice > to get his views, too, on transaction management. > > > > Thanks, > > > > Tanmay > > > > > > ________________________________ > > From: Carl Trieloff [mailto:[EMAIL PROTECTED] > Sent: Tuesday, August 28, 2007 2:45 PM > To: Goel, Tanmay > Cc: [EMAIL PROTECTED] > Subject: Re: [rabbitmq-discuss] Transactions in AMQP > > > > > In 0-10 AMQP has full DTX and TX. we assume a reliable transport. AMQP > does not yet have a UDP transport so the behavior for UDP is undefined, > especially with txn. Any UDP L1 AMQP binding would need frame > reliability to be able to support txns (we could do this, but it is > currently unspecified.) > > Arnuad Simon, Mark Little, Gordon Sim drove most of the transaction work > with Cisco. I can connect you with Arnuad directly if you like. In terms > of txns spanning queue.declare etc, the spec supports/makes that > possible. > > dtx for 0-10 is here https://jira.amqp.org/jira/browse/AMQP-4 > tx is existing in the spec docs. > > "2. Is the protocol currently assuming a reliable transport layer (TCP, > SCTP, etc) and no unreliable connection (say, UDP)? If yes, will the > later releases work on including unreliable transport layer protocols"" > > -- > we have framework that will allow us to make UDP reliable, this > work has not been done in the spec(provision has been made so it can be > done), but if done then we 'should' be able to support txns on UDP. > > Do you have an interest in UDP, and if so why? > > > Hope that helps > Carl. > > > Goel, Tanmay wrote: > > > > Hi, > > > > 1. I have a basic question about transactions in AMQP. What exactly does > a transaction include/mean? For example, does it include the > administrative features like creation of queues, exchanges, etc that > will be auto-deleted if the client aborts the transaction. I'm not > entirely clear about this, so I'm just trying to understand and figure > out possibilities. The spec says that messages and acknowledgments are > covered in a transaction. What would happen if the publisher sends a > message to the broker/queue, consumer receives (reads it from the queue) > and processes it, sends an ack to the broker and then the publisher > decides to abort the transaction? How can this transaction be atomic > and/or rolled-back since the message is already gone? > > > > Please help me understand this. I'm trying to understand the scope of a > transaction. > > > > 2. Is the protocol currently assuming a reliable transport layer (TCP, > SCTP, etc) and no unreliable connection (say, UDP)? If yes, will the > later releases work on including unreliable transport layer protocols? > > > > Thanks, > > > > Tanmay > > > > > > > > > > ________________________________ > > > > > > > _______________________________________________ > rabbitmq-discuss mailing list > [EMAIL PROTECTED] > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss > > > > > > >
