Hi Praveen, I've checked in my changes on trunk if you want to have a play around... let me know if you run into any issues,
Cheers, Rob On 20 January 2012 16:57, Praveen M <[email protected]> wrote: > Hi Rob, > > Thanks for writing. Please see inline. > > On Fri, Jan 20, 2012 at 1:27 AM, Rob Godfrey <[email protected] > >wrote: > > > Hi Praveen, > > > > apologies for the slow response here... > > > > On 14 January 2012 03:27, Praveen M <[email protected]> wrote: > > > > > Hi, > > > > > > I would like to exploit the Berkeley DB's nosyncCommit (lazy commit) > > > option to exploit enqueue performance. > > > I see that in the current BDBMessageStore implementation there is no > way > > to > > > override the syncCommit Boolean value and > > > would like to know if this could be made into an option. > > > > > > Also, that said It'll be nice to know if someone had used the > > > nosyncCommit() option that BDB provides and if using it does any > > invisible > > > harm. (DB corruptions etc.). > > > > > > > > So, I have coincidentally been working on a patch to increase throughput > > for non transactional messages which makes greater use of no-sync > commits. > > The patch is basically to allow a certain amount of "out-of-order" > > processing while still maintaining the guarantees on durability that the > > AMQP specification states that you should get. For non transactional > > durable messages this is giving me something like a 6.5x increase in max > > total throughput. > > > > *Ah, this is great. Exactly what I was looking for. * > > > > > Unfortunately this patch did uncover a couple of issues, one being the > need > > to have some code in Qpid to handle occasional BDB lock timeouts... and > the > > other a very occasional bug in BDB itself which they guys at Oracle now > > have a fix for, but won't be publicly released until their regular > update. > > There is a reasonably easy workaround for this which I shall incorporate > > into my patch, but it may take me a couple more days to get it all > > complete. > > > > Note that my patch is a little different to what you are suggesting - in > > your case a message may go on to a queue in memory and then be delivered > > without the guarantee that it was sync'd to disk... in my patch the > message > > won;t be released into the in-memory queue until it is sync'd, however I > am > > batching up more such enqueues to get better performance. In my tests > the > > total max throughput with these two possible solutions is pretty much > > identical, however the latency on individual messages is likely to be a > > little different. > > > > *That sounds even better. Your solution fits the bill perfectly.* > *I'd prefer the message getting sync'd to disk first before delivery too.* > * * > > > > > Now, to why do I want to override that value to false? (I do realize that > > > turning it to false, doesn't guarantee that the message was written to > > > disk) > > > In my use case. I have a database outside Qpid which persists all the > > > messages enqueued (for durability and transactionality with message > > > callback operations.). > > > > > > My sequence of operations is, > > > > > > 1) Enqueue messages -> persist message in my database outside Qpid -> > > > enqueue to Qpid in persistant mode > > > Since my message is already persisted in my database, I'm ok with > losing > > a > > > message in BDB, and would like to enhance throughput of my overall > > enqueue > > > operation by using the nosyncCommit option. > > > The reason I'm not using non-persistant mode in Qpid is because I don't > > > want to overwhelm the memory available by flooding the broker with > > > messages. > > > > > > In the event of a broker crash, I plan to restore the messages from my > > > database outside Qpid. (I realize this could turn out a little > > expensive). > > > I've been thinking > > > that if there is a replication/HA solution to have messages go to a > > primary > > > and secondary broker (in a 2 instance cluster), in the event of a > crash, > > > I'll have to > > > only restore the most recent messages from my database outside Qpid > upto > > a > > > certain checkpoint to which the primary and secondary are sync'ed (Not > > sure > > > if this is achievable.). > > > > > > I do realize that my architecture is a little complicated. Please do > let > > me > > > know if you have more questions about what I'm trying to achieve here, > > I'll > > > be happy to > > > clarify doubts. > > > > > > I'd really like to know if I can add an option to override the > syncCommit > > > to false without any major side-effects apart from what I expect. > > > > > > > > > > > Does my patch seem like it would satisfy your requirements? > > > > *Yes. It's quite exactly what I was looking for. I have a couple of > questions on the HA* > *state of things though. Please do see our other thread on that. It will be > really * > *valuable if you mention what you think.* > * > * > *Thank you Rob.* > * * > > > > > Cheers, > > Rob > > > > > > -- > -Praveen >
