Emmanuel Lécharny wrote: > Hi ! > > A few insomnias later... ;-) > > So I'm almost done with transaction implementation in the server (with a > LOT of fixes on the client side too...) > > I have run some perf tests where I inject a number of entries, with a > different set of transaction size. Here are the results : > > Nb updates/transaction Nb add/s > > 0 116 > 1 108 > 2 158 > 5 208 > 10 243 > 20 270 > 50 285 > 100 303 > 200 322 > 500 357 > 1000 400 > > As we can see, we are slightly slower when we send a transaction > operation for every add, compared to no transaction being sent (which is > kind of making sense, as we send 3 operations instead of one - > startTxn/add/endTxn vs add -). > > When we gather 2 adds into one single transaction, we get an almost 50% > speed improvement, which tops around a 4x improvement when we gather > 1000 adds within one single transaction. > > Note that the test I run do commit every N addition, and I'm injecting > 10000 entries total (numbers aren't changing significantly if I inject > 100 000 entries, it's just that a run would take many minutes instead of > 20s to 3mins). > > So it does the job, and it improves performance.
Congratulations! > Now, there is more work to do : > o rollback hasn't been tested yet > o we need to forbid concurrent writes while a transaction is being > executed - which means that a transaction is seen as one single giant > write - with the consequence that a rogue client starting a transaction > but never finishing it will potentially lock the server. > o a mechanism to protect the server against such rogues client has to be > implemented (could be a limitation in the number of update per > transaction and/or a time limit for a transaction). In OpenLDAP we simply queue the incoming ops until the Commit is received, then execute them all at once. And yes, there's a limit on the number of ops allowed in a txn. > Note that while a > transaction is being conducted, nothing is flushed on disk. > o more tests are to be conducted (do we see entries being updated during > a transaction if we do a search aside the transaction ?) The RFC doesn't require searches to honor transactions. The spec really just allows you to perform multiple write ops as a single atomic operation. > o controls are not returned back atm, as they should > > Otherwise, I have met some issues in other parts of the server whild > implementing transactions : > > o for most of the operations, controls aren't sent back to the client > (except for the Bind Operation) > o We should write a <Operation>Decorator instead of a request/response > implementation, to avoid a call to a method that is a big switch doing > the exact same thing. This is a bit cleaner. > > Also note that if the partition does not support transactions, then the > transaction requests are simply ignored. > > I will commit the code when the previous listed points have been > fixed/implemented. > > I do think we should leverage this extension when injecting a huge > number of entries into the server, like when using Studio to import a > LDIF file. I would suggest to gather updates in block of 100 to get a > good boost (x3) without blocking the writes for other clients for too > long (300ms). Yes, in OpenLDAP 2.5 we can use txns to batch replication updates, also for performance reasons. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
