A bit of improvement/cleanup and test later, I found some serious bugs and fixed them, then ran some performance test.
Injecting 100 000 elements (random order) then deleting them (random order too), with transaction, and different number of operation per transaction give those numbers : 1 op/txn : ---------- Delta for 100000 inserts : 38573 2 592/s Delta for 100000 deletes : 39331 2 542/s 2 ops/txn : ----------- Delta for 100000 inserts : 20,969 4 769/s Delta for 100000 deletes : 22,619 4 421/s 5 ops/txn : ----------- Delta for 100000 inserts : 11,430 8 749/s Delta for 100000 deletes : 12,745 7 846/s 10 ops/txn : ------------ Delta for 100000 inserts : 8,108 12 333/s Delta for 100000 deletes : 8,243 12 131/s 20 ops/txn : ------------ Delta for 100000 inserts : 6,467 15 463/s Delta for 100000 deletes : 8,119 12 316/s 50 ops/txn : ------------ Delta for 100000 inserts : 4,946 20 218/s Delta for 100000 deletes : 5,886 16 989/s 100 ops/txn : ------------- Delta for 100000 inserts : 4,442 22 512/s Delta for 100000 deletes : 5,305 18 850/s 1000 ops/txn : -------------- Delta for 100000 inserts : 2,694 37 119/s Delta for 100000 deletes : 5,784 17 289/s As we can see, the Insert operation is slightly faster than the delete operation (which is a bit more complex), and the delete operation peaks at some point (ie, it's not useful to gather more operations within one transaction). More to come. On 2017-09-27 12:16, Emmanuel Lécharny <[email protected]> wrote: > Hi guys, > > > I was working on Mavibot those last days, in order to get it ready by > mid-october. My last mail was back to june, since then, I was busy on > many other things. > > So... > > > 1) Transaction support > > It has been greatly improved, and it's now working for Browse, add and > Delete. That means it's now completed, which was the main feature that > was missing. > > Transactions are cross-B-tree, and many operations can be applied within > a transaction. For the record, if you update a B-tree 100 times within a > single transaction, instead of committing the transaction 100 times, > updates goes 40 times faster. On my machine, storing 100 000 long and > the associated value as a String (ie <104, "104">) takes 36 seconds if I > commit the transaction for each tuple being added (2750 updates/s), and > 0.9s if I commit every 100 updates (108 500 updates/s). > > I still have to clean up the code, and to add some more tests. > > FTR, I blogged about Transactions on > https://blogs.apache.org/directory/entry/mavibot-transaction > > > 2) Free Page management > > To make it simpler, I just put aside this aspect while working on > transaction. I still have to get it back in the code, and more > important, have the page-reclaimer working. That will require a bit of work > > > 3) Cache > > Currently, the code does not use any cache. I have remove the B-tree > cache atm. > > The idea is to have oneglobal cache, based on page offsets. Currently we > use a PageHolder in Nodes, I may replace it by an offset to the page it > refers to, the cache will hld the page or will grab it from disk. > > > Anyway, I'll work on that aspect soon and will update you with what I > come with. > > > 4) Bulk Loader > > This has to be reviewed, accordingly to the previous changes. > > > > All in all, the new code is more compact, as I removed many features > that were not really useful atm, like : > > - sub-btree support > > - per B-tree cache > > - in-memory B-trees support > > > More to come soon ! > > > > -- > Emmanuel Lecharny > > Symas.com > directory.apache.org > >
