Re: Possibility of replacing SQLite with LMDB from the OpenLDAP project (or replacing the SQLite backend with LMDB)
On Wednesday, July 10, 2013 8:42:42 AM UTC-7, Marco Bonardo wrote: On 06/07/2013 11:26, Philip Chee wrote: LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). It looks like LMDB has speed and size advantages on memory constrained systems like Android and B2G. Ehr, I wrongly replied instead of following-up, let me paste again: As for many other dbms around, comparisons are pretty much hard, just relying on microbenchmarking doesn't help much. What's best, LMDB, levelDB, kyotoCabinet, Sqlite4? It's hard to tell just by looking at these graphs, you'd need measurements done directly on your most compelling use-cases. Just getting those measures is a project by itself. First you need to distinguish SQLite (3 or 4) from the others, which are simply key/value stores. Do you really need complex multi-column tables and complex SQL queries for managing your data? I don't see why; you would probably do much better with just a K/V store. As for which is better - it's no contest. KyotoCabinet leaks pages like a sieve. http://www.anchor.com.au/blog/2013/05/second-strike-with-lightning/ Test LMDB in your own applications, you'll probably find that it massively outperforms whatever you were using before, as many others have. https://github.com/tspurway/pymdb-lightning Then there's the cost of conversion and added code size, for which Sqlite4 may be a winner, considered we'd need less changes and we'd likely have more code reuse from Sqlite3. Then there's memshrink, I couldn't find the maximum and average used memory in the reported benchmarks, we are currently using a max of 2MB per DB connection, would it be comparable? I finally wonder how they solved the problem of mmap corruption, also Sqlite can use mmap IO, and it may be up to ten times faster, but has no protection for memory corruption due to stray pointers or buffer overflows in the application. LMDB is protected from all of these problems. It uses a read-only memory map, so stray pointer writes will kill the app with SEGV, nothing corrupts the data on disk. So, it's definitely an option to consider when we decide to start working on a new generic storage, but we need far more details to be able to properly evaluate. Fwiw, Sqlite4 will support swappable engines, so you can use lmdb as the Sqlite4 engine, and a backend is already under development from what I read on the lmdb page. Yes, but SQLite4 is very very far away from ready for public consumption. The storage engine interface is pretty simple, plugging LMDB in is trivial. But until the frontend work is done (query optimizers and all that) it's not going to do anyone any good. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Possibility of replacing SQLite with LMDB from the OpenLDAP project (or replacing the SQLite backend with LMDB)
On Saturday, July 6, 2013 2:26:27 AM UTC-7, Philip Chee wrote: 1. OpenLDAP Lightning Memory-Mapped Database (LMDB): http://symas.com/mdb/ 2. Benchmarks: http://symas.com/mdb/microbench/ 3. SQLite3 ported to use MDB instead of its original Btree code: https://gitorious.org/mdb/sqlightning Fwiw, I currently run my own build of Seamonkey using SQLightning on my laptops. I also have a custom build of XDAndroid running on my HTC TouchPro2 using this as well. There's a noticeable improvement in battery life. LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). It looks like LMDB has speed and size advantages on memory constrained systems like Android and B2G. Cons: 1. Still very new. I have no idea what their code coverage looks like. 2. Uses their own OpenLDAP Public Licence (a BSD variant). There are claims that this is OSI-approved, however the OSI site itself doesn't list it. 3. Migration costs (If it ain't broke, don't fix it) not to mention more churn. Phil -- Philip Chee phi...@aleytys.pc.my, philip.c...@gmail.com http://flashblock.mozdev.org/ http://xsidebar.mozdev.org Guard us from the she-wolf and the wolf, and guard us from the thief, oh Night, and so be good for us to pass. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Possibility of replacing SQLite with LMDB from the OpenLDAP project (or replacing the SQLite backend with LMDB)
1. OpenLDAP Lightning Memory-Mapped Database (LMDB): http://symas.com/mdb/ 2. Benchmarks: http://symas.com/mdb/microbench/ 3. SQLite3 ported to use MDB instead of its original Btree code: https://gitorious.org/mdb/sqlightning LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). It looks like LMDB has speed and size advantages on memory constrained systems like Android and B2G. Cons: 1. Still very new. I have no idea what their code coverage looks like. 2. Uses their own OpenLDAP Public Licence (a BSD variant). There are claims that this is OSI-approved, however the OSI site itself doesn't list it. 3. Migration costs (If it ain't broke, don't fix it) not to mention more churn. Phil -- Philip Chee phi...@aleytys.pc.my, philip.c...@gmail.com http://flashblock.mozdev.org/ http://xsidebar.mozdev.org Guard us from the she-wolf and the wolf, and guard us from the thief, oh Night, and so be good for us to pass. ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Possibility of replacing SQLite with LMDB from the OpenLDAP project (or replacing the SQLite backend with LMDB)
On 06/07/2013 11:26, Philip Chee wrote: LMDB is an ultra-fast, ultra-compact key-value data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases, and is only limited to the size of the virtual address space, (it is not limited to the size of physical RAM). It looks like LMDB has speed and size advantages on memory constrained systems like Android and B2G. Ehr, I wrongly replied instead of following-up, let me paste again: As for many other dbms around, comparisons are pretty much hard, just relying on microbenchmarking doesn't help much. What's best, LMDB, levelDB, kyotoCabinet, Sqlite4? It's hard to tell just by looking at these graphs, you'd need measurements done directly on your most compelling use-cases. Just getting those measures is a project by itself. Then there's the cost of conversion and added code size, for which Sqlite4 may be a winner, considered we'd need less changes and we'd likely have more code reuse from Sqlite3. Then there's memshrink, I couldn't find the maximum and average used memory in the reported benchmarks, we are currently using a max of 2MB per DB connection, would it be comparable? I finally wonder how they solved the problem of mmap corruption, also Sqlite can use mmap IO, and it may be up to ten times faster, but has no protection for memory corruption due to stray pointers or buffer overflows in the application. So, it's definitely an option to consider when we decide to start working on a new generic storage, but we need far more details to be able to properly evaluate. Fwiw, Sqlite4 will support swappable engines, so you can use lmdb as the Sqlite4 engine, and a backend is already under development from what I read on the lmdb page. -m ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
