Date: Fri, 3 May 2013 12:05:14 -0600
From: Jon Meredith <[email protected]>
To: Brian Hong <[email protected]>
Hi Brian,
Experimental backends for Riak are always exciting. I haven't played with
it personally, and Basho has no current plans to support it as a leveldb
alternative.
It's worth adding two notes of caution. First, stores that use mmap for
persistence can suffer from problems around dirty pages. If you have a
very low update volume and a nice hot set that the operating system can
keep the pages in memory they work nicely.
LMDB doesn't accumulate dirty pages. In its default mode all writes are fully
synchronous and thus fsync'd immediately.
On some operating systems (specifically Linux), if you have a high update
load, and consequently a large volume of dirty pages (more than
dirty_ratio), I believe all OS level threads for the process are suspended
until the condition is resolved by writing out the pages when the process
is scheduled.
This is bad for latencies in endurance tests. For something like LDAP
this tradeoff is probably a good one, for Riak it concerns me, and other
platforms may be better behaved to make the option interesting, and the
linux kernel may have begun
Second, I worry about crash resilience - can the internal memory structures
tolerate a kernel panic where the dirty pages are not written, or
potentially worse torn with a partial write.
LMDB uses copy-on-write. It is impossible to corrupt the DB structure in a
crash. This has already been proven in heavy-duty testing at a couple of
telcos over the course of several months. Judging from the traffic on this
list, LevelDB is nowhere near carrier-grade reliability.
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-March/011320.html
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-February/011246.html
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-January/010692.html
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-November/010171.html
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-November/010287.html
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2012-September/009351.html
Good luck with your experiments,
Jon
On Wed, May 1, 2013 at 11:05 AM, Brian Hong <[email protected]> wrote:
OpenLDAP Lightning Memory-Mapped Database seems to be getting traction for
it's high performance and similar query (iteration) functionality with
leveldb:
http://symas.com/mdb/
There seems to be an experimental backend for Riak:
https://github.com/alepharchives/emdb
Does anybody know of it's usefulness? Is there any benchmarks on it?
I've heard that leveldb suffers write performance issues:
https://news.ycombinator.com/item?id=5621884
Any chances of Basho guys supporting lmdb as an leveldb alternative to
Riak? It would be awesome! :D
--
Brian(Sung-jin) Hong
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
