reposting reply originally at: https://news.ycombinator.com/item?id=5833414
As the person doing the github:basho/leveldb work, I agree with the above (very professional reply, not something you expect to see on the Internet, thank you). We are optimizing to our individual environments. I do need to review the compaction algorithms to see if there is benefit to Basho. However, it will be a few weeks before that happens. The write multi-threading does not help Basho's Riak 1.x series. We parallelize by using multiple leveldb database (Riak term vnodes) and do NOT parallelize individual database until 2.0. Therefore, unfair to measure that feature agains Riak. Our compaction code is adjusted with an emphasis on running multiple compactions of varied priority. Our use of multiple databases got hung up on leveldb's single compaction thread. Again, this complete difference of environments would be unfair in a Hyperleveldb direct comparison. And the most difficult issue for dropping hyper into Riak is that our leveldb performs the write throttling, hyperleveldb leaves that to HyperDex. This is yet again an environment design decision ... but says coding is require to make hyperleveldb "just work" with Riak. That will be a while. I therefore do not claim that Basho's leveldb would be better with HyperDex and suspect that today's hyperleveldb would not be better with Basho's Riak. We optimized to our different pain points. Matthew On Jun 6, 2013, at 3:08 PM, Alexander Sicular <[email protected]> wrote: > If you haven't seen it on HN [0], Hyperdex just dropped a leveldb fork they > are calling HyperLevelDB [1]. It seems like they are looking for drop-in > replacement compatibility. I'm interested to hear thoughts from the Basho > folks (or anyone else) once they've had a chance to look it over. One of > their specific goals is maintaining performance during compaction which I > know has been an issue for some people using the leveldb backend. > > If you read their post there seems to be three things they are doing to reach > their higher performance: > > 1. multi-threaded synchronized write ordering. > "LevelDB uses very coarse-grained synchronization which forces all writes to > proceed in an ordered, first-come-first-served fashion, effectively reducing > throughput to that of a single thread. HyperLevelDB increases concurrency by > allowing multiple threads to agree on the order of their respective writes, > and then independently apply the writes in a manner consistent with the > agreed-upon order." > > 2. tuned write delay on compaction. > "HyperLevelDB removes this artificial delay, allowing the application (in our > case, HyperDex) to independently decide to delay writes, using information > available outside the scope of LevelDB." > > 3. tuned intra-level re-writes. > "LevelDB's compaction algorithm is not efficient, and in the "fillrand" > benchmark will, on average, rewrite 3MB of data in the upper level for every > 1MB of data in the lower level. HyperLevelDB avoids this waste by selecting > the compaction with the smallest overhead." > > > Thanks, > -Alexander Sicular > > @siculars > > > [0] https://news.ycombinator.com/item?id=5833414 > [1] http://hyperdex.org/performance/leveldb/. > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
