[
https://issues.apache.org/jira/browse/SAMZA-543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299394#comment-14299394
]
Chris Riccomini commented on SAMZA-543:
---------------------------------------
[~theduderog], I ran a little test. WAL disabling seems to have a large effect
on performance. My SSD is at ~170MB/s both with and without WAL disabled (this
is well below it's 400MB/s peak that it should be able to do, mind you), but
the records/sec is dramatically increased.
With WAL:
{noformat}
100000 rows in 649ms
200000 rows in 1276ms
300000 rows in 1894ms
400000 rows in 2522ms
500000 rows in 3177ms
600000 rows in 3849ms
700000 rows in 4500ms
800000 rows in 5133ms
900000 rows in 5770ms
1000000 rows in 6439ms
1100000 rows in 7145ms
1200000 rows in 7937ms
1300000 rows in 8596ms
1400000 rows in 9236ms
1500000 rows in 9913ms
{noformat}
Without WAL:
{noformat}
100000 rows in 224ms
200000 rows in 434ms
300000 rows in 629ms
400000 rows in 828ms
500000 rows in 1180ms
600000 rows in 1465ms
700000 rows in 1650ms
800000 rows in 1855ms
900000 rows in 2045ms
1000000 rows in 3458ms
1100000 rows in 3654ms
1200000 rows in 3857ms
1300000 rows in 4290ms
1400000 rows in 4484ms
1500000 rows in 4774ms
{noformat}
This isn't meant to be an exhaustive proof, but it does show ~2x improvement
when WAL is turned off. You can turn it off with this:
{code}
val writeOptions = new WriteOptions
writeOptions.setDisableWAL(true)
db.put(writeOptions, key, value)
{code}
I manually edited RocksDbKeyValueStore to do this, as a test.
> Disable WAL in RocksDB KV store
> -------------------------------
>
> Key: SAMZA-543
> URL: https://issues.apache.org/jira/browse/SAMZA-543
> Project: Samza
> Issue Type: Bug
> Components: kv
> Affects Versions: 0.9.0
> Reporter: Chris Riccomini
> Fix For: 0.9.0
>
>
> RocksDB uses a write-ahead log by default. This is unnecessary in Samza,
> since we have full durability from a state store's changelog topic. We should
> [disable the
> WAL|https://github.com/facebook/rocksdb/wiki/Basic-Operations#asynchronous-writes]
> in the RocksDB KV store.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
