[
https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13596465#comment-13596465
]
John Vines edited comment on ACCUMULO-1083 at 3/7/13 10:23 PM:
---------------------------------------------------------------
I just finished running a handful of tests against different versions of
Accumulo with walogs on and off. We are running against hadoop-1.0.3.16 and
zookeeper-3.3.4.15.
Here are the continuous ingest results:
||version||table.walog.enabled||MB/s||entries||
|accumulo-1.4.2|true|6.691|1024000000|
|accumulo-1.4.2|false|23.583|4096000000|
|accumulo-1.5.0-SNAPSHOT|true|5.734|1024000000|
|accumulo-1.5.0-SNAPSHOT |false|27.321|4096000000|
The penalties for walogs are about 72% for 1.4.2 and 79% for 1.5.0. So, it's
not an extreme decrease in performance, but there is certainly a dip. We
suspect that the performance hit has to do with the Hadoop bottleneck described
above.
I have not modified the replication factor, but may run those tests too if time
permits.
was (Author: brassard):
I just finished running a handful of tests against different versions of
Accumulo with walogs on and off. We are running against hadoop-1.0.3.16 and
zookeeper-3.3.4.15.
Here are the results:
||version||table.walog.enabled||MB/s||
|accumulo-1.4.2|true|6.691|
|accumulo-1.4.2|false|23.583|
|accumulo-1.5.0-SNAPSHOT|true|5.734|
|accumulo-1.5.0-SNAPSHOT |false|27.321|
The penalties for walogs are about 72% for 1.4.2 and 79% for 1.5.0. So, it's
not an extreme decrease in performance, but there is certainly a dip. We
suspect that the performance hit has to do with the Hadoop bottleneck described
above.
I have not modified the replication factor, but may run those tests too if time
permits.
> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
> Key: ACCUMULO-1083
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Adam Fuchs
> Fix For: 1.6.0
>
> Attachments: walog-performance.jpg
>
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead
> log can be a serious bottleneck. Today we ran a test of 1.5-SNAPSHOT on an
> 8-node (plus a master node) cluster in which the nodes had 32 cores and 15
> drives each. Running with write-ahead log off resulted in a >4x performance
> improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per
> tablet server, which means HDFS is only appending to one drive (plus
> replicas). If we increase the number of concurrent WAL files supported on a
> tablet server we could probably drastically improve the performance on
> systems with many disks. As it stands, I believe Accumulo is significantly
> more optimized for a larger number of smaller nodes (3-4 drives).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
