[
https://issues.apache.org/jira/browse/ACCUMULO-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Vines updated ACCUMULO-1083:
---------------------------------
Fix Version/s: (was: 1.6.0)
1.7.0
> add concurrency to HDFS write-ahead log
> ---------------------------------------
>
> Key: ACCUMULO-1083
> URL: https://issues.apache.org/jira/browse/ACCUMULO-1083
> Project: Accumulo
> Issue Type: Improvement
> Components: tserver
> Reporter: Adam Fuchs
> Fix For: 1.7.0
>
> Attachments: walog-performance.jpg,
> walog-replication-factor-performance.jpg
>
>
> When running tablet servers on beefy nodes (lots of disks), the write-ahead
> log can be a serious bottleneck. Today we ran a continuous ingest test of
> 1.5-SNAPSHOT on an 8-node (plus a master node) cluster in which the nodes had
> 32 cores and 15 drives each. Running with write-ahead log off resulted in a
> >4x performance improvement sustained over a long period.
> I believe the culprit is that the WAL is only using one file at a time per
> tablet server, which means HDFS is only appending to one drive (plus
> replicas). If we increase the number of concurrent WAL files supported on a
> tablet server we could probably drastically improve the performance on
> systems with many disks. As it stands, I believe Accumulo is significantly
> more optimized for a larger number of smaller nodes (3-4 drives).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
