[
https://issues.apache.org/jira/browse/ACCUMULO-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Park updated ACCUMULO-2668:
------------------------------------
Attachment: ACCUMULO-2668.0.patch.txt
Reuploading file from format-patch command
microbenchmark:
- Ran continuous ingest on my laptop (2013 mbp: 2.6 GHz quad core i7, 16 GB
RAM) using default 3GB accumulo config using native maps. Used a single
continuous ingester instance against a table with 4 tablets.
results:
with fix: 120K entries/s, 12.66 MB/s
without fix: 83K entries/s, 9.05 MB/s
#s were obtained at some point in time during the ingest
> slow WAL writes
> ---------------
>
> Key: ACCUMULO-2668
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2668
> Project: Accumulo
> Issue Type: Bug
> Affects Versions: 1.6.0
> Reporter: Jonathan Park
> Assignee: Jonathan Park
> Priority: Blocker
> Labels: 16_qa_bug
> Fix For: 1.6.1
>
> Attachments: ACCUMULO-2668.0.patch.txt, noflush.diff
>
>
> During continuous ingest, we saw over 70% of our ingest time taken up by
> writes to the WAL. When we ran the DfsLogger in isolation (created one
> outside of the Tserver), we saw about ~25MB/s throughput as opposed to nearly
> 100MB/s from just writing directly to an hdfs outputstream (computed by
> taking the estimated size of the mutations sent to the DfsLogger class
> divided by the time it took for it to flush + sync the data to HDFS).
> After investigating, we found one possible culprit was the
> NoFlushOutputStream. It is a subclass of java.io.FilterOutputStream but does
> not override the write(byte[], int, int) method signature. The javadoc
> indicates that subclasses of the FilterOutputStream should provide a more
> efficient implementation.
> I've attached a small diff that illustrates and addresses the issue but this
> may not be how we ultimately want to fix it.
> As a side note, I may be misreading the implementation of DfsLogger, but it
> looks like we always make use of the NoFlushOutputStream, even if encryption
> isn't enabled. There appears to be a faulty check in the DfsLogger.open()
> implementation that I don't believe can be satisfied (line 384).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
