[ https://issues.apache.org/jira/browse/ACCUMULO-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15170131#comment-15170131 ]
ASF GitHub Bot commented on ACCUMULO-4154: ------------------------------------------ GitHub user keith-turner opened a pull request: https://github.com/apache/accumulo/pull/76 ACCUMULO-4154 add fate dump command You can merge this pull request into a Git repository by running: $ git pull https://github.com/keith-turner/accumulo ACCUMULO-4152 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/accumulo/pull/76.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #76 ---- commit f0752df8b62ead96080f2d1852f36c5bc1cedd8b Author: Keith Turner <ktur...@apache.org> Date: 2016-02-24T23:20:32Z ACCUMULO-4154 add fate dump command ---- > Improve batch writer > -------------------- > > Key: ACCUMULO-4154 > URL: https://issues.apache.org/jira/browse/ACCUMULO-4154 > Project: Accumulo > Issue Type: Improvement > Reporter: Keith Turner > > The batch writer currently has two drawbacks : > * It waits for its memory to be half full and then bins mutations for send > threads. I don't think this is optimal. Think it would be better to keep > the send threads busy. As soon as there are mutation start working on them. > If the send threads can not keep up, then work will naturally build up (w/o > waiting for memory to be .5 full) > * The flush method blocks threads trying to add anything to the batch writer. > Thinking of implementing the following model for the batch writer, which is > similar to how the conditional writer works. > * Have a queue that all incoming mutations are added to. > * Have a queue per tablet server > * Have a single thread thats constantly taking batches of mutations off the > incoming queue, binning them, and placing them on tablet server queues. > * When a send thread becomes idle, have it select and reserver the tablet > server queue with the most work on it. > * when mutations fail, send threads can add them back to the incoming queue > To get better flushing behavior, as each mutation is added to the batch > writer it can be assigned a one up counter. We can keep track of the > minimum in progress mutation. Flush can inspect this counter and wait for > the minimum active mutation to reach a certain count. -- This message was sent by Atlassian JIRA (v6.3.4#6332)