[
https://issues.apache.org/jira/browse/HBASE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14983338#comment-14983338
]
stack commented on HBASE-14540:
-------------------------------
Bumping threads to 100 bumps the write throughput only slightly. Threads are
doing this kinda stuff... waiting:
{code}
Thread 102 (PriorityRpcServer.handler=19,queue=1,port=16020):
State: TIMED_WAITING
Blocked count: 78747
Waited count: 110565
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1356)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1350)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1456)
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7901)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3150)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
Thread 101 (PriorityRpcServer.handler=18,queue=0,port=16020):
State: TIMED_WAITING
Blocked count: 78786
Waited count: 110881
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:167)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.blockOnSync(FSHLog.java:1356)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.publishSyncThenBlockOnCompletion(FSHLog.java:1350)
org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1456)
org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:7901)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3150)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
Thread 100 (PriorityRpcServer.handler=17,queue=1,port=16020):
State: TIMED_WAITING
Blocked count: 78899
Waited count: 110706
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.waitForRead(MultiVersionConcurrencyControl.java:218)
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.completeAndWait(MultiVersionConcurrencyControl.java:149)
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3166)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2812)
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2754)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:779)
org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:741)
org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2156)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32462)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2179)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
java.lang.Thread.run(Thread.java:745)
{code}
I tried to go to 300 threads with 100 clients on one of three servers but won't
go above half the write rate of default strategy. Maybe I'm doing something
wrong or the fact tthat this is apache hbase branch-1?
> Write Ahead Log Batching Optimization
> -------------------------------------
>
> Key: HBASE-14540
> URL: https://issues.apache.org/jira/browse/HBASE-14540
> Project: HBase
> Issue Type: Improvement
> Reporter: John Leach
> Assignee: John Leach
> Attachments: 14540.txt, HBaseWALBlockingWaitStrategy.java, writes.png
>
>
> The new write ahead log mechanism seems to batch too few mutations when
> running inside the disruptor. As we scaled our load up (many threads with
> small writes), we saw the number of hdfs sync operations grow in concert with
> the number of writes. Generally, one would expect the size of the batches to
> grow but the number of actual sync operations to settle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
