[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644676#comment-13644676
]
Nicolas Liochon commented on HBASE-6295:
----------------------------------------
Example of exception (with 6295 on):
{noformat}
2013-04-29 12:05:51,515 DEBUG [IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root] ipc.HBaseClient:
IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root: got response
header exception { exceptionClassName:
"org.apache.hadoop.hbase.exceptions.RegionTooBusyException" stackTrace:
"org.apache.hadoop.hbase.exceptions.RegionTooBusyException: region is
flushing\n\tat
org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2477)\n\tat
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:1869)\n\tat
org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:3822)\n\tat
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3237)\n\tat
sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)\n\tat
java.lang.reflect.Method.invoke(Method.java:597)\n\tat
org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:174)\n\tat
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1874)\n" }
2013-04-29 12:05:51,519 DEBUG [IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root] ipc.HBaseClient:
IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root: closing ipc
connection to ip-10-68-155-141.ec2.internal/10.68.155.141:60020: Protocol
message tag had invalid wire type.
com.google.protobuf.InvalidProtocolBufferException: Protocol message tag had
invalid wire type.
at
com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:78)
at
com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:498)
at
com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438)
at
org.apache.hadoop.hbase.protobuf.generated.RPCProtos$ExceptionResponse$Builder.mergeFrom(RPCProtos.java:2225)
at
org.apache.hadoop.hbase.protobuf.generated.RPCProtos$ExceptionResponse$Builder.mergeFrom(RPCProtos.java:2071)
at
com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:275)
at
org.apache.hadoop.hbase.protobuf.generated.RPCProtos$ResponseHeader$Builder.mergeFrom(RPCProtos.java:3713)
at
org.apache.hadoop.hbase.protobuf.generated.RPCProtos$ResponseHeader$Builder.mergeFrom(RPCProtos.java:3541)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
at
com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
at
com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
at
com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
at
com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
at
org.apache.hadoop.hbase.protobuf.generated.RPCProtos$ResponseHeader.parseDelimitedFrom(RPCProtos.java:3498)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.readResponse(HBaseClient.java:994)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:661)
2013-04-29 12:05:51,521 DEBUG [IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root] ipc.HBaseClient:
IPC Client (1280551684) connection to
ip-10-68-155-141.ec2.internal/10.68.155.141:60020 from root: closed
{noformat}
I'm unclear on the impact, but at a point it stops working, so it could have
one.
> Possible performance improvement in client batch operations: presplit and
> send in background
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
> Affects Versions: 0.95.2
> Reporter: Nicolas Liochon
> Assignee: Nicolas Liochon
> Labels: noob
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch,
> 6295.v4.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
