[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13636364#comment-13636364
]
Nicolas Liochon commented on HBASE-6295:
----------------------------------------
It seems that in this test, on AWS, we're limited by the bandwidth between the
client and the region server. The 25% is the time spent by the client waiting
for an answer from the servers.
This stack occurs sometimes with plain trunk and very often with 6295:
{noformat}
2013-04-19 09:22:45,991 WARN [IPC Client (682317035) connection to
ip-10-6-131-32.ec2.internal/10.6.131.32:60020 from root] ipc.HBaseClient
(HBaseClient.java:run(664)) - IPC Client (682317035) connection to
ip-10-6-131-32.ec2.internal/10.6.131.32:60020 from root: unexpected exception
receiving call responses
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.readResponse(HBaseClient.java:1017)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:661)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.readResponse(HBaseClient.java:1013)
... 1 more
1197 sec: 3411081 operations; 324,27 current ops/sec; [INSERT
AverageLatency(us)=29332,6]
{noformat}
[[email protected]], is this code changed / removed by the refactoring in
progress in the client?
> Possible performance improvement in client batch operations: presplit and
> send in background
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
> Affects Versions: 0.95.2
> Reporter: Nicolas Liochon
> Assignee: Nicolas Liochon
> Labels: noob
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
