[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13698756#comment-13698756
]
Nicolas Liochon commented on HBASE-6295:
----------------------------------------
1) Settings
I don't think that the settings are good today in the trunk.
In hbase-common/resources; we still have:
- hbase.client.pause = 100 (It's 1000 in the code)
- hbase.client.retries.number = 14 (it's 20 in the code)
As a consequence, we retry for ~30s before failing.
We see this in Elliott's log above.
client.AsyncProcess: Attempt 10/14 failed for 1 operations [...]- sleeping 3201
ms.
=> the max retry is 14, not 20.
=> after 10 failure, we still sleep for only 3.2 seconds.
I personally think that we should get rid of the hbase-*.xml in our package to
be sure we're using the code defaults. Today, we have:
- the defaults in the code
- hbase-defaults.xml in hbase-common (seems to be used when do the integration
test with a cluster)
- hbase-site.xml in hbase-server/test (seems to be used when you run the
integration test with a minicluster)
- hbase-site.xml in hbase-client
- hbase-site.xml in conf
2) Previously, when we were sending a single multi call with 100 puts in it to
the server and it was failing we were counting 100 errors. We now count 1
error. The previous behavior was a bug, but it's consequence is that the
backoff time was always the max, hiding the impact of the first point
(basically we were sleeping 14 times 6 seconds, while now, during the first
attempts we sleep 100 ms)
3) For completeness, this patch degrades the MTTR, especially in tests. Two
reasons for this. First, the clients send more writes to the server: that's why
we have better performances with YCSB. Second, during a failure, the clients
can still send writes to the other regions instead of waiting for the result of
a write on the recovering regions. This means that there are less resources
available to do the recovery, as the clients are not stuck anymore and are
sending more writes. This is especially visible in tests, because we try to
write as much as possible (while in real life the writes are more immutable as
they are coming from an external source).
I'm not saying that there is not bug in the code I committed. It's complex and
it's not production proven, so I'm actually quite sure there are bugs :-). What
I'm saying is
- on my tests, when the integration test was failing is was because we reached
the maximum number of retries.
- finding out what are the actual settings used is pure hell
- and I was expecting that HBASE-8776 would do this.
I'm gonna hijack HBASE-8776 to propose some settings.
> Possible performance improvement in client batch operations: presplit and
> send in background
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
> Affects Versions: 0.95.2
> Reporter: Nicolas Liochon
> Assignee: Nicolas Liochon
> Labels: noob
> Fix For: 0.98.0, 0.95.2
>
> Attachments: 6295.addendum.patch, 6295.v11.patch, 6295.v12.patch,
> 6295.v14.patch, 6295.v15.patch, 6295.v1.patch, 6295.v2.patch, 6295.v3.patch,
> 6295.v4.patch, 6295.v5.patch, 6295.v6.patch, 6295.v8.patch, 6295.v9.patch,
> hbase-ycsb-workloads Build time trend.png
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
