[
https://issues.apache.org/jira/browse/HBASE-576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636791#action_12636791
]
stack commented on HBASE-576:
-----------------------------
Above comment 'stack - 02/Oct/08 10:48 PM' is incorrect. In fact it's a Client
per socketfactory rather than a Client per remote host. A Client goes against
many hosts. For each, Client keeps a cache of Connections keyed by remote
address (and then some). An invocation on the remote host sends the request
then sleeps till the response comes back. During sending and receipt of
response, the connection is devoted but otherwise, is available. Unless the
request or response large, socket is being 'multiplexed' (if large, need to
chunk request/response).
Here's sample from a log with the RPC logging enabled plus some logging I
added. The log is of 4 threads running in single JVM each running
PerformanceEvaluation test (need above patch applied). The lone number after
the name of the class is the thread name. Threads were named 0, 1, 2, and 3.
The numbers at the end with the '####' in front are request ids. See how they
are interlaced.
{code}
...
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1330
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1327
08/10/03 23:16:03 DEBUG ipc.Client: 0 FINISHED WAITING ON ###1327
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 80
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1331
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1328
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1329
08/10/03 23:16:03 DEBUG ipc.Client: 1 FINISHED WAITING ON ###1328
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.Client: 2 FINISHED WAITING ON ###1329
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 97
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 98
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1332
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1333
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1330
08/10/03 23:16:03 DEBUG ipc.Client: 3 FINISHED WAITING ON ###1330
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 99
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1334
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1331
08/10/03 23:16:03 DEBUG ipc.Client: 0 FINISHED WAITING ON ###1331
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 113
08/10/03 23:16:03 INFO ipc.HBaseClient: 0 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1335
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1332
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1333
08/10/03 23:16:03 DEBUG ipc.Client: 1 FINISHED WAITING ON ###1332
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.Client: 2 FINISHED WAITING ON ###1333
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 178
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 [EMAIL PROTECTED] done
08/10/03 23:16:03 DEBUG ipc.HbaseRPC: Call: get 178
08/10/03 23:16:03 INFO ipc.HBaseClient: 1 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 INFO ipc.HBaseClient: 2 [EMAIL PROTECTED] start
/208.76.44.139:60020
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1336
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user sending #####1337
08/10/03 23:16:03 DEBUG ipc.Client: IPC Client (47) connection to
/208.76.44.139:60020 from an unknown user got value #####1334
08/10/03 23:16:03 DEBUG ipc.Client: 3 FINISHED WAITING ON ###1334
08/10/03 23:16:03 INFO ipc.HBaseClient: 3 [EMAIL PROTECTED] done
...
{code}
Hadoop RPC doesn't seem to have a 'big lock' at its core, not unless the
request or response large. Its pretty concurrent.
Will look now at trying to hack in grizzly in the least intrusive manner just
to see if grizzly is faster.
> Investigate IPC performance
> ---------------------------
>
> Key: HBASE-576
> URL: https://issues.apache.org/jira/browse/HBASE-576
> Project: Hadoop HBase
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.1.0, 0.1.1, 0.1.2, 0.2.0
> Reporter: Jim Kellerman
> Assignee: stack
> Attachments: pe.patch
>
>
> Turning off all file I/O, and running the PerformanceEvaluation test, of
> 1,048,576 sequential writes to HBase managed to achieve only 7,285 IPCs per
> second.
> Running PerformanceEvaluation sequential write test modified to do an abort
> instead of a commit, it was possible to do 68,337 operations per second. We
> are obviously spending a lot of time doing IPCs.
> We need to investigate to find the bottleneck. Marshalling and unmarshalling?
> Socket setup and teardown?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
