[
https://issues.apache.org/jira/browse/HBASE-14431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900725#comment-14900725
]
Yu Li commented on HBASE-14431:
-------------------------------
>From the HadoopQA report of HBASE-14448, I found TestFastFail failed with
>below log:
{noformat}
2015-09-21 11:42:58,768 WARN [AsyncRpcChannel-pool2-t17]
logging.Slf4JLogger(151): An exception was thrown by
org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete()
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.AsyncRpcClient.removeConnection(AsyncRpcClient.java:406)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.close(AsyncRpcChannel.java:537)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.retryOrClose(AsyncRpcChannel.java:300)
at
org.apache.hadoop.hbase.ipc.AsyncRpcChannel.access$200(AsyncRpcChannel.java:82)
{noformat}
Checking line 406 of AsyncRpcClient.java, we could find below changes in this
JIRA:
{noformat}
- int connectionHashCode = connection.getConnectionHashCode();
+ int connectionHashCode = connection.hashCode();
synchronized (connections) {
// we use address as cache key, so we should check here to prevent
removing the
// wrong connection
AsyncRpcChannel connectionInPool =
this.connections.get(connectionHashCode);
- if (connectionInPool == connection) {
+ if (connectionInPool.equals(connection)) {
{noformat}
And line 406 is
{code}
if (connectionInPool.equals(connection)) {
{code}
I think we lack a null pointer check here, and attached is a straight addendum.
> AsyncRpcClient#removeConnection() never removes connection from connections
> pool if server fails
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-14431
> URL: https://issues.apache.org/jira/browse/HBASE-14431
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
> Affects Versions: 2.0.0, 1.0.2, 1.1.2
> Reporter: Samir Ahmic
> Assignee: Samir Ahmic
> Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3
>
> Attachments: HBASE-14431-v2.patch, HBASE-14431.patch
>
>
> I was playing with master branch in distributed mode (3 rs + master +
> backup_master) and notice strange behavior when i was testing this sequence
> of events on single rs: /kill/start/run_balancer while client was writing
> data to cluster (LoadTestTool).
> I have notice that LTT fails with following:
> {code}
> 2015-09-09 11:05:58,364 INFO [main] client.AsyncProcess: #2, waiting for
> some tasks to finish. Expected max=0, tasksInProgress=35
> Exception in thread "main"
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1
> action: BindException: 1 time,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:228)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:208)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1697)
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:211)
> {code}
> After some digging and adding some more logging in code i have notice that
> following condition in {code}AsyncRpcClient.removeConnection(AsyncRpcChannel
> connection) {code} is never true:
> {code}
> if (connectionInPool == connection) {
> {code}
> causing that {code}AsyncRpcChannel{code} connection is never removed from
> {code}connections{code} pool in case rs fails.
> After changing this condition to:
> {code}
> if (connectionInPool.address.equals(connection.address)) {
> {code}
> issue was resolved and client was removing failed server from connections
> pool.
> I will attach patch after running some more tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
