[
https://issues.apache.org/jira/browse/HBASE-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14712316#comment-14712316
]
Elliott Clark commented on HBASE-14313:
---------------------------------------
Just saw this during a deploy. A thrift proxy restarted, then a while later the
regionserver on the same host is restarted. After that the HConnection the
thrift proxy is using gets stuck and will not make any progress.
At 5:10 the regionserver is restarted.
At 5:50 the hconnection is still not healed so the thrift server is restarted.
{code}
15/08/25 17:56:45 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=10/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=10021ms, replay=55ops
15/08/25 17:56:45 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=10
15/08/25 17:56:55 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=11/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=10008ms, replay=55ops
15/08/25 17:56:55 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=11
15/08/25 17:57:05 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=12/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20164ms, replay=55ops
15/08/25 17:57:05 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=12
15/08/25 17:57:25 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=13/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20070ms, replay=55ops
15/08/25 17:57:25 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=13
15/08/25 17:57:45 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=14/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20096ms, replay=55ops
15/08/25 17:57:45 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=14
15/08/25 17:58:05 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=15/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20107ms, replay=55ops
15/08/25 17:58:05 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=15
15/08/25 17:58:25 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=16/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20119ms, replay=55ops
15/08/25 17:58:25 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=16
15/08/25 17:58:45 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=17/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20148ms, replay=55ops
15/08/25 17:58:45 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=17
15/08/25 17:59:05 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=18/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20155ms, replay=55ops
15/08/25 17:59:05 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=18
15/08/25 17:59:25 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=19/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20016ms, replay=55ops
15/08/25 17:59:26 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=19
15/08/25 17:59:46 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=20/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20068ms, replay=55ops
15/08/25 17:59:46 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=20
15/08/25 18:00:06 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=21/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20055ms, replay=55ops
15/08/25 18:00:06 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=21
15/08/25 18:00:26 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=22/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20129ms, replay=55ops
15/08/25 18:00:26 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=22
15/08/25 18:00:46 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=23/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20011ms, replay=55ops
15/08/25 18:00:46 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=23
15/08/25 18:01:06 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=24/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20081ms, replay=55ops
15/08/25 18:01:06 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=24
15/08/25 18:01:26 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=25/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20003ms, replay=55ops
15/08/25 18:01:26 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=25
15/08/25 18:01:46 INFO client.AsyncProcess: #87993, table=my_table_name,
attempt=26/35 failed=55ops, last exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: IPC Client
(1114475730) connection to hbase1393.prn2.facebook.com/10.104.160.42:16020 from
hadoop is closing on hbase1393.prn2.facebook.com,16020,1440548090986, tracking
started null, retrying after=20137ms, replay=55ops
15/08/25 18:01:46 INFO client.AsyncProcess: #87993, waiting for some tasks to
finish. Expected max=0, tasksInProgress=26
{code}
> After a thrift-two proxy sees ConnectionClosingException on a connection it
> never recovers
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-14313
> URL: https://issues.apache.org/jira/browse/HBASE-14313
> Project: HBase
> Issue Type: Bug
> Reporter: Elliott Clark
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
