[
https://issues.apache.org/jira/browse/HBASE-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-3128:
-------------------------
Attachment: 3128.txt
This exception is way louder than its bite. In fact, its harmless -- not the
reason for the test failure.
The noise comes of the fact that we had weird catch inside in ServerManager
openRegion where we'd print exception, convert IOE to a RuntimeException then
up in AssignmentManager assign we'd catch this exception, print it out again,
then go reassign but to a different server. Here's the log after the big spew:
{code}
2010-10-19 13:29:56,117 DEBUG [Master:0;172.24.152.112:51845]
master.AssignmentManager(801): No previous transition plan for
.META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=,
dest=172.24.152.112,51852,1287520190000; 3 (online=3,
exclude=serverName=172.24.152.112,51850,1287520189982, load=(requests=0,
regions=0, usedHeap=60, maxHeap=123)) available servers
{code}
This happens a few times in this log.
Actual issue that broke test might be this:
{code}
2010-10-19 13:29:56,235 INFO [Master:0;172.24.152.112:51845]
master.AssignmentManager(1004): Bulk assigning done
2010-10-19 13:29:56,237 DEBUG [MASTER_OPEN_REGION-172.24.152.112:51845-1]
handler.OpenedRegionHandler(87): Handling OPENED event for
f98a67e5ccb2d3d04e8c1d20a71c20db; deleting unassigned node
2010-10-19 13:29:56,237 DEBUG [MASTER_OPEN_REGION-172.24.152.112:51845-1]
zookeeper.ZKAssign(349): master:51845-0x12bc62e746c0014 Deleting existing
unassigned node for f98a67e5ccb2d3d04e8c1d20a71c20db that is in expected state
RS_ZK_REGION_OPENED
2010-10-19 13:29:56,238 DEBUG [MASTER_OPEN_REGION-172.24.152.112:51845-1]
zookeeper.ZKUtil(1012): master:51845-0x12bc62e746c0014 Retrieved 112 byte(s) of
data from znode /hbase/unassigned/f98a67e5ccb2d3d04e8c1d20a71c20db; data=
> On assign, if ConnectException, reassign another server
> -------------------------------------------------------
>
> Key: HBASE-3128
> URL: https://issues.apache.org/jira/browse/HBASE-3128
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.90.0
>
> Attachments: 3128.txt
>
>
> {code}
> 2010-10-19 13:29:56,116 DEBUG [Thread-216-EventThread]
> master.AssignmentManager(321): Handling transition=M_ZK_REGION_OFFLINE,
> server=172.24.152.112:51845, region=1028785192/.META.
> 2010-10-19 13:29:56,116 ERROR [Master:0;172.24.152.112:51845]
> master.ServerManager(573): Error connecting to region server
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up
> proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to
> /172.24.152.112:51850 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:351)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:936)
> at
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:568)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:511)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:726)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:640)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(AssignmentManager.java:935)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:430)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:377)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:266)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:310)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:860)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:728)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy10.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:412)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:388)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:435)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:345)
> ... 10 more
> 2010-10-19 13:29:56,116 WARN [Master:0;172.24.152.112:51845]
> master.AssignmentManager(730): Failed assignment of .META.,,1.1028785192 to
> serverName=172.24.152.112,51850,1287520189982, load=(requests=0, regions=0,
> usedHeap=60, maxHeap=123)
> java.lang.RuntimeException: Fatal error connection to RS
> at
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:574)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:511)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:726)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:640)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assignMeta(AssignmentManager.java:935)
> at
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:430)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:377)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:266)
> at java.lang.Thread.run(Thread.java:637)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to
> /172.24.152.112:51850 after attempts=1
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:351)
> at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:936)
> at
> org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:568)
> ... 8 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:310)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:860)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:728)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:255)
> at $Proxy10.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:412)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:388)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:435)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:345)
> ... 10 more
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
