[
https://issues.apache.org/jira/browse/HBASE-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688173#comment-13688173
]
rajeshbabu commented on HBASE-8667:
-----------------------------------
bq. When we did NOT supply where to bind, what was client using? Some default?
If we dont configure bind address then primary hostname will be used as bind
address.
Here are the test I have done,tried some permutations also.All cases master and
rs communicating properly and cluster is fine.
{code}
Test 1:
=======
Master bind address: 10.18.40.29
RS bind address : 127.0.0.1
RS RPC client bind address : Taken same ip of RS bind address (127.0.0.1:49503)
tcp 0 0 10.18.40.29:60000 :::* LISTEN
19113/java
tcp 0 0 127.0.0.1:60020 :::* LISTEN
19558/java
tcp 0 0 10.18.40.29:60000 127.0.0.1:49503 ESTABLISHED
19113/java
Test 2:
=======
Master bind address: 192.168.1.111
RS bind address : 10.18.40.29
RS RPC client bind address : Taken same ip of RS bind address
(10.18.40.29:61297)
tcp 0 0 10.18.40.29:60020 :::* LISTEN
22408/java
tcp 0 0 192.168.1.111:60000 :::* LISTEN
22277/java
tcp 0 0 192.168.1.111:60000 10.18.40.29:61297 ESTABLISHED
22277/java
Test 3:
=======
Master bind address: Dint specify in configuration (it will take primary
hostname - in my case ip of primary host name is 10.18.40.29)
RS bind address : Didnt specify (primary host name is 10.18.40.29)
RS RPC client bind address : Taken same ip of RS bind address(10.18.40.29:20302)
tcp 0 0 10.18.40.29:60020 :::* LISTEN
23952/java
tcp 0 0 10.18.40.29:60000 :::* LISTEN
23823/java
tcp 0 0 10.18.40.29:60000 10.18.40.29:20302 ESTABLISHED
23823/java
{code}
Thanks
> Master and Regionserver not able to communicate if both bound to different
> network interfaces on the same machine.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-8667
> URL: https://issues.apache.org/jira/browse/HBASE-8667
> Project: HBase
> Issue Type: Bug
> Components: IPC/RPC
> Reporter: rajeshbabu
> Fix For: 0.98.0, 0.95.2, 0.94.9
>
> Attachments: HBASE-8667_trunk.patch, HBASE-8667_Trunk.patch,
> HBASE-8667_Trunk-V2.patch
>
>
> While testing HBASE-8640 fix found that master and regionserver running on
> different interfaces are not communicating properly.
> I have two interfaces 1) lo 2) eth0 in my machine and default hostname
> interface is lo.
> I have configured master ipc address to ip of eth0 interface.
> Started master and regionserver on the same machine.
> 1) master rpc server bound to eth0 and RS rpc server bound to lo
> 2) Since rpc client is not binding to any ip address, when RS is reporting RS
> startup its getting registered with eth0 ip address(but actually it should
> register localhost)
> Here are RS logs:
> {code}
> 2013-05-31 06:05:28,608 WARN [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: reportForDuty failed;
> sleeping and then retrying.
> 2013-05-31 06:05:31,609 INFO [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
> Master server at 192.168.0.100,60000,1369960497008
> 2013-05-31 06:05:31,609 INFO [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
> 192.168.0.100,60000,1369960497008 that we are up with port=60020,
> startcode=1369960502544
> 2013-05-31 06:05:31,618 DEBUG [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
> hbase.rootdir=hdfs://localhost:2851/hbase
> 2013-05-31 06:05:31,618 DEBUG [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master:
> fs.default.name=hdfs://localhost:2851
> 2013-05-31 06:05:31,618 INFO [regionserver60020]
> org.apache.hadoop.hbase.regionserver.HRegionServer: Master passed us a
> different hostname to use; was=localhost, but now=192.168.0.100
> {code}
> Here are master logs:
> {code}
> 2013-05-31 06:05:31,615 INFO [IPC Server handler 9 on 60000]
> org.apache.hadoop.hbase.master.ServerManager: Registering
> server=192.168.0.100,60020,1369960502544
> {code}
> Since master has wrong rpc server address of RS, META is not getting assigned.
> {code}
> 2013-05-31 06:05:34,362 DEBUG [master-192.168.0.100,60000,1369960497008]
> org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan
> was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so
> generated a random one; hri=.META.,,1.1028785192, src=,
> dest=192.168.0.100,60020,1369960502544; 1 (online=1, available=1) available
> servers, forceNewPlan=false
> -----
> org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
> .META.,,1.1028785192 to 192.168.0.100,60020,1369960502544, trying to assign
> elsewhere instead; try=1 of 10
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:549)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:813)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1422)
> at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1315)
> at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1532)
> at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1587)
> at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:15039)
> at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:627)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1826)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1453)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1432)
> at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.addToRITandCallClose(AssignmentManager.java:699)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:584)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransition(AssignmentManager.java:517)
> at
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionInTransitionAndBlockUntilAssigned(AssignmentManager.java:473)
> at org.apache.hadoop.hbase.master.HMaster.assignMeta(HMaster.java:917)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:803)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:547)
> at java.lang.Thread.run(Thread.java:636)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
