It sounds like the client is trying to connect through a public network, so (bare with me here as my understanding of zookeeper and how it interacts with hbase is very limited) when zookeeper, or the hmaster returns the ip that the currently requested region is on, it is returning the private ip. If it returned the hostname, and the clients /etc/hosts file contains the proper public ip mapping to each nodes hostname, it may just work.
Shuja, Try setting each nodes /etc/hosts to resolve private IPs, and then set your clients /etc/hosts to resolve to the public IPs. If the client is being returned an ip instead of a hostname, then I'm not sure that there is anything that can be done to connect that way. Maybe you could set up another node to host some kind of VPN solution, and VPN the client into the same network as the cluster, so that it appears on the same local network. Travis Hegner http://www.travishegner.com/ (P.S. If you tried viewing the DNS tutorial from a previous message in this thread, I apologize as my ISP went down right after sending that email. The site is back up now.) -----Original Message----- From: Michael Segel [mailto:[email protected]] Sent: Friday, September 03, 2010 11:04 AM To: [email protected] Subject: RE: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: /10.0.3.85:60020 Eeewww... Haven't tried running this through NAT, but from what I've seen, I wouldn't be surprised if that gets things mucked up. Shooting from the hip... If your entire cloud is all on virtual machines with the 10.x network... then set the name resolution to only use the local /etc/hosts you should be ok... But I don't have any way to test that theory. BTW this would also include your ZK nodes too. > Date: Fri, 3 Sep 2010 17:48:42 +0500 > Subject: Re: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem > connecting to server: /10.0.3.85:60020 > From: [email protected] > To: [email protected] > > Hello , > > my setup is running on VMs and these VMs have local address ( 10.0.3.85 .. > etc ) . When i try to access then i use global IPs like 203.14.166.12 and > they are then NATted do the local IPs . I have made required entries in the > /etc/hosts file . and i can confirm that forward and reverse DNS resolution > are both working . > > when my client tries to connect with it . it gets successful in the initial > stage . but then it returns this the local ip and my client starts to fail . > following is the log > > INFO zookeeper.ZooKeeper: Initiating client connection, connectString= > myserver.mycompany.com:2181 sessionTimeout=60000 > watcher=org.apache.hadoop. > hbase.client.hconnectionmanager$clientzkwatc...@fd54d6 > 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Opening socket connection to > server myserver.mycompany.com/203.14.166.12:2181< > http://myserver.mycompany.com/20314.166.12:2181> > 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Socket connection established > to myserver.mycompany.com/203.14.166.12:2181, initiating session > 10/08/31 20:27:34 INFO zookeeper.ClientCnxn: Session establishment > complete > on server myserver.mycompany.com/203.14.166.12:2181, sessionid = > 0x12acaa705e30004, negotiated timeout = 40000 > 10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: / > 10.0.3.85:60020 > 10/08/31 20:28:15 INFO ipc.HbaseRPC: Problem connecting to server: / > 10.0.3.85:60020 > 10/08/31 20:28:36 INFO ipc.HbaseRPC: Problem connecting to server: / > > regards > > On Thu, Sep 2, 2010 at 6:56 PM, Hegner, Travis <[email protected]>wrote: > > > I think we need a little more information about your setup. > > > > Are you running in EC2 or something similar? > > You are referring to "Global" and "Local" ip addresses, by this I assume > > you mean "Public" and "Private"? > > > > To understand how your cluster's nodes communicate with eachother, and how > > your clients communicate with the cluster, we need to understand your > > environment. > > > > Can each of your nodes communicate with the other nodes through a private > > IP address? Are they all on the same local network? > > > > How are you attempting to connect a client to the cluster? From a public > > network? > > > > I have a dns tutorial on my website ( > > http://www.travishegner.com/2009/06/authoritative-dns.html), that may help > > you get DNS configured properly. But that will only help you if your nodes > > must communicate with eachother over public addresses, and yet each node > > sees it's own address as a private one. If that is the case, then you'll > > have to change every /etc/hosts file of every node to contain only that > > nodes resolution to the private ip address, and then rely on DNS to resolve > > the public address of other nodes in the cluster. > > > > Travis Hegner > > http://www.travishegner.com/ > > > > -----Original Message----- > > From: Shuja Rehman [mailto:[email protected]] > > Sent: Thursday, September 02, 2010 5:17 AM > > To: [email protected] > > Subject: Re: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem > > connecting to server: /10.0.3.85:60020 > > > > Hegner, > > The DNS service is running on cluster. So what should I do next to resolve > > it. > > > > > > On Wed, Sep 1, 2010 at 5:34 PM, Hegner, Travis <[email protected] > > >wrote: > > > > > Shuja, > > > > > > If you are not running any type of DNS/rDNS service, then make sure the > > > /etc/hosts file on each of your nodes maps each node to the IP address > > you > > > want it to resolve to. > > > > > > Thanks, > > > Travis Hegner > > > http://www.travishegner.com/ > > > > > > -----Original Message----- > > > From: Shuja Rehman [mailto:[email protected]] > > > Sent: Wednesday, September 01, 2010 6:02 AM > > > To: [email protected] > > > Subject: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem > > > connecting to server: /10.0.3.85:60020 > > > > > > Hi All > > > > > > I have used these configuration settings to access hbase server from java > > > client > > > > > > HBaseConfiguration config = new HBaseConfiguration(); > > > config.clear(); > > > config.set("hbase.zookeeper.quorum", "myserver.mycompany.com:2181"); > > > config.set("hbase.zookeeper.property.clientPort","2181"); > > > > > > The problem is that its trying to connect to local ip address instead of > > > global IP. > > > > > > INFO zookeeper.ZooKeeper: Initiating client connection, connectString= > > > myserver.mycompany.com:2181 sessionTimeout=60000 > > > > > > > > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@fd54d6 > > > 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Opening socket connection to > > > server myserver.mycompany.com/203.14.166.12:2181< > > > http://myserver.mycompany.com/20314.166.12:2181> > > > 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Socket connection > > established > > > to myserver.mycompany.com/203.14.166.12:2181, initiating session > > > 10/08/31 20:27:34 INFO zookeeper.ClientCnxn: Session establishment > > complete > > > on server myserver.mycompany.com/203.14.166.12:2181, sessionid = > > > 0x12acaa705e30004, negotiated timeout = 40000 > > > > > > 10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: / > > > 10.0.3.85:60020 > > > 10/08/31 20:28:15 INFO ipc.HbaseRPC: Problem connecting to server: / > > > 10.0.3.85:60020 > > > 10/08/31 20:28:36 INFO ipc.HbaseRPC: Problem connecting to server: / > > > 10.0.3.85:60020 > > > > > > but it should try to connect to global ip not local ip. so need to > > resolve > > > this issue.I think we might need to put somewhere global Ip Addresss. I > > > have > > > checked the zk dump and it has the following information > > > > > > HBase tree in ZooKeeper is rooted at /hbase > > > Cluster up? true > > > Master address: 10.0.3.85:60000 > > > Region server holding ROOT: 10.0.3.85:60020 > > > Region servers: > > > - 10.0.3.100:60020 > > > - 10.0.3.85:60020 > > > Quorum Server Statistics: > > > - hadoop.zoniversal.com:2181 > > > Zookeeper version: 3.3.1-942149, built on 05/07/2010 17:14 GMT > > > Clients: > > > /10.0.3.85:57636[1](queued=0,recved=1994,sent=1994) > > > /10.0.3.85:57654[1](queued=0,recved=1991,sent=1991) > > > /10.0.3.85:49253[0](queued=0,recved=1,sent=0) > > > /10.0.3.85:46493[1](queued=0,recved=2041,sent=2042) > > > /10.0.3.100:50387[1](queued=0,recved=2016,sent=2017) > > > > > > Latency min/avg/max: 0/0/118 > > > Received: 8056 > > > Sent: 8057 > > > Outstanding: 0 > > > Zxid: 0xda > > > Mode: standalone > > > Node count: 11 > > > > > > > > > > > > Anybody there to help me out?? > > > > > > Thanks > > > > > > -- > > > Regards > > > Shuja-ur-Rehman Baig > > > http://pk.linkedin.com/in/shujamughal > > > Cell: +92 3214207445 > > > > > > The information contained in this communication is confidential and is > > > intended only for the use of the named recipient. Unauthorized use, > > > disclosure, or copying is strictly prohibited and may be unlawful. If > > you > > > have received this communication in error, you should know that you are > > > bound to confidentiality, and should please immediately notify the sender > > or > > > our IT Department at 866.459.4599. > > > > > > > > > > > -- > > Regards > > Shuja-ur-Rehman Baig > > http://pk.linkedin.com/in/shujamughal > > Cell: +92 3214207445 > > > > The information contained in this communication is confidential and is > > intended only for the use of the named recipient. Unauthorized use, > > disclosure, or copying is strictly prohibited and may be unlawful. If you > > have received this communication in error, you should know that you are > > bound to confidentiality, and should please immediately notify the sender or > > our IT Department at 866.459.4599. > > > > > > -- > Regards > Shuja-ur-Rehman Baig > http://pk.linkedin.com/in/shujamughal > Cell: +92 3214207445 The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
