Re: PleaseHoldException when Master is clearly running as JPS
Okay I finally fixed and the problem... The reason it took long is because it's tricky to know if hmaster is failing due to network errors vs file system or other lower level errors. In my case, it was failing because of both, and i was just restarting the master, which masked the error (second time I started master, I would get a PleaseWaitException, whereas first time, I would get a couldn't find master error). The trickier part is that there are alot of (okay) connect exceptions thrown in region servers which do not necessarily indicate failure (found this after some googleing) thus creating many red-herrings which made it tricky to tell wether the errors occuring in a run of hbase-start.sh are on the region end, or the master end, or simply network related. Now - i dont fully recall wether this was the exact case - but im willing to venture to say that Using telnet it became clear to me that in fact the master ports were accessible, but oddly, that the master wasn't working correctly . Maybe this could be changed? That is - maybe the master startup could be an all-or-nothing task which had a clear end result... Maybe a two part JIRA -- 1) if start-hbase.sh cannot start the master, start-hbase.sh should fail fast, or at least print a severe error message, before starting region servers. 2) the starting of master should clearly indicate as the last line in the logs wether or not it started successfully. On May 21, 2013, at 7:07 PM, Jay Vyas jayunit...@gmail.com wrote: Hi again ! Well.. to make this easier, Ive created a script which greps the hotspots in log files out and cleans them, restarting the entire cluster. Overall, it appears that my region servers cant see my hbase-master due to closed connections, or something of that sort. These errors are now reproducible so hopefully someone can shed some light. - EXCEPTIONS hbase-master - tcp0 0 :::3888 :::* LISTEN 0 172888 4674/java 2013-05-21 18:59:16,537 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 4729@hbase-master 2013-05-21 18:59:16,562 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hbase-master/192.168.122.200:. Will not attempt to authenticate using SASL (unknown error) 2013-05-21 18:59:16,628 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hbase-master/192.168.122.200:, initiating session 2013-05-21 18:59:16,669 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-05-21 18:59:16,842 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-master 2013-05-21 18:59:16,842 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1... Tue May 21 18:59:15 EDT 2013 Starting regionserver on hbase-master core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited -- 2013-05-21 18:59:14,579 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state) 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.203:33665 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state) 2013-05-21 18:59:14,582 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831) -- 2013-05-21 18:59:16,146 INFO org.apache.zookeeper.server.quorum.Learner: Getting a diff from the leader 0x0 2013-05-21 18:59:16,190 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0 2013-05-21 18:59:16,524 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0 2013-05-21 18:59:16,594
Re: PleaseHoldException when Master is clearly running as JPS
#This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Jay, I am not sure if this is network related, but the host file doesn't look correct. Here is what I would add there: #This is my /etc/hosts file --- 127.0.0.1 localhost.localdomain localhost4 localhost4.localdomain4 localhost #::1 localhost.localdomain localhost6 localhost6.localdomain6 localhost 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 3:42 PM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Hi Jay, I think Gmail is cutting that off for you as it doesn't show duplicate data. Here you go: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 It is what you should have for your host file. I removed the pointer to hbase-master from the 127 address, as that is not localhost. Here is mine: [root@cdh4-oozie-1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 10.20.217.122 cdh4-oozie-1.ent.cloudera.com cdh4-oozie-1 10.20.216.222 cdh4-oozie-2.ent.cloudera.com cdh4-oozie-2 10.20.216.237 cdh4-oozie-3.ent.cloudera.com cdh4-oozie-3 On Tue, May 21, 2013 at 3:51 PM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.com wrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
Re: PleaseHoldException when Master is clearly running as JPS
Here is my new /etc/hosts file 127.0.0.1 localhost.localdomain localhost4 localhost4.localdomain4 localhost #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 Im once again getting the PleaseHoldException hmmm . Sorry to conflate this again, but , does the above look correct? On Tue, May 21, 2013 at 5:06 PM, Jay Vyas jayunit...@gmail.com wrote: Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.comwrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO
Re: PleaseHoldException when Master is clearly running as JPS
No prob. I was referring to this : 127.0.0.1 hbase-master 192.168.122.200 hbase-master I was thinking that this is your HBase master. Correct me if i'm wrong. Could you please show me your logs? Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:36 AM, Jay Vyas jayunit...@gmail.com wrote: Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.com wrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment: host.name =localhost 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 13/05/21
Re: PleaseHoldException when Master is clearly running as JPS
https://gist.github.com/anonymous/5623327 -- all logs for starting up hbase and master On Tue, May 21, 2013 at 5:13 PM, Mohammad Tariq donta...@gmail.com wrote: No prob. I was referring to this : 127.0.0.1 hbase-master 192.168.122.200 hbase-master I was thinking that this is your HBase master. Correct me if i'm wrong. Could you please show me your logs? Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:36 AM, Jay Vyas jayunit...@gmail.com wrote: Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.com wrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF Setting DEBUG log level... HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013 create 't1','f1' 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
Re: PleaseHoldException when Master is clearly running as JPS
hmmm...all looks fine to me. we can do a few checks though to make sure everything is in place. start with making sure that all the RSs are properly up. Also make sure HDFS is out of safemode. What does 192.168.122.200:60010 show? Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:48 AM, Jay Vyas jayunit...@gmail.com wrote: https://gist.github.com/anonymous/5623327 -- all logs for starting up hbase and master On Tue, May 21, 2013 at 5:13 PM, Mohammad Tariq donta...@gmail.com wrote: No prob. I was referring to this : 127.0.0.1 hbase-master 192.168.122.200 hbase-master I was thinking that this is your HBase master. Correct me if i'm wrong. Could you please show me your logs? Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:36 AM, Jay Vyas jayunit...@gmail.com wrote: Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.com wrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]#
Re: PleaseHoldException when Master is clearly running as JPS
I see: 2013-05-21 17:15:07,914 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=hbase-regionserver1,60020,1369170595340, region=70236052/-ROOT- Over and over. Look in the region server logs, you should see fat stack traces on why it's failing to open the -ROOT- region. Maybe related to your gluster setup. J-D On Tue, May 21, 2013 at 2:18 PM, Jay Vyas jayunit...@gmail.com wrote: https://gist.github.com/anonymous/5623327 -- all logs for starting up hbase and master On Tue, May 21, 2013 at 5:13 PM, Mohammad Tariq donta...@gmail.com wrote: No prob. I was referring to this : 127.0.0.1 hbase-master 192.168.122.200 hbase-master I was thinking that this is your HBase master. Correct me if i'm wrong. Could you please show me your logs? Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:36 AM, Jay Vyas jayunit...@gmail.com wrote: Hmmm... what do you mean have your hostname in there? sorry -- just curious about which hostname you are referring to...? Im now getting a new exception: 13/05/21 17:02:44 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 2 of 7 failed; retrying after sleep of 1002 On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq donta...@gmail.com wrote: OK..You already have your hostname in there. But it is appearing twice. Comment out 127.0.0.1 hbase-master. This might be a reason. I did not notice that you are on a distributed setup. RS IPs and hostnames are fine. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:21 AM, Jay Vyas jayunit...@gmail.com wrote: Hi kevin : So you don't have any region servers defined in your /etc/hosts ? On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq donta...@gmail.com wrote: Sorry, my bad. By that I meant 127.0.0.1hostname..To me it seems like HBase is not able to connect to localhost using 127.0.0.1 Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 2:12 AM, Jay Vyas jayunit...@gmail.com wrote: Thanks, but adding 127.0.0.1 localhost to the top seems redundant... right? I did so but still no luck :(. 1) OS? This is fedora 16. 2) any thoughts on why the PleaseHoldException is being triggered ? On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq donta...@gmail.com wrote: OS?Add 127.0.0.1 localhost and see if it makes any difference. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:57 AM, Jay Vyas jayunit...@gmail.com wrote: #This is my /etc/hosts file --- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 127.0.0.1 hbase-master 192.168.122.200 hbase-master 192.168.122.201 hbase-regionserver1 192.168.122.202 hbase-regionserver2 192.168.122.203 hbase-regionserver3 On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq donta...@gmail.com wrote: Hello Jay, Please change the line containing 127.0.1.1 in your /etc/hosts to 127.0.0.1 and see if it works. Warm Regards, Tariq cloudfront.blogspot.com On Wed, May 22, 2013 at 1:47 AM, Jay Vyas jayunit...@gmail.com wrote: Hi folks: Hope someone can shed some light on this - I cannot run hbase shell create commands because of the PleaseHoldException on a fresh install of hbase. Im not finding much in the error logs, and all nodes appear to be up and running, including hbase master. Version : hbase-0.94.7 Error: I am getting the Please hold exception... on my hbase shell. When running create table 't1','f1'... But... But wait :) theres more ! ... clearly, the hbase master is running : [root@hbase-master ~]# jps 11896 HQuorumPeer 12914 Jps 9894 Main 5879 Main ** 12279 HMaster ** 5779 Main 11714 ZKServerTool 12058 HRegionServer 12860 Main 8369 Main And finally - here is a dump of the output from the shell --- any thoughts? [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/ hbase shell -d EOF create 't1','f1' EOF
Re: PleaseHoldException when Master is clearly running as JPS
Hi again ! Well.. to make this easier, Ive created a script which greps the hotspots in log files out and cleans them, restarting the entire cluster. Overall, it appears that my region servers cant see my hbase-master due to closed connections, or something of that sort. These errors are now reproducible so hopefully someone can shed some light. - EXCEPTIONS hbase-master - tcp0 0 :::3888 :::* LISTEN 0 172888 4674/java 2013-05-21 18:59:16,537 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 4729@hbase-master 2013-05-21 18:59:16,562 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hbase-master/192.168.122.200:. Will not attempt to authenticate using SASL (unknown error) 2013-05-21 18:59:16,628 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hbase-master/192.168.122.200:, initiating session 2013-05-21 18:59:16,669 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect 2013-05-21 18:59:16,842 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-master 2013-05-21 18:59:16,842 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1... Tue May 21 18:59:15 EDT 2013 Starting regionserver on hbase-master core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited -- 2013-05-21 18:59:14,579 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state) 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.203:33665 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state) 2013-05-21 18:59:14,582 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831) -- 2013-05-21 18:59:16,146 INFO org.apache.zookeeper.server.quorum.Learner: Getting a diff from the leader 0x0 2013-05-21 18:59:16,190 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0 2013-05-21 18:59:16,524 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0 2013-05-21 18:59:16,594 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /192.168.122.200:40223 2013-05-21 18:59:16,661 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2013-05-21 18:59:16,661 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.122.200:40223 (no session established for client) --- hbase-regionserver1 - tcp0 0 :::3888 :::* LISTEN 0 94882 2306/java 2013-05-21 18:59:14,442 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888 2013-05-21 18:59:14,451 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING 2013-05-21 18:59:14,453 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id = 1, proposed zxid=0x0 2013-05-21 18:59:14,458 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 0 at election address hbase-master/192.168.122.200:3888 java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) -- at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365) at java.lang.Thread.run(Thread.java:679) 2013-05-21
