You were correct. The HBase jobs were hanging in this zombie state because the HBASE_CONF_DIR environment variable had not been configured for this cluster. It had worked coincidentally in the past because zookeeper had been running locally on all the nodes of this tiny test cluster.
"[email protected]" <[email protected]> wrote: >I'm using Pig 0.11.1. I don't recall having to bundle specific >hbase-site.xml in my pig jar in the past, but it's certainly worth >checking. > >Ted Yu <[email protected]> wrote: > >>From the stack trace, looks like TableOutputFormat had trouble reading >>from >>zookeeper. >> >>What pig version were you using ? >> >>Cheers >> >> >>On Tue, Jan 28, 2014 at 7:57 PM, <[email protected]> wrote: >> >>> >>> Has anyone encountered this problem? Maybe a known issue or common >>mistake? >>> >>> I'm executing a Pig script that uses hbase storage. It's a 3-node >>cluster >>> (0.94.7) and I'm executing the script from the same machine running >>the >>> HBase master. The M/R job hangs in preparation state (4). A dump >of >>the >>> solitary map process reveals that it is hanging indefinitely here: >>> >>> java.lang.Thread.State: TIMED_WAITING (sleeping) >>> .... >>> at >>> >>org.apache.hadoop.hbase.util.RetryCounter.sleepUntilNextRetry(RetryCounter.java:54) >>> at >>> >>org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:304) >>> .... >>> at >>> >>org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:124) >>> - locked <0x00000007b237bd78> (a >>> org.apache.hadoop.hbase.zookeeper.RootRegionTracker) >>> at >>> >>org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:83) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:989) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1102) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1000) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1102) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1004) >>> at >>> >>org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961) >>> at >org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:227) >>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) >>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:129) >>> at >>> >>org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:201) >>> .... >>> >>> There are no compactions running. There are no other M/R jobs >>running. >>> HBase shell is functioning from that node. The destination table >is >>new, >>> empty, and plain-jane. >>> >>> The worst part is that the task tracker isn't smart enough to kill >>the >>> process after I kill the M/R job and I have to chase the task around >>the >>> nodes: kill, kill, kill, ... >>> >>> >>> regards, >>> >>> -Jess >>> >>> > >-- >Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.
