Hi, 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server /127.0.0.1:2181
This message seems wrong. I assume that you're running the ZK quorum on remote machines, but it is trying to connect to localhost. Can you try to set "hbase.zookeeper.quorum" in "pig.properties" as follows: "hbase.zookeeper.quorum=<your ZK quorum host:port>" ? Thanks, Cheolsoo On Wed, Sep 12, 2012 at 12:39 AM, lulynn_2008 <[email protected]> wrote: > Hi Cheolsoo, > Current TestJobSubmission and TestHBaseStorage passed. But if I run > following scripts with hbase-0.94/zookeeper-3.4.3 cluster: > > the same issue happened. > > 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server /127.0.0.1:2181 > 2012-09-12 00:30:07,199 INFO > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of > this process is [email protected] > 2012-09-12 00:30:07,212 WARN > org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: > java.lang.SecurityException: Unable to locate a login configuration > occurred when trying to find JAAS configuration. > 2012-09-12 00:30:07,213 INFO > org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not > SASL-authenticate because the default JAAS configuration section 'Client' > could not be found. If you are not using SASL, you may ignore this. On the > other hand, if you expected SASL to work, please fix your JAAS > configuration. > 2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 > for server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) > > Reproduce steps: > > Create hbase table: > ./hbase shell: > create 'employees', 'SN', 'department', 'address' > put 'employees', 'Hong', 'address:country', 'China' > > Run following pig commands: > ./pig > A = load 'hbase://employees' using > org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country', > '-loadKey true') as (SN:bytearray,country:bytearray); > B = filter A by SN == 'Hong'; > dump B; > > > > > > At 2012-08-22 05:51:38,"Cheolsoo Park" <[email protected]> wrote: > >OK, I got TestJobSubmission passing. Please apply the following diff to > >your Pig: > > > > > >diff --git test/org/apache/pig/test/TestJobSubmission.java > >test/org/apache/pig/test/TestJobSubmission.java > >index 89ac132..211c4ae 100644 > >--- test/org/apache/pig/test/TestJobSubmission.java > >+++ test/org/apache/pig/test/TestJobSubmission.java > >@@ -26,8 +26,10 @@ import java.util.Random; > > > > import org.apache.hadoop.conf.Configuration; > > import org.apache.hadoop.hbase.HBaseTestingUtility; > >*+import org.apache.hadoop.hbase.HConstants;* > > import org.apache.hadoop.hbase.MiniHBaseCluster; > > import org.apache.hadoop.hbase.util.Bytes; > >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;* > > import org.apache.hadoop.mapred.jobcontrol.Job; > > import org.apache.hadoop.mapred.jobcontrol.JobControl; > > import org.apache.pig.ExecType; > >@@ -541,7 +543,7 @@ public class TestJobSubmission { > > // use the estimation > > Configuration conf = cluster.getConfiguration(); > > HBaseTestingUtility util = new HBaseTestingUtility(conf); > >*- util.startMiniZKCluster();* > >*+ int clientPort = util.startMiniZKCluster().getClientPort();* > > util.startMiniHBaseCluster(1, 1); > > > > String query = "a = load '/passwd';" + > >@@ -553,6 +555,7 @@ public class TestJobSubmission { > > > > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", > >"100"); > > pc.getConf().setProperty("pig.exec.reducers.max", "10"); > >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, > >Integer.toString(clientPort));* > > ConfigurationValidator.validatePigProperties(pc.getProperties()); > > conf = ConfigurationUtil.toConfiguration(pc.getProperties()); > > JobControlCompiler jcc = new JobControlCompiler(pc, conf); > > > > > >There are several changes to MiniZooKeeperCluster in HBase 0.94 compared > to > >0.90. In particular, the ZK client port is no longer 21818 by default. In > >fact, it is randomized in 0.94. So the client port has to be explicitly > set > >in PigContext at runtime. > > > >Now I am able to run both TestJobSubmission and TestHBaseStorage. Please > >let me know if this works for you. I am going to file a jira to capture > >what I found, so we can make use of it when bumping the Hbase version in > >the future. > > > >Thanks, > >Cheolsoo > > > > > >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[email protected] > >wrote: > > > >> Hi, > >> > >> TestJobSubmission doesn't pass for me either. But it doesn't throw the > >> error that you're seeing. Instead, it hangs at line 583 > >> in TestJobSubmission.java. I am not sure yet what's happening though. > >> > >> Thanks, > >> Cheolsoo > >> > >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[email protected]> > wrote: > >> > >>> Hi Cheolsoo, > >>> Please help to check whether TestJobSunbmission passed in your > >>> environment. Thanks > >>> > >>> > >>> Hi Cheolsoo, > >>> TestHBaseStorage passed now with your fix. Thanks. > >>> But TestJobSubmission still failed with the same previous error with > >>> TestHBaseStorage. Please help to check. Thanks > >>> > >>> > >>> > >>> > >>> > >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[email protected]> wrote: > >>> >Hi, > >>> > > >>> >I tried the same version of libraries as you're using and was able to > get > >>> >TestHBaseStorage passing in trunk. Here is what I did: > >>> > > >>> >1) Updated the version of hadoop, hbase, and zookeeper in > >>> >ivy/libraries.properties: > >>> > > >>> >hadoop-core.version=1.0.3 > >>> >> hadoop-test.version=1.0.3 > >>> >> hbase.version=0.94.0 > >>> >> zookeeper.version=3.4.3 > >>> > > >>> > > >>> >2) Updated TestHBaseStorage.java since it doesn't compile with > >>> hbase-0.94.0: > >>> > > >>> >HTable table = new HTable(tableName); => HTable table = new > HTable(conf, > >>> >> tableName); > >>> > > >>> > > >>> >There are 6 places to fix. > >>> > > >>> >3) Added the following lines to ivy.xml: > >>> > > >>> ><dependency org="com.github.stephenc.high-scale-lib" > >>> name="high-scale-lib" > >>> >> rev="${high-scale-lib.version}" conf="test->default"/> > >>> >> <dependency org="com.google.protobuf" name="protobuf-java" > >>> >> rev="${protobuf-java.version}" conf="test->default"/> > >>> > > >>> > > >>> >I also added the following line to ivy/libraries.properties: > >>> > > >>> >high-scale-lib.version=1.1.1 > >>> > > >>> > > >>> >In particular, I was able to reproduce your error without this line in > >>> >ivy.xml: > >>> > > >>> ><dependency org="com.google.protobuf" name="protobuf-java" > >>> >> rev="${protobuf-java.version}" conf="test->default"/> > >>> > > >>> > > >>> >With all these changes, TestHBaseStorage passes for me. Please let me > >>> know > >>> >if this works for you. > >>> > > >>> >Thanks, > >>> >Cheolsoo > >>> > > >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <[email protected]> > >>> wrote: > >>> > > >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3. > >>> >> > >>> >> > >>> >> Hi Cheolsoo, > >>> >> You are right. TestHBaseStorage also failed with the same error. > >>> >> > >>> >> I got following region server information, and seems hbase region > has > >>> been > >>> >> started successfully. But hbase can not connect to zookeeper > >>> >> > >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level > >>> >> KeeperException when processing sessionid:0x1391cf90aed0000 > type:delete > >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error > >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for > >>> >> /hbase/root-region-server > >>> >> > >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion > >>> pigtable_1 > >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig', > >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false', > COMPRESSION => > >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true', > MIN_VERSIONS => > >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false', > BLOOMFILTER > >>> => > >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE => > '65536'}]} > >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name == > >>> pigtable_1 > >>> >> > >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket > connection > >>> to > >>> >> server shihc015/127.0.0.1:21818 > >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient: > SecurityException: > >>> >> java.lang.SecurityException: Unable to locate a login configuration > >>> >> occurred when trying to find JAAS configuration. > >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not > >>> >> SASL-authenticate because the default JAAS configuration section > >>> 'Client' > >>> >> could not be found. If you are not using SASL, you may ignore this. > On > >>> the > >>> >> other hand, if you expected SASL to work, please fix your JAAS > >>> >> configuration. > >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server > >>> null, > >>> >> unexpected error, closing socket connection and attempting reconnect > >>> >> java.net.ConnectException: Connection refused > >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>> >> at > >>> >> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610) > >>> >> at > >>> >> > >>> > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) > >>> >> at > >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) > >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly > >>> transient > >>> >> ZooKeeper exception: > >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server > >>> >> > >>> >> > >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection > >>> Received > >>> >> unexpected KeeperException, re-throwing exception > >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server > >>> >> at > >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > >>> >> at > >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > >>> >> at > org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) > >>> >> at > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > >>> >> at > >>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) > >>> >> at > >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) > >>> >> at > >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) > >>> >> at > >>> >> > >>> > org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551) > >>> >> at > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385) > >>> >> at > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) > >>> >> at > >>> >> > >>> > org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492) > >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >>> >> at > >>> >> > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) > >>> >> at > >>> >> > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) > >>> >> at java.lang.reflect.Method.invoke(Method.java:611) > >>> >> at > >>> >> > >>> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) > >>> >> at > >>> >> > >>> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > >>> >> at > >>> >> > >>> > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) > >>> >> at > >>> >> > >>> > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > >>> >> at > >>> >> > >>> > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > >>> >> at > >>> >> > >>> > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) > >>> >> at > >>> >> > >>> > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > >>> >> at > >>> >> > >>> > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <[email protected]> > wrote: > >>> >> >Hi, > >>> >> > > >>> >> >Caused by: > org.apache.hadoop.hbase.client.NoServerForRegionException: > >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. > >>> >> > > >>> >> > > >>> >> >It looks like the region server is not available. > >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same > >>> error. > >>> >> >2) In your log, do you see any errors related to the HBase region > >>> server? > >>> >> >For example, java.lang.IllegalStateException: Could not > instantiate a > >>> >> >region instance. > >>> >> > > >>> >> >If the region server failed to start, you will see this failure. > >>> >> > > >>> >> >Thanks, > >>> >> >Cheolsoo > >>> >> > > >>> >> > > >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <[email protected] > > > >>> >> wrote: > >>> >> > > >>> >> >> Hi All, > >>> >> >> > >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with > following > >>> >> error > >>> >> >> information. Please give a glance to check what the problem is. > >>> Thanks. > >>> >> >> > >>> >> >> Error information: > >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec > >>> >> >> Caused an ERROR > >>> >> >> Internal error creating job configuration. > >>> >> >> > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: > >>> >> >> ERROR 2017: Internal error creating job configuration. > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491) > >>> >> >> Caused by: > >>> org.apache.hadoop.hbase.client.NoServerForRegionException: > >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > >>> >> >> at > >>> >> >> > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) > >>> >> >> at > >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) > >>> >> >> at > >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545) > >>> >> >> at > >>> >> >> > >>> >> > >>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384) > >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> > >>> > >>> > >>> > >> >
