Both UT passed now:-) Thanks Cheolsoo.
At 2012-08-22 05:51:38,"Cheolsoo Park" <[email protected]> wrote: >OK, I got TestJobSubmission passing. Please apply the following diff to >your Pig: > > >diff --git test/org/apache/pig/test/TestJobSubmission.java >test/org/apache/pig/test/TestJobSubmission.java >index 89ac132..211c4ae 100644 >--- test/org/apache/pig/test/TestJobSubmission.java >+++ test/org/apache/pig/test/TestJobSubmission.java >@@ -26,8 +26,10 @@ import java.util.Random; > > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.hbase.HBaseTestingUtility; >*+import org.apache.hadoop.hbase.HConstants;* > import org.apache.hadoop.hbase.MiniHBaseCluster; > import org.apache.hadoop.hbase.util.Bytes; >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;* > import org.apache.hadoop.mapred.jobcontrol.Job; > import org.apache.hadoop.mapred.jobcontrol.JobControl; > import org.apache.pig.ExecType; >@@ -541,7 +543,7 @@ public class TestJobSubmission { > // use the estimation > Configuration conf = cluster.getConfiguration(); > HBaseTestingUtility util = new HBaseTestingUtility(conf); >*- util.startMiniZKCluster();* >*+ int clientPort = util.startMiniZKCluster().getClientPort();* > util.startMiniHBaseCluster(1, 1); > > String query = "a = load '/passwd';" + >@@ -553,6 +555,7 @@ public class TestJobSubmission { > > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", >"100"); > pc.getConf().setProperty("pig.exec.reducers.max", "10"); >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, >Integer.toString(clientPort));* > ConfigurationValidator.validatePigProperties(pc.getProperties()); > conf = ConfigurationUtil.toConfiguration(pc.getProperties()); > JobControlCompiler jcc = new JobControlCompiler(pc, conf); > > >There are several changes to MiniZooKeeperCluster in HBase 0.94 compared to >0.90. In particular, the ZK client port is no longer 21818 by default. In >fact, it is randomized in 0.94. So the client port has to be explicitly set >in PigContext at runtime. > >Now I am able to run both TestJobSubmission and TestHBaseStorage. Please >let me know if this works for you. I am going to file a jira to capture >what I found, so we can make use of it when bumping the Hbase version in >the future. > >Thanks, >Cheolsoo > > >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[email protected]>wrote: > >> Hi, >> >> TestJobSubmission doesn't pass for me either. But it doesn't throw the >> error that you're seeing. Instead, it hangs at line 583 >> in TestJobSubmission.java. I am not sure yet what's happening though. >> >> Thanks, >> Cheolsoo >> >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[email protected]> wrote: >> >>> Hi Cheolsoo, >>> Please help to check whether TestJobSunbmission passed in your >>> environment. Thanks >>> >>> >>> Hi Cheolsoo, >>> TestHBaseStorage passed now with your fix. Thanks. >>> But TestJobSubmission still failed with the same previous error with >>> TestHBaseStorage. Please help to check. Thanks >>> >>> >>> >>> >>> >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[email protected]> wrote: >>> >Hi, >>> > >>> >I tried the same version of libraries as you're using and was able to get >>> >TestHBaseStorage passing in trunk. Here is what I did: >>> > >>> >1) Updated the version of hadoop, hbase, and zookeeper in >>> >ivy/libraries.properties: >>> > >>> >hadoop-core.version=1.0.3 >>> >> hadoop-test.version=1.0.3 >>> >> hbase.version=0.94.0 >>> >> zookeeper.version=3.4.3 >>> > >>> > >>> >2) Updated TestHBaseStorage.java since it doesn't compile with >>> hbase-0.94.0: >>> > >>> >HTable table = new HTable(tableName); => HTable table = new HTable(conf, >>> >> tableName); >>> > >>> > >>> >There are 6 places to fix. >>> > >>> >3) Added the following lines to ivy.xml: >>> > >>> ><dependency org="com.github.stephenc.high-scale-lib" >>> name="high-scale-lib" >>> >> rev="${high-scale-lib.version}" conf="test->default"/> >>> >> <dependency org="com.google.protobuf" name="protobuf-java" >>> >> rev="${protobuf-java.version}" conf="test->default"/> >>> > >>> > >>> >I also added the following line to ivy/libraries.properties: >>> > >>> >high-scale-lib.version=1.1.1 >>> > >>> > >>> >In particular, I was able to reproduce your error without this line in >>> >ivy.xml: >>> > >>> ><dependency org="com.google.protobuf" name="protobuf-java" >>> >> rev="${protobuf-java.version}" conf="test->default"/> >>> > >>> > >>> >With all these changes, TestHBaseStorage passes for me. Please let me >>> know >>> >if this works for you. >>> > >>> >Thanks, >>> >Cheolsoo >>> > >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <[email protected]> >>> wrote: >>> > >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3. >>> >> >>> >> >>> >> Hi Cheolsoo, >>> >> You are right. TestHBaseStorage also failed with the same error. >>> >> >>> >> I got following region server information, and seems hbase region has >>> been >>> >> started successfully. But hbase can not connect to zookeeper >>> >> >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level >>> >> KeeperException when processing sessionid:0x1391cf90aed0000 type:delete >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for >>> >> /hbase/root-region-server >>> >> >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion >>> pigtable_1 >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig', >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false', COMPRESSION => >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true', MIN_VERSIONS => >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false', BLOOMFILTER >>> => >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE => '65536'}]} >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name == >>> pigtable_1 >>> >> >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket connection >>> to >>> >> server shihc015/127.0.0.1:21818 >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient: SecurityException: >>> >> java.lang.SecurityException: Unable to locate a login configuration >>> >> occurred when trying to find JAAS configuration. >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not >>> >> SASL-authenticate because the default JAAS configuration section >>> 'Client' >>> >> could not be found. If you are not using SASL, you may ignore this. On >>> the >>> >> other hand, if you expected SASL to work, please fix your JAAS >>> >> configuration. >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server >>> null, >>> >> unexpected error, closing socket connection and attempting reconnect >>> >> java.net.ConnectException: Connection refused >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> >> at >>> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610) >>> >> at >>> >> >>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >>> >> at >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly >>> transient >>> >> ZooKeeper exception: >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >>> >> >>> >> >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection >>> Received >>> >> unexpected KeeperException, re-throwing exception >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >>> >> at >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >>> >> at >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >>> >> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131) >>> >> at >>> >> >>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) >>> >> at >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) >>> >> at >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) >>> >> at >>> >> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) >>> >> at >>> >> >>> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >>> >> at >>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) >>> >> at >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) >>> >> at >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) >>> >> at >>> >> >>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551) >>> >> at >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385) >>> >> at >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) >>> >> at >>> >> >>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492) >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >> at >>> >> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) >>> >> at >>> >> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) >>> >> at java.lang.reflect.Method.invoke(Method.java:611) >>> >> at >>> >> >>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) >>> >> at >>> >> >>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) >>> >> at >>> >> >>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) >>> >> at >>> >> >>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) >>> >> at >>> >> >>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) >>> >> at >>> >> >>> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) >>> >> at >>> >> >>> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) >>> >> at >>> >> >>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <[email protected]> wrote: >>> >> >Hi, >>> >> > >>> >> >Caused by: org.apache.hadoop.hbase.client.NoServerForRegionException: >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. >>> >> > >>> >> > >>> >> >It looks like the region server is not available. >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same >>> error. >>> >> >2) In your log, do you see any errors related to the HBase region >>> server? >>> >> >For example, java.lang.IllegalStateException: Could not instantiate a >>> >> >region instance. >>> >> > >>> >> >If the region server failed to start, you will see this failure. >>> >> > >>> >> >Thanks, >>> >> >Cheolsoo >>> >> > >>> >> > >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <[email protected]> >>> >> wrote: >>> >> > >>> >> >> Hi All, >>> >> >> >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with following >>> >> error >>> >> >> information. Please give a glance to check what the problem is. >>> Thanks. >>> >> >> >>> >> >> Error information: >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec >>> >> >> Caused an ERROR >>> >> >> Internal error creating job configuration. >>> >> >> >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: >>> >> >> ERROR 2017: Internal error creating job configuration. >>> >> >> at >>> >> >> >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731) >>> >> >> at >>> >> >> >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) >>> >> >> at >>> >> >> >>> >> >>> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491) >>> >> >> Caused by: >>> org.apache.hadoop.hbase.client.NoServerForRegionException: >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926) >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) >>> >> >> at >>> >> >> >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >>> >> >> at >>> >> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) >>> >> >> at >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) >>> >> >> at >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) >>> >> >> at >>> >> >> >>> >> >>> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545) >>> >> >> at >>> >> >> >>> >> >>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384) >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >>> >>> >>> >>> >>
