Hi Cheolsoo, Yes, you are right. I am running the ZK quorum on remote machines and you way works. -- But I have put hbase-site.xml(which include hbase.zookeeper.quorum) in pig classpath. But seems pig did not store these hbase configurations to job. Is this the design of pig-0.10.0? -- Seems pig just puts hadoop configurations(hadoop-site.xml, core-site.xml...) into job.xml during create job. Is this correct? -- I remembered that pig-0.9.1 store hbase configurations into job.xml. Did I miss anything?
Thanks. At 2012-09-13 02:24:18,"Cheolsoo Park" <[email protected]> wrote: >Hi, > >2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening >> socket connection to server /127.0.0.1:2181 > > >This message seems wrong. I assume that you're running the ZK quorum on >remote machines, but it is trying to connect to localhost. Can you try to >set "hbase.zookeeper.quorum" in "pig.properties" as follows: >"hbase.zookeeper.quorum=<your >ZK quorum host:port>" ? > >Thanks, >Cheolsoo > >On Wed, Sep 12, 2012 at 12:39 AM, lulynn_2008 <[email protected]> wrote: > >> Hi Cheolsoo, >> Current TestJobSubmission and TestHBaseStorage passed. But if I run >> following scripts with hbase-0.94/zookeeper-3.4.3 cluster: >> >> the same issue happened. >> >> 2012-09-12 00:30:07,198 INFO org.apache.zookeeper.ClientCnxn: Opening >> socket connection to server /127.0.0.1:2181 >> 2012-09-12 00:30:07,199 INFO >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of >> this process is [email protected] >> 2012-09-12 00:30:07,212 WARN >> org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: >> java.lang.SecurityException: Unable to locate a login configuration >> occurred when trying to find JAAS configuration. >> 2012-09-12 00:30:07,213 INFO >> org.apache.zookeeper.client.ZooKeeperSaslClient: Client will not >> SASL-authenticate because the default JAAS configuration section 'Client' >> could not be found. If you are not using SASL, you may ignore this. On the >> other hand, if you expected SASL to work, please fix your JAAS >> configuration. >> 2012-09-12 00:30:07,221 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 >> for server null, unexpected error, closing socket connection and attempting >> reconnect >> java.net.ConnectException: Connection refused >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610) >> at >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> at >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> >> Reproduce steps: >> >> Create hbase table: >> ./hbase shell: >> create 'employees', 'SN', 'department', 'address' >> put 'employees', 'Hong', 'address:country', 'China' >> >> Run following pig commands: >> ./pig >> A = load 'hbase://employees' using >> org.apache.pig.backend.hadoop.hbase.HBaseStorage( 'address:country', >> '-loadKey true') as (SN:bytearray,country:bytearray); >> B = filter A by SN == 'Hong'; >> dump B; >> >> >> >> >> >> At 2012-08-22 05:51:38,"Cheolsoo Park" <[email protected]> wrote: >> >OK, I got TestJobSubmission passing. Please apply the following diff to >> >your Pig: >> > >> > >> >diff --git test/org/apache/pig/test/TestJobSubmission.java >> >test/org/apache/pig/test/TestJobSubmission.java >> >index 89ac132..211c4ae 100644 >> >--- test/org/apache/pig/test/TestJobSubmission.java >> >+++ test/org/apache/pig/test/TestJobSubmission.java >> >@@ -26,8 +26,10 @@ import java.util.Random; >> > >> > import org.apache.hadoop.conf.Configuration; >> > import org.apache.hadoop.hbase.HBaseTestingUtility; >> >*+import org.apache.hadoop.hbase.HConstants;* >> > import org.apache.hadoop.hbase.MiniHBaseCluster; >> > import org.apache.hadoop.hbase.util.Bytes; >> >*+import org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster;* >> > import org.apache.hadoop.mapred.jobcontrol.Job; >> > import org.apache.hadoop.mapred.jobcontrol.JobControl; >> > import org.apache.pig.ExecType; >> >@@ -541,7 +543,7 @@ public class TestJobSubmission { >> > // use the estimation >> > Configuration conf = cluster.getConfiguration(); >> > HBaseTestingUtility util = new HBaseTestingUtility(conf); >> >*- util.startMiniZKCluster();* >> >*+ int clientPort = util.startMiniZKCluster().getClientPort();* >> > util.startMiniHBaseCluster(1, 1); >> > >> > String query = "a = load '/passwd';" + >> >@@ -553,6 +555,7 @@ public class TestJobSubmission { >> > >> > pc.getConf().setProperty("pig.exec.reducers.bytes.per.reducer", >> >"100"); >> > pc.getConf().setProperty("pig.exec.reducers.max", "10"); >> >*+ pc.getConf().setProperty(HConstants.ZOOKEEPER_CLIENT_PORT, >> >Integer.toString(clientPort));* >> > ConfigurationValidator.validatePigProperties(pc.getProperties()); >> > conf = ConfigurationUtil.toConfiguration(pc.getProperties()); >> > JobControlCompiler jcc = new JobControlCompiler(pc, conf); >> > >> > >> >There are several changes to MiniZooKeeperCluster in HBase 0.94 compared >> to >> >0.90. In particular, the ZK client port is no longer 21818 by default. In >> >fact, it is randomized in 0.94. So the client port has to be explicitly >> set >> >in PigContext at runtime. >> > >> >Now I am able to run both TestJobSubmission and TestHBaseStorage. Please >> >let me know if this works for you. I am going to file a jira to capture >> >what I found, so we can make use of it when bumping the Hbase version in >> >the future. >> > >> >Thanks, >> >Cheolsoo >> > >> > >> >On Tue, Aug 21, 2012 at 1:25 AM, Cheolsoo Park <[email protected] >> >wrote: >> > >> >> Hi, >> >> >> >> TestJobSubmission doesn't pass for me either. But it doesn't throw the >> >> error that you're seeing. Instead, it hangs at line 583 >> >> in TestJobSubmission.java. I am not sure yet what's happening though. >> >> >> >> Thanks, >> >> Cheolsoo >> >> >> >> On Tue, Aug 21, 2012 at 12:24 AM, lulynn_2008 <[email protected]> >> wrote: >> >> >> >>> Hi Cheolsoo, >> >>> Please help to check whether TestJobSunbmission passed in your >> >>> environment. Thanks >> >>> >> >>> >> >>> Hi Cheolsoo, >> >>> TestHBaseStorage passed now with your fix. Thanks. >> >>> But TestJobSubmission still failed with the same previous error with >> >>> TestHBaseStorage. Please help to check. Thanks >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> At 2012-08-18 14:26:32,"Cheolsoo Park" <[email protected]> wrote: >> >>> >Hi, >> >>> > >> >>> >I tried the same version of libraries as you're using and was able to >> get >> >>> >TestHBaseStorage passing in trunk. Here is what I did: >> >>> > >> >>> >1) Updated the version of hadoop, hbase, and zookeeper in >> >>> >ivy/libraries.properties: >> >>> > >> >>> >hadoop-core.version=1.0.3 >> >>> >> hadoop-test.version=1.0.3 >> >>> >> hbase.version=0.94.0 >> >>> >> zookeeper.version=3.4.3 >> >>> > >> >>> > >> >>> >2) Updated TestHBaseStorage.java since it doesn't compile with >> >>> hbase-0.94.0: >> >>> > >> >>> >HTable table = new HTable(tableName); => HTable table = new >> HTable(conf, >> >>> >> tableName); >> >>> > >> >>> > >> >>> >There are 6 places to fix. >> >>> > >> >>> >3) Added the following lines to ivy.xml: >> >>> > >> >>> ><dependency org="com.github.stephenc.high-scale-lib" >> >>> name="high-scale-lib" >> >>> >> rev="${high-scale-lib.version}" conf="test->default"/> >> >>> >> <dependency org="com.google.protobuf" name="protobuf-java" >> >>> >> rev="${protobuf-java.version}" conf="test->default"/> >> >>> > >> >>> > >> >>> >I also added the following line to ivy/libraries.properties: >> >>> > >> >>> >high-scale-lib.version=1.1.1 >> >>> > >> >>> > >> >>> >In particular, I was able to reproduce your error without this line in >> >>> >ivy.xml: >> >>> > >> >>> ><dependency org="com.google.protobuf" name="protobuf-java" >> >>> >> rev="${protobuf-java.version}" conf="test->default"/> >> >>> > >> >>> > >> >>> >With all these changes, TestHBaseStorage passes for me. Please let me >> >>> know >> >>> >if this works for you. >> >>> > >> >>> >Thanks, >> >>> >Cheolsoo >> >>> > >> >>> >On Thu, Aug 16, 2012 at 11:02 PM, lulynn_2008 <[email protected]> >> >>> wrote: >> >>> > >> >>> >> BTW, I am using hadoop-1.0.3, hbase-0.94.0 and zookeeper-3.4.3. >> >>> >> >> >>> >> >> >>> >> Hi Cheolsoo, >> >>> >> You are right. TestHBaseStorage also failed with the same error. >> >>> >> >> >>> >> I got following region server information, and seems hbase region >> has >> >>> been >> >>> >> started successfully. But hbase can not connect to zookeeper >> >>> >> >> >>> >> 12/08/13 06:34:12 INFO server.PrepRequestProcessor: Got user-level >> >>> >> KeeperException when processing sessionid:0x1391cf90aed0000 >> type:delete >> >>> >> cxid:0x24 zxid:0x10 txntype:-1 reqpath:n/a Error >> >>> >> Path:/hbase/root-region-server Error:KeeperErrorCode = NoNode for >> >>> >> /hbase/root-region-server >> >>> >> >> >>> >> 12/08/13 06:34:16 INFO regionserver.HRegion: creating HRegion >> >>> pigtable_1 >> >>> >> HTD == {NAME => 'pigtable_1', FAMILIES => [{NAME => 'pig', >> >>> >> REPLICATION_SCOPE => '0', KEEP_DELETED_CELLS => 'false', >> COMPRESSION => >> >>> >> 'NONE', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true', >> MIN_VERSIONS => >> >>> >> '0', DATA_BLOCK_ENCODING => 'NONE', IN_MEMORY => 'false', >> BLOOMFILTER >> >>> => >> >>> >> 'NONE', TTL => '2147483647', VERSIONS => '3', BLOCKSIZE => >> '65536'}]} >> >>> >> RootDir = hdfs://shihc015:54670/user/root/hbase Table name == >> >>> pigtable_1 >> >>> >> >> >>> >> 12/08/13 06:34:43 INFO zookeeper.ClientCnxn: Opening socket >> connection >> >>> to >> >>> >> server shihc015/127.0.0.1:21818 >> >>> >> 12/08/13 06:34:43 WARN client.ZooKeeperSaslClient: >> SecurityException: >> >>> >> java.lang.SecurityException: Unable to locate a login configuration >> >>> >> occurred when trying to find JAAS configuration. >> >>> >> 12/08/13 06:34:43 INFO client.ZooKeeperSaslClient: Client will not >> >>> >> SASL-authenticate because the default JAAS configuration section >> >>> 'Client' >> >>> >> could not be found. If you are not using SASL, you may ignore this. >> On >> >>> the >> >>> >> other hand, if you expected SASL to work, please fix your JAAS >> >>> >> configuration. >> >>> >> 12/08/13 06:34:43 WARN zookeeper.ClientCnxn: Session 0x0 for server >> >>> null, >> >>> >> unexpected error, closing socket connection and attempting reconnect >> >>> >> java.net.ConnectException: Connection refused >> >>> >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> >>> >> at >> >>> >> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:610) >> >>> >> at >> >>> >> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286) >> >>> >> at >> >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035) >> >>> >> 12/08/13 06:34:43 WARN zookeeper.RecoverableZooKeeper: Possibly >> >>> transient >> >>> >> ZooKeeper exception: >> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> >>> >> >> >>> >> >> >>> >> 12/08/16 22:33:05 ERROR zookeeper.ZooKeeperWatcher: hconnection >> >>> Received >> >>> >> unexpected KeeperException, re-throwing exception >> >>> >> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >>> >> KeeperErrorCode = ConnectionLoss for /hbase/root-region-server >> >>> >> at >> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> >>> >> at >> >>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> >>> >> at >> org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1131) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:264) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataInternal(ZKUtil.java:522) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.zookeeper.ZKUtil.getDataAndWatch(ZKUtil.java:498) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.getData(ZooKeeperNodeTracker.java:156) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.zookeeper.RootRegionTracker.getRootRegionLocation(RootRegionTracker.java:62) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:821) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) >> >>> >> at >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >> >>> >> at >> >>> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) >> >>> >> at >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) >> >>> >> at >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) >> >>> >> at >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:551) >> >>> >> at >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:385) >> >>> >> at >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) >> >>> >> at >> >>> >> >> >>> >> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:492) >> >>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >>> >> at >> >>> >> >> >>> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) >> >>> >> at >> >>> >> >> >>> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) >> >>> >> at java.lang.reflect.Method.invoke(Method.java:611) >> >>> >> at >> >>> >> >> >>> >> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) >> >>> >> at >> >>> >> >> >>> >> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) >> >>> >> at >> >>> >> >> >>> >> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) >> >>> >> at >> >>> >> >> >>> >> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) >> >>> >> at >> >>> >> >> >>> >> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) >> >>> >> at >> >>> >> >> >>> >> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31) >> >>> >> at >> >>> >> >> >>> >> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) >> >>> >> at >> >>> >> >> >>> >> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> At 2012-08-16 01:30:33,"Cheolsoo Park" <[email protected]> >> wrote: >> >>> >> >Hi, >> >>> >> > >> >>> >> >Caused by: >> org.apache.hadoop.hbase.client.NoServerForRegionException: >> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. >> >>> >> > >> >>> >> > >> >>> >> >It looks like the region server is not available. >> >>> >> >1) Does TestHBaseStorage pass? I guess that it fails with the same >> >>> error. >> >>> >> >2) In your log, do you see any errors related to the HBase region >> >>> server? >> >>> >> >For example, java.lang.IllegalStateException: Could not >> instantiate a >> >>> >> >region instance. >> >>> >> > >> >>> >> >If the region server failed to start, you will see this failure. >> >>> >> > >> >>> >> >Thanks, >> >>> >> >Cheolsoo >> >>> >> > >> >>> >> > >> >>> >> >On Wed, Aug 15, 2012 at 12:56 AM, lulynn_2008 <[email protected] >> > >> >>> >> wrote: >> >>> >> > >> >>> >> >> Hi All, >> >>> >> >> >> >>> >> >> In TestJobSubmission, testReducerNumEstimation failed with >> following >> >>> >> error >> >>> >> >> information. Please give a glance to check what the problem is. >> >>> Thanks. >> >>> >> >> >> >>> >> >> Error information: >> >>> >> >> Testcase: testReducerNumEstimation took 1,809.419 sec >> >>> >> >> Caused an ERROR >> >>> >> >> Internal error creating job configuration. >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: >> >>> >> >> ERROR 2017: Internal error creating job configuration. >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:731) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:259) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.test.TestJobSubmission.testReducerNumEstimation(TestJobSubmission.java:491) >> >>> >> >> Caused by: >> >>> org.apache.hadoop.hbase.client.NoServerForRegionException: >> >>> >> >> Unable to find region for passwd,,99999999999999 after 10 tries. >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:926) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) >> >>> >> >> at >> >>> >> >> >> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) >> >>> >> >> at >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) >> >>> >> >> at >> >>> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.hbase.HBaseStorage.setLocation(HBaseStorage.java:545) >> >>> >> >> at >> >>> >> >> >> >>> >> >> >>> >> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:384) >> >>> >> >> >> >>> >> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >>> >> >>> >> >>> >> >> >>
