Hi, I'll follow the problem I sent before.
I checked the followings in Apache trunk version. = how to reproduce the problem - Add HRegionPartitioner.class to the 4th argument of TableMapReduceUtil#initTableReducerJob() at line around 133 in src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java - Change or remove "hbase.zookeeper.property.clientPort" property in hbase-site.xml ( see attached file: changed to 12345 ). - run testMultiRegionTable() = error logs I got the error message as following: 2011-09-12 22:28:51,020 DEBUG [Thread-832] zookeeper.ZKUtil(93): hconnection opening connection to ZooKeeper with ensemble (localhost:12345) 2011-09-12 22:28:51,022 INFO [Thread-832] zookeeper.RecoverableZooKeeper(89): The identifier of this process is [email protected] 2011-09-12 22:28:51,123 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2011-09-12 22:28:51,123 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:28:52,367 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2011-09-12 22:28:52,367 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 2 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:28:53,783 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2011-09-12 22:28:53,783 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 3 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:28:55,260 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master 2011-09-12 22:28:55,260 ERROR [Thread-832] zookeeper.RecoverableZooKeeper(163): ZooKeeper exists failed after 3 retries 2011-09-12 22:28:55,261 WARN [Thread-832] zookeeper.ZKUtil(230): hconnection Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:531) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:28:55,262 ERROR [Thread-832] zookeeper.ZooKeeperWatcher(358): hconnection Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:531) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:28:55,263 FATAL [Thread-832] client.HConnectionManager$HConnectionImplementation(1661): Unexpected exception during initialization, aborting org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:531) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:28:57,204 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server 2011-09-12 22:28:57,204 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 1 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:28:58,498 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server 2011-09-12 22:28:58,498 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 2 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:29:00,211 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server 2011-09-12 22:29:00,211 INFO [Thread-832] zookeeper.RecoverableZooKeeper(173): The 3 times to retry ZooKeeper after sleeping 1000 ms 2011-09-12 22:29:02,417 WARN [Thread-832] zookeeper.RecoverableZooKeeper(161): Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server 2011-09-12 22:29:02,417 ERROR [Thread-832] zookeeper.RecoverableZooKeeper(163): ZooKeeper exists failed after 3 retries 2011-09-12 22:29:02,417 WARN [Thread-832] zookeeper.ZKUtil(230): hconnection Unable to set watcher on znode /hbase/root-region-server org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:534) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:29:02,417 ERROR [Thread-832] zookeeper.ZooKeeperWatcher(358): hconnection Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:534) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:29:02,418 FATAL [Thread-832] client.HConnectionManager$HConnectionImplementation(1661): Unexpected exception during initialization, aborting org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/root-region-server at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:156) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:221) at org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:534) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:520) at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:178) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:192) at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:170) at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:123) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 2011-09-12 22:29:02,418 ERROR [Thread-832] mapreduce.HRegionPartitioner(125): java.io.IOException: org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2e54e48d closed 2011-09-12 22:29:02,422 WARN [Thread-832] mapred.LocalJobRunner$Job(256): job_local_0001 java.lang.NullPointerException at org.apache.hadoop.hbase.mapreduce.HRegionPartitioner.setConf(HRegionPartitioner.java:128) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:527) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) = what's the problem? I think HTable should connect to ZooKeeper at port 21818 configured at client side instead of 12345 in hbase-site.xml and It might be caused by "HBaseConfiguration.addHbaseResources(conf);" in HRegionPartitioner#setConf(Configuration). And this might mean that all of client side configurations, also configured in hbase-site.xml, are overwritten caused by this problem. Thanks. 2011/8/21 Takuya UESHIN <[email protected]>: > Hi, > > I found that HRegionPartitioner has the same problem as HBASE-3111. > > Some tests I'm writing for MapReduce using HRegionPartitioner don't succeed > because HRegionPartitioiner overwrites the test specific configuration > "hbase.zookeeper.property.clientPort" : 21818 > with that in hbase-site.xml (or hbase-default.xml). > > I think this is caused by > "HBaseConfiguration.addHbaseResources(conf);" in > HRegionPartitioner#setConf(Configuration). > > > You can check this situation using TestTableMapReduce. > > Add HRegionPartitioner.class to the 4th argument of > TableMapReduceUtil#initTableReducerJob(). > And delete "hbase.zookeeper.property.clientPort" property from > hbase-site.xml or change it. > > > Thanks. > > -- > Takuya UESHIN > Tokyo, Japan > > http://twitter.com/ueshin > -- Takuya UESHIN Tokyo, Japan http://twitter.com/ueshin
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- /** * Copyright 2007 The Apache Software Foundation * * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ --> <configuration> <property> <name>hbase.regionserver.msginterval</name> <value>1000</value> <description>Interval between messages from the RegionServer to HMaster in milliseconds. Default is 15. Set this value low if you want unit tests to be responsive. </description> </property> <property> <name>hbase.client.pause</name> <value>1000</value> <description>General client pause value. Used mostly as value to wait before running a retry of a failed get, region lookup, etc.</description> </property> <property> <name>hbase.client.retries.number</name> <value>10</value> <description>Maximum retries. Used as maximum for all retryable operations such as fetching of the root region from root region server, getting a cell's value, starting a row update, etc. Default: 10. </description> </property> <property> <name>hbase.server.thread.wakefrequency</name> <value>1000</value> <description>Time to sleep in between searches for work (in milliseconds). Used as sleep interval by service threads such as META scanner and log roller. </description> </property> <property> <name>hbase.master.event.waiting.time</name> <value>50</value> <description>Time to sleep between checks to see if a table event took place. </description> </property> <property> <name>hbase.regionserver.handler.count</name> <value>5</value> <description>Count of RPC Server instances spun up on RegionServers Same property is used by the HMaster for count of master handlers. Default is 10. </description> </property> <property> <name>hbase.master.info.port</name> <value>-1</value> <description>The port for the hbase master web UI Set to -1 if you do not want the info server to run. </description> </property> <property> <name>hbase.regionserver.info.port</name> <value>-1</value> <description>The port for the hbase regionserver web UI Set to -1 if you do not want the info server to run. </description> </property> <property> <name>hbase.regionserver.info.port.auto</name> <value>true</value> <description>Info server auto port bind. Enables automatic port search if hbase.regionserver.info.port is already in use. Enabled for testing to run multiple tests on one machine. </description> </property> <property> <name>hbase.master.lease.thread.wakefrequency</name> <value>3000</value> <description>The interval between checks for expired region server leases. This value has been reduced due to the other reduced values above so that the master will notice a dead region server sooner. The default is 15 seconds. </description> </property> <property> <name>hbase.regionserver.optionalcacheflushinterval</name> <value>1000</value> <description> Amount of time to wait since the last time a region was flushed before invoking an optional cache flush. Default 60,000. </description> </property> <property> <name>hbase.regionserver.safemode</name> <value>false</value> <description> Turn on/off safe mode in region server. Always on for production, always off for tests. </description> </property> <property> <name>hbase.hregion.max.filesize</name> <value>67108864</value> <description> Maximum desired file size for an HRegion. If filesize exceeds value + (value / 2), the HRegion is split in two. Default: 256M. Keep the maximum filesize small so we split more often in tests. </description> </property> <property> <name>hadoop.log.dir</name> <value>${user.dir}/../logs</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>12345</value> <description>Property from ZooKeeper's config zoo.cfg. The port at which the clients will connect. </description> </property> <property> <name>hbase.defaults.for.version.skip</name> <value>true</value> <description> Set to true to skip the 'hbase.defaults.for.version'. Setting this to true can be useful in contexts other than the other side of a maven generation; i.e. running in an ide. You'll want to set this boolean to true to avoid seeing the RuntimException complaint: "hbase-default.xml file seems to be for and old version of HBase (@@@VERSION@@@), this version is X.X.X-SNAPSHOT" </description> </property> </configuration>
