Davey Yan created HBASE-7139: -------------------------------- Summary: RegionSplitter's rollingSplit cannot split empty table or region Key: HBASE-7139 URL: https://issues.apache.org/jira/browse/HBASE-7139 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.94.1 Environment: Ubuntu Server 10.04, Hadoop 1.0.3 Reporter: Davey Yan
Execute rollingSplit on empty table or region will cause original region in SPLITTING state, and will be not out of transition list unless unassigning it (with forse) in shell. Steps to reproduce: {code} shell> create 'test2', 'i' {code} {noformat} $ bin/hbase org.apache.hadoop.hbase.util.RegionSplitter -r -o 2 test2 HexStringSplit 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:host.name=dev-vm0 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_29 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/jdk1.6.0_29/jre 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/opt/hbase/bin/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/opt/hbase/bin/..:/opt/hbase/bin/../hbase-0.94.1.jar:/opt/hbase/bin/../hbase-0.94.1-tests.jar:/opt/hbase/bin/../lib/activation-1.1.jar:/opt/hbase/bin/../lib/asm-3.1.jar:/opt/hbase/bin/../lib/avro-1.5.3.jar:/opt/hbase/bin/../lib/avro-ipc-1.5.3.jar:/opt/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/hbase/bin/../lib/commons-cli-1.2.jar:/opt/hbase/bin/../lib/commons-codec-1.4.jar:/opt/hbase/bin/../lib/commons-collections-3.2.1.jar:/opt/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/hbase/bin/../lib/commons-digester-1.8.jar:/opt/hbase/bin/../lib/commons-el-1.0.jar:/opt/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/hbase/bin/../lib/commons-io-2.1.jar:/opt/hbase/bin/../lib/commons-lang-2.5.jar:/opt/hbase/bin/../lib/commons-logging-1.1.1.jar:/opt/hbase/bin/../lib/commons-math-2.1.jar:/opt/hbase/bin/../lib/commons-net-1.4.1.jar:/opt/hbase/bin/../lib/core-3.1.1.jar:/opt/hbase/bin/../lib/guava-11.0.2.jar:/opt/hbase/bin/../lib/hadoop-core-1.0.3.jar:/opt/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/opt/hbase/bin/../lib/httpclient-4.1.2.jar:/opt/hbase/bin/../lib/httpcore-4.1.3.jar:/opt/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/opt/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/opt/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/opt/hbase/bin/../lib/jackson-xc-1.8.8.jar:/opt/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/opt/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/opt/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/opt/hbase/bin/../lib/jaxb-api-2.1.jar:/opt/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/bin/../lib/jersey-core-1.8.jar:/opt/hbase/bin/../lib/jersey-json-1.8.jar:/opt/hbase/bin/../lib/jersey-server-1.8.jar:/opt/hbase/bin/../lib/jettison-1.1.jar:/opt/hbase/bin/../lib/jetty-6.1.26.jar:/opt/hbase/bin/../lib/jetty-util-6.1.26.jar:/opt/hbase/bin/../lib/jruby-complete-1.6.5.jar:/opt/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/opt/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/bin/../lib/jsr305-1.3.9.jar:/opt/hbase/bin/../lib/junit-4.10-HBASE-1.jar:/opt/hbase/bin/../lib/libthrift-0.8.0.jar:/opt/hbase/bin/../lib/log4j-1.2.16.jar:/opt/hbase/bin/../lib/metrics-core-2.1.2.jar:/opt/hbase/bin/../lib/netty-3.2.4.Final.jar:/opt/hbase/bin/../lib/protobuf-java-2.4.0a.jar:/opt/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/bin/../lib/slf4j-api-1.4.3.jar:/opt/hbase/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/opt/hbase/bin/../lib/stax-api-1.0.1.jar:/opt/hbase/bin/../lib/velocity-1.7.jar:/opt/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hbase/bin/../lib/zookeeper-3.4.3.jar:/opt/hbase/bin/../libextra/mybk-commons-cc.jar:/opt/hbase/bin/../libextra/hbase.jar:/opt/hbase/bin/../libextra/sfdcloud-hbase.jar: 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/opt/hbase/bin/../lib/native/Linux-i386-32 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-33-generic 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:user.name=pcer 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/pcer 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:user.dir=/opt/hbase-0.94.1 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=dev-vm0:2181 sessionTimeout=60000 watcher=hconnection 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Opening socket connection to server /172.16.20.140:2181 12/11/09 16:24:57 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16033@dev-vm0 12/11/09 16:24:57 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 12/11/09 16:24:57 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Socket connection established to dev-vm0/172.16.20.140:2181, initiating session 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Session establishment complete on server dev-vm0/172.16.20.140:2181, sessionid = 0x13ad3e9ba7001c3, negotiated timeout = 40000 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a; serverName=dev-vm0,46871,1352175042380 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is dev-vm0:46871 12/11/09 16:24:58 DEBUG client.MetaScanner: Scanning .META. starting at row=test2,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is dev-vm0:46871 12/11/09 16:24:59 DEBUG util.RegionSplitter: No _balancedSplit file. Calculating splits... 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at row=test2,,00000000000000 for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:24:59 DEBUG util.RegionSplitter: Table test2 has 1 regions that will be split. 12/11/09 16:24:59 DEBUG util.RegionSplitter: Will Split [00000000 , ffffffff) at 7fffffff 12/11/09 16:24:59 DEBUG util.RegionSplitter: Bucketing regions by regionserver... 12/11/09 16:24:59 DEBUG util.RegionSplitter: Done with bucketing. Split time! 12/11/09 16:24:59 DEBUG util.RegionSplitter: 1 RS have regions to splt. 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at row=test2,,00000000000000 for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:24:59 DEBUG util.RegionSplitter: Finding a region on dev-vm0:46871 12/11/09 16:24:59 DEBUG util.RegionSplitter: Splitting at 7fffffff 12/11/09 16:24:59 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=dev-vm0:2181 sessionTimeout=60000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Opening socket connection to server /172.16.20.140:2181 12/11/09 16:24:59 WARN client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configuration. 12/11/09 16:24:59 INFO client.ZooKeeperSaslClient: Client will not SASL-authenticate because the default JAAS configuration section 'Client' could not be found. If you are not using SASL, you may ignore this. On the other hand, if you expected SASL to work, please fix your JAAS configuration. 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Socket connection established to dev-vm0/172.16.20.140:2181, initiating session 12/11/09 16:24:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 16033@dev-vm0 12/11/09 16:24:59 DEBUG catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@691dee 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Session establishment complete on server dev-vm0/172.16.20.140:2181, sessionid = 0x13ad3e9ba7001c4, negotiated timeout = 40000 12/11/09 16:24:59 DEBUG client.ClientScanner: Creating scanner over .META. starting at key 'test2,,' 12/11/09 16:24:59 DEBUG client.ClientScanner: Advancing internal scanner to startKey at 'test2,,' 12/11/09 16:24:59 DEBUG client.ClientScanner: Creating scanner over .META. starting at key 'test2,,' 12/11/09 16:24:59 DEBUG client.ClientScanner: Advancing internal scanner to startKey at 'test2,,' 12/11/09 16:24:59 DEBUG client.ClientScanner: Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 12/11/09 16:24:59 DEBUG catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@691dee 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: EventThread shut down 12/11/09 16:24:59 INFO zookeeper.ZooKeeper: Session: 0x13ad3e9ba7001c4 closed 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a; serverName=dev-vm0,46871,1352175042380 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is dev-vm0:46871 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at row=test2,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is dev-vm0:46871 12/11/09 16:24:59 DEBUG util.RegionSplitter: Split Scan: 0 finished / 1 split wait / 0 reference wait 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation: Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a; serverName=dev-vm0,46871,1352175042380 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for .META.,,1.1028785192 is dev-vm0:46871 12/11/09 16:25:29 DEBUG client.MetaScanner: Scanning .META. starting at row=test2,,00000000000000 for max=10 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation: Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is dev-vm0:46871 12/11/09 16:25:29 DEBUG util.RegionSplitter: Split Scan: 0 finished / 1 split wait / 0 reference wait {noformat} Log of server site: {noformat} 2012-11-09 16:24:59,597 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.. compaction_queue=(0:0), split_queue=1 2012-11-09 16:24:59,597 INFO org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of region test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. 2012-11-09 16:24:59,597 DEBUG org.apache.hadoop.hbase.regionserver.SplitTransaction: regionserver:46871-0x13ad3e9ba700001 Creating ephemeral node for 6c53236aff39bca3e77e79fcf2419ba5 in SPLITTING state 2012-11-09 16:24:59,601 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:46871-0x13ad3e9ba700001 Attempting to transition node 6c53236aff39bca3e77e79fcf2419ba5 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2012-11-09 16:24:59,613 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination for sessionid: 0x13ad3e9ba7001c4 2012-11-09 16:24:59,615 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:46871-0x13ad3e9ba700001 Successfully transitioned node 6c53236aff39bca3e77e79fcf2419ba5 from RS_ZK_REGION_SPLITTING to RS_ZK_REGION_SPLITTING 2012-11-09 16:24:59,618 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.: disabling compactions & flushes 2012-11-09 16:24:59,618 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. 2012-11-09 16:24:59,624 WARN org.apache.zookeeper.server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13ad3e9ba7001c4, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) at java.lang.Thread.run(Thread.java:662) 2012-11-09 16:24:59,625 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /172.16.20.140:38233 which had sessionid 0x13ad3e9ba7001c4 2012-11-09 16:24:59,626 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_SPLITTING, server=dev-vm0,46871,1352175042380, region=6c53236aff39bca3e77e79fcf2419ba5 2012-11-09 16:24:59,627 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed i 2012-11-09 16:24:59,627 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. 2012-11-09 16:24:59,627 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup of failed split of test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.; null java.lang.IllegalArgumentException at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:589) at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:514) at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:84) at org.apache.hadoop.hbase.regionserver.SplitTransaction.splitStoreFiles(SplitTransaction.java:592) at org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:290) at org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:452) at org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-11-09 16:24:59,655 INFO org.apache.hadoop.hbase.regionserver.Store: time to purge deletes set to 0ms in store null 2012-11-09 16:24:59,661 INFO org.apache.hadoop.hbase.regionserver.SplitTransaction: Cleaned up old failed split transaction detritus: hdfs://dev-vm0:9000/hbase/test2/6c53236aff39bca3e77e79fcf2419ba5/.splits 2012-11-09 16:24:59,662 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.; next sequenceid=1 2012-11-09 16:24:59,663 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:46871-0x13ad3e9ba700001 Deleting existing unassigned node for 6c53236aff39bca3e77e79fcf2419ba5 that is in expected state RS_ZK_REGION_SPLITTING 2012-11-09 16:24:59,667 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. has been deleted. 2012-11-09 16:24:59,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:46871-0x13ad3e9ba700001 Successfully deleted unassigned node for region 6c53236aff39bca3e77e79fcf2419ba5 in expected state RS_ZK_REGION_SPLITTING 2012-11-09 16:24:59,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:46871-0x13ad3e9ba700001 Deleting existing unassigned node for 6c53236aff39bca3e77e79fcf2419ba5 that is in expected state RS_ZK_REGION_SPLITTING 2012-11-09 16:24:59,669 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of failed split of test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. ... org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira