[
https://issues.apache.org/jira/browse/HBASE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638145#comment-13638145
]
Yu Li commented on HBASE-7139:
------------------------------
Met with the same issue.
We have one case here that the empty table is pre-created w/o region pre-split,
and to improve data loading rate, we prefer to use bulkload, so we need to
split the empty table after it's created. Although we can work-around it by
getting the table descriptor, dropping the table and re-creating it with region
pre-split, I think the issue described here still worth to resolve.
Meanwhile, not sure whether and how this is related to HBASE-3492(maybe a
special case that fix for HBASE-3492 not covered?), will look into more details
when time allows.
> RegionSplitter's rollingSplit cannot split empty table or region
> ----------------------------------------------------------------
>
> Key: HBASE-7139
> URL: https://issues.apache.org/jira/browse/HBASE-7139
> Project: HBase
> Issue Type: Bug
> Components: util
> Affects Versions: 0.94.1
> Environment: Ubuntu Server 10.04, Hadoop 1.0.3
> Reporter: Davey Yan
>
> Execute rollingSplit on empty table or region will cause original region in
> SPLITTING state, and will be not out of transition list unless unassigning it
> (with forse) in shell.
> Steps to reproduce:
> {code}
> shell> create 'test2', 'i'
> {code}
> {noformat}
> $ bin/hbase org.apache.hadoop.hbase.util.RegionSplitter -r -o 2 test2
> HexStringSplit
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 GMT
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:host.name=dev-vm0
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_29
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/jdk1.6.0_29/jre
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=/opt/hbase/bin/../conf:/usr/lib/jvm/default-java/lib/tools.jar:/opt/hbase/bin/..:/opt/hbase/bin/../hbase-0.94.1.jar:/opt/hbase/bin/../hbase-0.94.1-tests.jar:/opt/hbase/bin/../lib/activation-1.1.jar:/opt/hbase/bin/../lib/asm-3.1.jar:/opt/hbase/bin/../lib/avro-1.5.3.jar:/opt/hbase/bin/../lib/avro-ipc-1.5.3.jar:/opt/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/opt/hbase/bin/../lib/commons-beanutils-core-1.8.0.jar:/opt/hbase/bin/../lib/commons-cli-1.2.jar:/opt/hbase/bin/../lib/commons-codec-1.4.jar:/opt/hbase/bin/../lib/commons-collections-3.2.1.jar:/opt/hbase/bin/../lib/commons-configuration-1.6.jar:/opt/hbase/bin/../lib/commons-digester-1.8.jar:/opt/hbase/bin/../lib/commons-el-1.0.jar:/opt/hbase/bin/../lib/commons-httpclient-3.1.jar:/opt/hbase/bin/../lib/commons-io-2.1.jar:/opt/hbase/bin/../lib/commons-lang-2.5.jar:/opt/hbase/bin/../lib/commons-logging-1.1.1.jar:/opt/hbase/bin/../lib/commons-math-2.1.jar:/opt/hbase/bin/../lib/commons-net-1.4.1.jar:/opt/hbase/bin/../lib/core-3.1.1.jar:/opt/hbase/bin/../lib/guava-11.0.2.jar:/opt/hbase/bin/../lib/hadoop-core-1.0.3.jar:/opt/hbase/bin/../lib/high-scale-lib-1.1.1.jar:/opt/hbase/bin/../lib/httpclient-4.1.2.jar:/opt/hbase/bin/../lib/httpcore-4.1.3.jar:/opt/hbase/bin/../lib/jackson-core-asl-1.8.8.jar:/opt/hbase/bin/../lib/jackson-jaxrs-1.8.8.jar:/opt/hbase/bin/../lib/jackson-mapper-asl-1.8.8.jar:/opt/hbase/bin/../lib/jackson-xc-1.8.8.jar:/opt/hbase/bin/../lib/jamon-runtime-2.3.1.jar:/opt/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/opt/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/opt/hbase/bin/../lib/jaxb-api-2.1.jar:/opt/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/opt/hbase/bin/../lib/jersey-core-1.8.jar:/opt/hbase/bin/../lib/jersey-json-1.8.jar:/opt/hbase/bin/../lib/jersey-server-1.8.jar:/opt/hbase/bin/../lib/jettison-1.1.jar:/opt/hbase/bin/../lib/jetty-6.1.26.jar:/opt/hbase/bin/../lib/jetty-util-6.1.26.jar:/opt/hbase/bin/../lib/jruby-complete-1.6.5.jar:/opt/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/opt/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/opt/hbase/bin/../lib/jsr305-1.3.9.jar:/opt/hbase/bin/../lib/junit-4.10-HBASE-1.jar:/opt/hbase/bin/../lib/libthrift-0.8.0.jar:/opt/hbase/bin/../lib/log4j-1.2.16.jar:/opt/hbase/bin/../lib/metrics-core-2.1.2.jar:/opt/hbase/bin/../lib/netty-3.2.4.Final.jar:/opt/hbase/bin/../lib/protobuf-java-2.4.0a.jar:/opt/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/hbase/bin/../lib/slf4j-api-1.4.3.jar:/opt/hbase/bin/../lib/slf4j-log4j12-1.4.3.jar:/opt/hbase/bin/../lib/snappy-java-1.0.3.2.jar:/opt/hbase/bin/../lib/stax-api-1.0.1.jar:/opt/hbase/bin/../lib/velocity-1.7.jar:/opt/hbase/bin/../lib/xmlenc-0.52.jar:/opt/hbase/bin/../lib/zookeeper-3.4.3.jar:/opt/hbase/bin/../libextra/mybk-commons-cc.jar:/opt/hbase/bin/../libextra/hbase.jar:/opt/hbase/bin/../libextra/sfdcloud-hbase.jar:
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.library.path=/opt/hbase/bin/../lib/native/Linux-i386-32
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:os.arch=i386
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.32-33-generic
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client environment:user.name=pcer
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/home/pcer
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/hbase-0.94.1
> 12/11/09 16:24:57 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=dev-vm0:2181 sessionTimeout=60000 watcher=hconnection
> 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Opening socket connection to
> server /172.16.20.140:2181
> 12/11/09 16:24:57 INFO zookeeper.RecoverableZooKeeper: The identifier of this
> process is 16033@dev-vm0
> 12/11/09 16:24:57 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration occurred
> when trying to find JAAS configuration.
> 12/11/09 16:24:57 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS configuration.
> 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Socket connection established to
> dev-vm0/172.16.20.140:2181, initiating session
> 12/11/09 16:24:57 INFO zookeeper.ClientCnxn: Session establishment complete
> on server dev-vm0/172.16.20.140:2181, sessionid = 0x13ad3e9ba7001c3,
> negotiated timeout = 40000
> 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a;
> serverName=dev-vm0,46871,1352175042380
> 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for .META.,,1.1028785192 is dev-vm0:46871
> 12/11/09 16:24:58 DEBUG client.MetaScanner: Scanning .META. starting at
> row=test2,,00000000000000 for max=10 rows using
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:24:58 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is
> dev-vm0:46871
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: No _balancedSplit file.
> Calculating splits...
> 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at
> row=test2,,00000000000000 for max=2147483647 rows using
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Table test2 has 1 regions that
> will be split.
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Will Split [00000000 , ffffffff)
> at 7fffffff
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Bucketing regions by
> regionserver...
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Done with bucketing. Split time!
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: 1 RS have regions to splt.
> 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at
> row=test2,,00000000000000 for max=2147483647 rows using
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Finding a region on dev-vm0:46871
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Splitting at 7fffffff
> 12/11/09 16:24:59 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=dev-vm0:2181 sessionTimeout=60000
> watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Opening socket connection to
> server /172.16.20.140:2181
> 12/11/09 16:24:59 WARN client.ZooKeeperSaslClient: SecurityException:
> java.lang.SecurityException: Unable to locate a login configuration occurred
> when trying to find JAAS configuration.
> 12/11/09 16:24:59 INFO client.ZooKeeperSaslClient: Client will not
> SASL-authenticate because the default JAAS configuration section 'Client'
> could not be found. If you are not using SASL, you may ignore this. On the
> other hand, if you expected SASL to work, please fix your JAAS configuration.
> 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Socket connection established to
> dev-vm0/172.16.20.140:2181, initiating session
> 12/11/09 16:24:59 INFO zookeeper.RecoverableZooKeeper: The identifier of this
> process is 16033@dev-vm0
> 12/11/09 16:24:59 DEBUG catalog.CatalogTracker: Starting catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@691dee
> 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: Session establishment complete
> on server dev-vm0/172.16.20.140:2181, sessionid = 0x13ad3e9ba7001c4,
> negotiated timeout = 40000
> 12/11/09 16:24:59 DEBUG client.ClientScanner: Creating scanner over .META.
> starting at key 'test2,,'
> 12/11/09 16:24:59 DEBUG client.ClientScanner: Advancing internal scanner to
> startKey at 'test2,,'
> 12/11/09 16:24:59 DEBUG client.ClientScanner: Creating scanner over .META.
> starting at key 'test2,,'
> 12/11/09 16:24:59 DEBUG client.ClientScanner: Advancing internal scanner to
> startKey at 'test2,,'
> 12/11/09 16:24:59 DEBUG client.ClientScanner: Finished with scanning at {NAME
> => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}
> 12/11/09 16:24:59 DEBUG catalog.CatalogTracker: Stopping catalog tracker
> org.apache.hadoop.hbase.catalog.CatalogTracker@691dee
> 12/11/09 16:24:59 INFO zookeeper.ClientCnxn: EventThread shut down
> 12/11/09 16:24:59 INFO zookeeper.ZooKeeper: Session: 0x13ad3e9ba7001c4 closed
> 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a;
> serverName=dev-vm0,46871,1352175042380
> 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for .META.,,1.1028785192 is dev-vm0:46871
> 12/11/09 16:24:59 DEBUG client.MetaScanner: Scanning .META. starting at
> row=test2,,00000000000000 for max=10 rows using
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:24:59 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is
> dev-vm0:46871
> 12/11/09 16:24:59 DEBUG util.RegionSplitter: Split Scan: 0 finished / 1 split
> wait / 0 reference wait
> 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation:
> Looked up root region location,
> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a;
> serverName=dev-vm0,46871,1352175042380
> 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for .META.,,1.1028785192 is dev-vm0:46871
> 12/11/09 16:25:29 DEBUG client.MetaScanner: Scanning .META. starting at
> row=test2,,00000000000000 for max=10 rows using
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@ba5c7a
> 12/11/09 16:25:29 DEBUG client.HConnectionManager$HConnectionImplementation:
> Cached location for test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. is
> dev-vm0:46871
> 12/11/09 16:25:29 DEBUG util.RegionSplitter: Split Scan: 0 finished / 1 split
> wait / 0 reference wait
> ...repeat forever...
> {noformat}
> Log of server site:
> {noformat}
> 2012-11-09 16:24:59,597 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Split requested for
> test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5..
> compaction_queue=(0:0), split_queue=1
> 2012-11-09 16:24:59,597 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Starting split of
> region test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.
> 2012-11-09 16:24:59,597 DEBUG
> org.apache.hadoop.hbase.regionserver.SplitTransaction:
> regionserver:46871-0x13ad3e9ba700001 Creating ephemeral node for
> 6c53236aff39bca3e77e79fcf2419ba5 in SPLITTING state
> 2012-11-09 16:24:59,601 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:46871-0x13ad3e9ba700001 Attempting to transition node
> 6c53236aff39bca3e77e79fcf2419ba5 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLITTING
> 2012-11-09 16:24:59,613 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: Processed session
> termination for sessionid: 0x13ad3e9ba7001c4
> 2012-11-09 16:24:59,615 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:46871-0x13ad3e9ba700001 Successfully transitioned node
> 6c53236aff39bca3e77e79fcf2419ba5 from RS_ZK_REGION_SPLITTING to
> RS_ZK_REGION_SPLITTING
> 2012-11-09 16:24:59,618 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Closing test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.: disabling
> compactions & flushes
> 2012-11-09 16:24:59,618 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
> Updates disabled for region
> test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.
> 2012-11-09 16:24:59,624 WARN org.apache.zookeeper.server.NIOServerCnxn:
> caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x13ad3e9ba7001c4, likely client has closed socket
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224)
> at java.lang.Thread.run(Thread.java:662)
> 2012-11-09 16:24:59,625 INFO org.apache.zookeeper.server.NIOServerCnxn:
> Closed socket connection for client /172.16.20.140:38233 which had sessionid
> 0x13ad3e9ba7001c4
> 2012-11-09 16:24:59,626 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: Handling
> transition=RS_ZK_REGION_SPLITTING, server=dev-vm0,46871,1352175042380,
> region=6c53236aff39bca3e77e79fcf2419ba5
> 2012-11-09 16:24:59,627 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> closed i
> 2012-11-09 16:24:59,627 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Closed test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.
> 2012-11-09 16:24:59,627 INFO
> org.apache.hadoop.hbase.regionserver.SplitRequest: Running rollback/cleanup
> of failed split of test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.;
> null
> java.lang.IllegalArgumentException
> at
> java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:589)
> at
> java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:514)
> at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:84)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransaction.splitStoreFiles(SplitTransaction.java:592)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:290)
> at
> org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:452)
> at
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 2012-11-09 16:24:59,655 INFO org.apache.hadoop.hbase.regionserver.Store: time
> to purge deletes set to 0ms in store null
> 2012-11-09 16:24:59,661 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction: Cleaned up old failed
> split transaction detritus:
> hdfs://dev-vm0:9000/hbase/test2/6c53236aff39bca3e77e79fcf2419ba5/.splits
> 2012-11-09 16:24:59,662 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> Onlined test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.; next
> sequenceid=1
> 2012-11-09 16:24:59,663 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:46871-0x13ad3e9ba700001 Deleting existing unassigned node for
> 6c53236aff39bca3e77e79fcf2419ba5 that is in expected state
> RS_ZK_REGION_SPLITTING
> 2012-11-09 16:24:59,667 DEBUG
> org.apache.hadoop.hbase.master.AssignmentManager: The znode of region
> test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5. has been deleted.
> 2012-11-09 16:24:59,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:46871-0x13ad3e9ba700001 Successfully deleted unassigned node for
> region 6c53236aff39bca3e77e79fcf2419ba5 in expected state
> RS_ZK_REGION_SPLITTING
> 2012-11-09 16:24:59,668 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign:
> regionserver:46871-0x13ad3e9ba700001 Deleting existing unassigned node for
> 6c53236aff39bca3e77e79fcf2419ba5 that is in expected state
> RS_ZK_REGION_SPLITTING
> 2012-11-09 16:24:59,669 INFO
> org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of
> failed split of test2,,1352449471288.6c53236aff39bca3e77e79fcf2419ba5.
> ... org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition
> timed out
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
