>> If you trace all the references to 0000003404803994 in the region >> server, what do you see? I think that most of the time is spent >> opening the region sequentially, it'd be nice to confirm.
I have attached the log snippets on the master-RS-master conversation.. Vidhya In the master: 2010-08-27 22:59:02,642 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current assignment of DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d3 25932ff1. is not valid; serverAddress=, startCode=0 unknown.2010-08-27 22:59:02,643 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Creating UNASSIGNED region 73c0f8fdb8ffbc20b9a239d325932ff1 in state = M2ZK _REGION_OFFLINE2010-08-27 22:59:02,645 DEBUG org.apache.hadoop.hbase.master.HMaster: Event NodeCreated with state SyncConnected with path /hbase/UNASSIGNED/73c0f8fdb8ffbc20 b9a239d325932ff1 2010-08-27 22:59:02,645 DEBUG org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event NodeCreated with path /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239 d325932ff12010-08-27 22:59:02,645 DEBUG org.apache.hadoop.hbase.master.ZKUnassignedWatcher: ZK-EVENT-PROCESS: Got zkEvent NodeCreated state:SyncConnected path:/hbase/U NASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff12010-08-27 22:59:02,645 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: <b3130520.yst.yahoo.net,b3130560.yst.yahoo.net,b3130600.yst.yahoo.net,b3130 640.yst.yahoo.net,b3130680.yst.yahoo.net:/hbase,org.apache.hadoop.hbase.master.HMaster>Created ZNode /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff1 in ZooKeeper 2010-08-27 22:59:02,646 DEBUG org.apache.hadoop.hbase.master.RegionManager: Created/updated UNASSIGNED zNode DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. in state M2ZK_REGION_OFFLINE 2010-08-27 22:59:02,646 DEBUG org.apache.hadoop.hbase.master.ZKUnassignedWatcher: Got event type [ M2ZK_REGION_OFFLINE ] for region 73c0f8fdb8ffbc20b9a239d325932ff1 2010-08-27 22:59:02,646 DEBUG org.apache.hadoop.hbase.master.HMaster: Event NodeChildrenChanged with state SyncConnected with path /hbase/UNASSIGNED2010-08-27 22:59:02,646 DEBUG org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event NodeChildrenChanged with path /hbase/UNASSIGNED 2010-08-27 22:59:02,646 DEBUG org.apache.hadoop.hbase.master.ZKUnassignedWatcher: ZK-EVENT-PROCESS: Got zkEvent NodeChildrenChanged state:SyncConnected path:/hbase/UNASSIGNED 2010-08-27 22:59:02,647 DEBUG org.apache.hadoop.hbase.master.RegionManager: Assigning for serverName=b3130020.yst.yahoo.net,60020,1282940735627, load=(requests=0, regions=76, usedHeap=80, maxHeap=7993): total nregions to assign=1, regions to give other servers than this=0, isMetaAssign=false 2010-08-27 22:59:02,647 DEBUG org.apache.hadoop.hbase.master.RegionManager: Assigning serverName=b3130020.yst.yahoo.net,60020,1282940735627, load=(requests=0, regions=76, usedHeap=80, maxHeap=7993) 1 regions 2010-08-27 22:59:02,647 INFO org.apache.hadoop.hbase.master.RegionManager: Assigning region DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. to b3130020.yst.yahoo.net,60020,1282940735627 2010-08-27 22:59:02,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: No need to update UNASSIGNED region 73c0f8fdb8ffbc20b9a239d325932ff1 as it already exists in state = M2ZK_REGION_OFFLINE 2010-08-27 22:59:02,653 DEBUG org.apache.hadoop.hbase.master.RegionManager: Created UNASSIGNED zNode DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. in state M2ZK_REGION_OFFLINE 2010-08-27 22:59:02,657 DEBUG org.apache.hadoop.hbase.master.HMaster: Event NodeDataChanged with state SyncConnected with path /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff1 In the region server: 2010-08-27 22:59:02,658 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. 2010-08-27 22:59:02,658 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. 2010-08-27 22:59:02,659 DEBUG org.apache.hadoop.hbase.regionserver.RSZookeeperUpdater: Updating ZNode /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff1 with [RS2ZK_REGION_OPENING] expected version = 0 2010-08-27 22:59:02,660 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeDataChanged, path: /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff1 2010-08-27 22:59:02,660 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Creating region DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. 2010-08-27 22:59:02,718 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://b3130080.yst.yahoo.net:4600/hbase/DocDB/73c0f8fdb8ffbc20b9a239d325932ff1/bigColumn/7281813866313705842, isReference=false, isBulkLoadResult=true, bulkLoadTS=1282903668546, majorCompaction=true 2010-08-27 22:59:02,720 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1.; next sequenceid=1 2010-08-27 22:59:02,720 DEBUG org.apache.hadoop.hbase.regionserver.RSZookeeperUpdater: Updating ZNode /hbase/UNASSIGNED/73c0f8fdb8ffbc20b9a239d325932ff1 with [RS2ZK_REGION_OPENED] expected version = 1 (no more references in the region server..) And finally, in the master: 2010-08-27 23:18:53,591 INFO org.apache.hadoop.hbase.master.RegionServerOperation: DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d325932ff1. open on b3130020.yst.yahoo.net,60020,1282940735627 2010-08-27 23:18:53,591 INFO org.apache.hadoop.hbase.master.RegionServerOperation: Updated row DocDB,0000003404803994,1282947439133.73c0f8fdb8ffbc20b9a239d32 5932ff1. in region .META.,,1 with startcode=1282940735627, server=b3130020.yst.yahoo.net:60020
