Here is the tail of output from IntegrationTestRecoveryRSLossBetweenInserts
2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/opt/HDP/hadoop-2.1.0-SNAPSHOT/lib/native 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-279.el6.x86_64 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root 2013-07-08 17:43:01,171 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/opt/HDP/hbase-0.95.2-SNAPSHOT 2013-07-08 17:43:01,172 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000 watcher=hconnection-0x3ea86d12 2013-07-08 17:43:01,199 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3ea86d12 connecting to ZooKeeper ensemble= lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 2013-07-08 17:43:01,206 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 2013-07-08 17:43:01,212 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket connection established to lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session 2013-07-08 17:43:01,225 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid = 0x13fbfcea423068e, negotiated timeout = 30000 2013-07-08 17:43:01,401 INFO [main] mttr.AbstractIntegrationTestRecovery: Creating table myTableName_1373330579708 with 10 splits. 2013-07-08 17:43:01,423 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:02,387 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:02,496 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:02,605 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:02,714 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:02,924 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:06,324 INFO [main] mttr.AbstractIntegrationTestRecovery: Pre-split table created successfully in 4923ms 2013-07-08 17:43:06,326 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000 watcher=hconnection-0x244e5056 2013-07-08 17:43:06,327 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x244e5056 connecting to ZooKeeper ensemble= lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 2013-07-08 17:43:06,328 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 2013-07-08 17:43:06,330 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket connection established to lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session 2013-07-08 17:43:06,339 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid = 0x33fbfcea77e069a, negotiated timeout = 30000 2013-07-08 17:43:06,354 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:06,401 INFO [main] mttr.AbstractIntegrationTestRecovery: Moving regions of table myTableName_1373330579708 away from lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163 2013-07-08 17:43:06,405 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000 watcher=catalogtracker-on-hconnection-0x3ea86d12 2013-07-08 17:43:06,406 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 2013-07-08 17:43:06,406 DEBUG [main] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62 2013-07-08 17:43:06,407 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 2013-07-08 17:43:06,408 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket connection established to lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, initiating session 2013-07-08 17:43:06,416 INFO [main-SendThread( lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server lvsdpehdc25dn0484.stratus.lvs.ebay.com/10.8.142.13:2181, sessionid = 0x33fbfcea77e069b, negotiated timeout = 30000 2013-07-08 17:43:06,423 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,, 2013-07-08 17:43:06,432 DEBUG [main] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@6af37a62 2013-07-08 17:43:06,441 INFO [main] zookeeper.ZooKeeper: Session: 0x33fbfcea77e069b closed 2013-07-08 17:43:06,442 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2013-07-08 17:43:06,734 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 sessionTimeout=30000 watcher=catalogtracker-on-hconnection-0x3ea86d12 2013-07-08 17:43:06,736 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x3ea86d12 connecting to ZooKeeper ensemble=lvsdpehdc25dn0483.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181, lvsdpehdc25dn0484.stratus.lvs.ebay.com:2181 2013-07-08 17:43:06,736 DEBUG [main] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09 2013-07-08 17:43:06,736 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181. Will not attempt to authenticate using SASL (Unable to locate a login configuration) 2013-07-08 17:43:06,737 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Socket connection established to lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, initiating session 2013-07-08 17:43:06,744 INFO [main-SendThread( lvsdpehdc25dn0482.stratus.lvs.ebay.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server lvsdpehdc25dn0482.stratus.lvs.ebay.com/10.8.142.52:2181, sessionid = 0x13fbfcea423068f, negotiated timeout = 30000 2013-07-08 17:43:06,753 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,, 2013-07-08 17:43:06,764 DEBUG [main] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1e406b09 2013-07-08 17:43:06,772 INFO [main] zookeeper.ZooKeeper: Session: 0x13fbfcea423068f closed 2013-07-08 17:43:06,772 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2013-07-08 17:43:06,803 INFO [main] mttr.AbstractIntegrationTestRecovery: Moved 0 regions of table myTableName_1373330579708 away from lvsdpehdc25dn0406.stratus.lvs.ebay.com,60020,1373316676163 2013-07-08 17:43:06,806 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:43:07,184 INFO [main] hbase.ClusterManager: Executing remote command: sudo /sbin/iptables -A INPUT -i lo -j ACCEPT;sudo /sbin/iptables -A OUTPUT -o lo -j ACCEPT;sudo /sbin/iptables -A INPUT -p tcp -s 0/0 -d lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport 513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P FORWARD DROP; , hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com 2013-07-08 17:43:07,185 INFO [main] util.Shell: Executing full command [/usr/bin/ssh lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables -A INPUT -i lo -j ACCEPT;sudo /sbin/iptables -A OUTPUT -o lo -j ACCEPT;sudo /sbin/iptables -A INPUT -p tcp -s 0/0 -d lvsdpehdc25dn0615.stratus.lvs.ebay.com --sport 513:65535 --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT;sudo /sbin/iptables -A OUTPUT -p tcp -s lvsdpehdc25dn0615.stratus.lvs.ebay.com -d 0/0 --sport 22 --dport 513:65535 -m state --state ESTABLISHED -j ACCEPT;sudo /sbin/iptables -P INPUT DROP;sudo /sbin/iptables -P OUTPUT DROP;sudo /sbin/iptables -P FORWARD DROP;"] 2013-07-08 17:50:38,229 INFO [main] hbase.ClusterManager: Executed remote command, exit code:0 , output: 2013-07-08 17:50:38,662 INFO [main] mttr.AbstractIntegrationTestRecovery: Detection took: 29 2013-07-08 17:50:38,907 INFO [main] mttr.AbstractIntegrationTestRecovery: Regions available after: 245 2013-07-08 17:50:38,909 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:50:38,985 INFO [main] mttr.AbstractIntegrationTestRecovery: Total failure fix took: 323 2013-07-08 17:50:38,986 DEBUG [main] client.ClientScanner: Scan table=.META., startRow=myTableName_1373330579708,,00000000000000 2013-07-08 17:50:39,037 INFO [main] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x33fbfcea77e069a 2013-07-08 17:50:39,053 INFO [main] zookeeper.ZooKeeper: Session: 0x33fbfcea77e069a closed 2013-07-08 17:50:39,053 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2013-07-08 17:50:39,055 INFO [main] hbase.ClusterManager: Executing remote command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush , hostname:lvsdpehdc25dn0614.stratus.lvs.ebay.com 2013-07-08 17:50:39,056 INFO [main] util.Shell: Executing full command [/usr/bin/ssh lvsdpehdc25dn0614.stratus.lvs.ebay.com "sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush"] 2013-07-08 17:50:39,176 INFO [main] hbase.ClusterManager: Executed remote command, exit code:0 , output: 2013-07-08 17:50:39,177 INFO [main] hbase.ClusterManager: Executing remote command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush , hostname:lvsdpehdc25dn0615.stratus.lvs.ebay.com 2013-07-08 17:50:39,177 INFO [main] util.Shell: Executing full command [/usr/bin/ssh lvsdpehdc25dn0615.stratus.lvs.ebay.com "sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush"] 2013-07-08 17:50:39,293 INFO [main] hbase.ClusterManager: Executed remote command, exit code:0 , output: 2013-07-08 17:50:39,293 INFO [main] hbase.ClusterManager: Executing remote command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush , hostname:lvsdpehdc25dn0616.stratus.lvs.ebay.com 2013-07-08 17:50:39,293 INFO [main] util.Shell: Executing full command [/usr/bin/ssh lvsdpehdc25dn0616.stratus.lvs.ebay.com "sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush"] 2013-07-08 17:50:39,416 INFO [main] hbase.ClusterManager: Executed remote command, exit code:0 , output: 2013-07-08 17:50:39,416 INFO [main] hbase.ClusterManager: Executing remote command: sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush , hostname:lvsdpehdc25dn0617.stratus.lvs.ebay.com 2013-07-08 17:50:39,416 INFO [main] util.Shell: Executing full command [/usr/bin/ssh lvsdpehdc25dn0617.stratus.lvs.ebay.com "sudo /sbin/iptables -P INPUT ACCEPT;sudo /sbin/iptables -P OUTPUT ACCEPT;sudo /sbin/iptables -P FORWARD ACCEPT;sudo /sbin/iptables --flush"] 2013-07-08 17:50:39,538 INFO [main] hbase.ClusterManager: Executed remote command, exit code:0 , output: 2013-07-08 17:50:39,539 INFO [main] util.PerformanceChecker: max expected=45000, actual=29, ok=true 2013-07-08 17:50:39,539 INFO [main] util.PerformanceChecker: max expected=15000, actual=323, ok=true 2013-07-08 17:50:39,541 INFO [main] client.HBaseAdmin: Started disable of myTableName_1373330579708 2013-07-08 17:50:39,568 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:39,669 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:39,771 DEBUG [main] client.HBaseAdmin: Sleeping= 100ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:39,872 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:40,074 DEBUG [main] client.HBaseAdmin: Sleeping= 200ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:40,275 DEBUG [main] client.HBaseAdmin: Sleeping= 400ms, waiting for all regions to be disabled in myTableName_1373330579708 2013-07-08 17:50:40,677 INFO [main] client.HBaseAdmin: Disabled myTableName_1373330579708 2013-07-08 17:50:41,232 INFO [main] client.HBaseAdmin: Deleted myTableName_1373330579708 Time: 461.624
