See http://hudson.zones.apache.org/hudson/job/HBase-Patch/191/changes
Changes:
[stack] HBASE-709 Deadlock while rolling WAL-log while finishing flush
------------------------------------------
[...truncated 26426 lines...]
[junit] 2008-06-26 04:50:06,374 DEBUG [RegionServer:0.worker]
regionserver.HRegion(472): Next sequence id for region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 is 35164
[junit] 2008-06-26 04:50:06,376 INFO [RegionServer:0.worker]
regionserver.HRegion(497): region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468/1971203659 available
[junit] 2008-06-26 04:50:06,376 DEBUG [RegionServer:0.worker]
regionserver.CompactSplitThread(120): Compaction requested for region:
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:07,146 INFO [RegionManager.rootScanner]
master.BaseScanner(149): RegionManager.rootScanner scanning meta region
{regionname: -ROOT-,,0, startKey: <>, server: 140.211.11.106:42051}
[junit] 2008-06-26 04:50:07,146 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(987): Started memcache flush for region -ROOT-,,0. Current
region memcache size 93.0
[junit] 2008-06-26 04:50:07,202 DEBUG [RegionManager.rootScanner]
master.BaseScanner(175): RegionManager.rootScannerREGION => {NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE =>
{NAME => '.META.', FAMILIES => [{NAME => 'info', VERSIONS => 1, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}, {NAME => 'historian', VERSIONS => 2147483647,
COMPRESSION => 'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH =>
2147483647, TTL => FOREVER, BLOOMFILTER => NONE}]}}, SERVER =>
'140.211.11.106:42051', STARTCODE => 1214455795906
[junit] 2008-06-26 04:50:07,204 INFO [RegionManager.rootScanner]
master.BaseScanner(229): RegionManager.rootScanner scan of meta region
{regionname: -ROOT-,,0, startKey: <>, server: 140.211.11.106:42051} complete
[junit] 2008-06-26 04:50:07,207 INFO [IPC Server handler 4 on 60000]
master.ServerManager(327): Received MSG_REPORT_PROCESS_OPEN:
TestUpgrade,,1199736362468 from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,208 INFO [IPC Server handler 4 on 60000]
master.ServerManager(327): Received MSG_REPORT_PROCESS_OPEN: TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,209 INFO [IPC Server handler 4 on 60000]
master.ServerManager(327): Received MSG_REPORT_OPEN: TestUpgrade,,1199736362468
from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,210 INFO [IPC Server handler 4 on 60000]
master.ServerManager(327): Received MSG_REPORT_OPEN: TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,210 DEBUG [IPC Server handler 4 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:07,210 DEBUG [HMaster] master.HMaster(417): Main
processing loop: PendingOpenOperation from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,212 INFO [HMaster]
master.ProcessRegionOpen$1(68): TestUpgrade,,1199736362468 open on
140.211.11.106:42051
[junit] 2008-06-26 04:50:07,212 DEBUG [HMaster]
master.RegionServerOperation(75): numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
[junit] 2008-06-26 04:50:07,212 INFO [HMaster]
master.ProcessRegionOpen$1(79): updating row TestUpgrade,,1199736362468 in
region .META.,,1 with startcode 1214455795906 and server 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,215 DEBUG [HMaster] master.HMaster(417): Main
processing loop: PendingOpenOperation from 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,215 INFO [HMaster]
master.ProcessRegionOpen$1(68): TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 open on 140.211.11.106:42051
[junit] 2008-06-26 04:50:07,216 DEBUG [HMaster]
master.RegionServerOperation(75): numberOfMetaRegions: 1,
onlineMetaRegions.size(): 1
[junit] 2008-06-26 04:50:07,216 INFO [HMaster]
master.ProcessRegionOpen$1(79): updating row TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 in region .META.,,1 with
startcode 1214455795906 and server 140.211.11.106:42051
[junit] 2008-06-26 04:50:08,217 DEBUG [IPC Server handler 3 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:08,410 DEBUG [RegionServer:0.compactor]
regionserver.HStore(1129): moving
/user/hudson/hbase/TestUpgrade/compaction.dir/341377241/column_b/mapfiles/4588480365059607679
to
/user/hudson/hbase/TestUpgrade/341377241/column_b/mapfiles/8779852255731409560
[junit] 2008-06-26 04:50:08,589 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HStore(756): Added
/user/hudson/hbase/-ROOT-/70236052/info/mapfiles/8875332274811027126 with 2
entries, sequence id 35178, data size 93.0, file size 231.0
[junit] 2008-06-26 04:50:08,590 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(1058): Finished memcache flush for region -ROOT-,,0 in
1444ms, sequence id=35178, 0.0
[junit] 2008-06-26 04:50:08,591 DEBUG [RegionServer:0.cacheFlusher]
regionserver.CompactSplitThread(120): Compaction requested for region: -ROOT-,,0
[junit] 2008-06-26 04:50:08,591 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(987): Started memcache flush for region .META.,,1. Current
region memcache size 776.0
[junit] 2008-06-26 04:50:08,608 DEBUG [RegionServer:0.compactor]
regionserver.HStore(907): Completed compaction of 341377241/column_b store size
is 815.3k
[junit] 2008-06-26 04:50:08,610 DEBUG [RegionServer:0.compactor]
regionserver.HStore(888): started compaction of 1 files
[341377241/column_a/4514508232435632076-1396626490/7048898707195909278/bottom]
into
/user/hudson/hbase/TestUpgrade/compaction.dir/341377241/column_a/mapfiles/3144766613526269145
[junit] 2008-06-26 04:50:09,158 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HStore(756): Added
/user/hudson/hbase/.META./1028785192/historian/mapfiles/2529174255070031662
with 4 entries, sequence id 35185, data size 458.0, file size 624.0
[junit] 2008-06-26 04:50:09,277 DEBUG [IPC Server handler 0 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:09,957 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HStore(756): Added
/user/hudson/hbase/.META./1028785192/info/mapfiles/6443079880654428805 with 4
entries, sequence id 35185, data size 318.0, file size 484.0
[junit] 2008-06-26 04:50:09,958 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(1058): Finished memcache flush for region .META.,,1 in
1367ms, sequence id=35185, 0.0
[junit] 2008-06-26 04:50:09,958 DEBUG [RegionServer:0.cacheFlusher]
regionserver.CompactSplitThread(120): Compaction requested for region: .META.,,1
[junit] 2008-06-26 04:50:10,095 DEBUG [RegionServer:0.compactor]
regionserver.HStore(1129): moving
/user/hudson/hbase/TestUpgrade/compaction.dir/341377241/column_a/mapfiles/3144766613526269145
to
/user/hudson/hbase/TestUpgrade/341377241/column_a/mapfiles/3599707691212843065
[junit] 2008-06-26 04:50:10,225 DEBUG [RegionServer:0.compactor]
regionserver.HStore(907): Completed compaction of 341377241/column_a store size
is 815.3k
[junit] 2008-06-26 04:50:10,255 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region
TestUpgrade,,1199736362468 in 3sec
[junit] 2008-06-26 04:50:10,257 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:10,260 DEBUG [RegionServer:0.compactor]
regionserver.HStore(888): started compaction of 1 files
[1971203659/column_b/209479190043547321-1396626490/4973609345075242702/top]
into
/user/hudson/hbase/TestUpgrade/compaction.dir/1971203659/column_b/mapfiles/5373359994721484890
[junit] 2008-06-26 04:50:10,287 DEBUG [IPC Server handler 1 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:10,630 DEBUG [RegionServer:0.compactor]
regionserver.HStore(1129): moving
/user/hudson/hbase/TestUpgrade/compaction.dir/1971203659/column_b/mapfiles/5373359994721484890
to
/user/hudson/hbase/TestUpgrade/1971203659/column_b/mapfiles/1319975069603146331
[junit] 2008-06-26 04:50:10,741 DEBUG [RegionServer:0.compactor]
regionserver.HStore(907): Completed compaction of 1971203659/column_b store
size is 831.1k
[junit] 2008-06-26 04:50:10,743 DEBUG [RegionServer:0.compactor]
regionserver.HStore(888): started compaction of 1 files
[1971203659/column_a/3526482879590887371-1396626490/7048898707195909278/top]
into
/user/hudson/hbase/TestUpgrade/compaction.dir/1971203659/column_a/mapfiles/2193315150673109262
[junit] 2008-06-26 04:50:11,053 DEBUG [main]
client.HConnectionManager$TableServers(726): Found ROOT REGION => {NAME =>
'-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052, TABLE => {NAME
=> '-ROOT-', FAMILIES => [{NAME => 'info', VERSIONS => 1, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}]}
[junit] 2008-06-26 04:50:11,077 DEBUG [RegionServer:0.compactor]
regionserver.HStore(1129): moving
/user/hudson/hbase/TestUpgrade/compaction.dir/1971203659/column_a/mapfiles/2193315150673109262
to
/user/hudson/hbase/TestUpgrade/1971203659/column_a/mapfiles/2450352856258874149
[junit] 2008-06-26 04:50:11,112 INFO [main] util.TestMigrate(160): OPENING
SCANNER
[junit] 2008-06-26 04:50:11,113 DEBUG [main]
client.HTable$ClientScanner(1285): Creating scanner over TestUpgrade starting
at key ''
[junit] 2008-06-26 04:50:11,114 DEBUG [main]
client.HTable$ClientScanner(1346): Advancing internal scanner to startKey at
[junit] 2008-06-26 04:50:11,114 DEBUG [main]
client.HConnectionManager$TableServers(528): Cache hit in table locations for
row <> and tableName TestUpgrade: location server 140.211.11.106:42051,
location region name TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:11,297 DEBUG [RegionServer:0.compactor]
regionserver.HStore(907): Completed compaction of 1971203659/column_a store
size is 831.1k
[junit] 2008-06-26 04:50:11,297 DEBUG [IPC Server handler 0 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:11,318 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 in 1sec
[junit] 2008-06-26 04:50:11,320 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region -ROOT-,,0
[junit] 2008-06-26 04:50:11,348 DEBUG [RegionServer:0.compactor]
regionserver.HStore(888): started compaction of 3 files
[70236052/info/8875332274811027126, 70236052/info/6964344164495841824,
70236052/info/7214912435301412040] into
/user/hudson/hbase/-ROOT-/compaction.dir/70236052/info/mapfiles/2744811669103583584
[junit] 2008-06-26 04:50:11,664 DEBUG [RegionServer:0.compactor]
regionserver.HStore(1129): moving
/user/hudson/hbase/-ROOT-/compaction.dir/70236052/info/mapfiles/2744811669103583584
to /user/hudson/hbase/-ROOT-/70236052/info/mapfiles/8881324873849651680
[junit] 2008-06-26 04:50:11,783 INFO [main] util.TestMigrate(169):
Iterated over 1000 rows.
[junit] 2008-06-26 04:50:11,951 DEBUG [RegionServer:0.compactor]
regionserver.HStore(907): Completed compaction of 70236052/info store size is
593.0
[junit] 2008-06-26 04:50:11,969 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region -ROOT-,,0 in 0sec
[junit] 2008-06-26 04:50:11,969 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region .META.,,1
[junit] 2008-06-26 04:50:11,971 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region .META.,,1 in 0sec
[junit] 2008-06-26 04:50:12,307 DEBUG [IPC Server handler 1 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:12,392 INFO [main] util.TestMigrate(169):
Iterated over 2000 rows.
[junit] 2008-06-26 04:50:12,974 INFO [main] util.TestMigrate(169):
Iterated over 3000 rows.
[junit] 2008-06-26 04:50:13,317 DEBUG [IPC Server handler 2 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:13,732 INFO [main] util.TestMigrate(169):
Iterated over 4000 rows.
[junit] 2008-06-26 04:50:14,263 INFO [main] util.TestMigrate(169):
Iterated over 5000 rows.
[junit] 2008-06-26 04:50:14,367 DEBUG [IPC Server handler 4 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:14,531 INFO [main] util.TestMigrate(169):
Iterated over 6000 rows.
[junit] 2008-06-26 04:50:14,729 INFO [main] util.TestMigrate(169):
Iterated over 7000 rows.
[junit] 2008-06-26 04:50:14,916 INFO [main] util.TestMigrate(169):
Iterated over 8000 rows.
[junit] 2008-06-26 04:50:15,251 DEBUG [main]
client.HTable$ClientScanner(1330): Advancing forward from region REGION =>
{NAME => 'TestUpgrade,,1199736362468', STARTKEY => '', ENDKEY => '[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|', ENCODED => 341377241, TABLE => {NAME =>
'TestUpgrade', FAMILIES => [{NAME => 'column_a', VERSIONS => 3, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}, {NAME => 'column_b', VERSIONS => 3, COMPRESSION
=> 'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL
=> FOREVER, BLOOMFILTER => NONE}]}
[junit] 2008-06-26 04:50:15,252 DEBUG [main]
client.HTable$ClientScanner(1346): Advancing internal scanner to startKey at
[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|
[junit] 2008-06-26 04:50:15,387 DEBUG [IPC Server handler 3 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:15,427 INFO [main] util.TestMigrate(169):
Iterated over 9000 rows.
[junit] 2008-06-26 04:50:15,642 INFO [main] util.TestMigrate(169):
Iterated over 10000 rows.
[junit] 2008-06-26 04:50:15,791 INFO [main] util.TestMigrate(169):
Iterated over 11000 rows.
[junit] 2008-06-26 04:50:15,936 INFO [RegionManager.metaScanner]
master.BaseScanner(149): RegionManager.metaScanner scanning meta region
{regionname: .META.,,1, startKey: <>, server: 140.211.11.106:42051}
[junit] 2008-06-26 04:50:16,100 DEBUG [RegionManager.metaScanner]
master.BaseScanner(175): RegionManager.metaScannerREGION => {NAME =>
'TestUpgrade,,1199736332093', STARTKEY => '', ENDKEY => '', ENCODED =>
1396626490, OFFLINE => true, SPLIT => true, TABLE => {NAME => 'TestUpgrade',
FAMILIES => [{NAME => 'column_a', VERSIONS => 3, COMPRESSION => 'NONE',
IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL => FOREVER,
BLOOMFILTER => NONE}, {NAME => 'column_b', VERSIONS => 3, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}]}}, SERVER => '10.69.80.3:1845', STARTCODE =>
1199736332062
[junit] 2008-06-26 04:50:16,101 DEBUG [RegionManager.metaScanner]
master.BaseScanner(175): RegionManager.metaScannerREGION => {NAME =>
'TestUpgrade,,1199736362468', STARTKEY => '', ENDKEY => '[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|', ENCODED => 341377241, TABLE => {NAME =>
'TestUpgrade', FAMILIES => [{NAME => 'column_a', VERSIONS => 3, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}, {NAME => 'column_b', VERSIONS => 3, COMPRESSION
=> 'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL
=> FOREVER, BLOOMFILTER => NONE}]}}, SERVER => '140.211.11.106:42051',
STARTCODE => 1214455795906
[junit] 2008-06-26 04:50:16,101 DEBUG [RegionManager.metaScanner]
master.BaseScanner(175): RegionManager.metaScannerREGION => {NAME =>
'TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468', STARTKEY
=> '[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|', ENDKEY => '', ENCODED =>
1971203659, TABLE => {NAME => 'TestUpgrade', FAMILIES => [{NAME => 'column_a',
VERSIONS => 3, COMPRESSION => 'NONE', IN_MEMORY => false, BLOCKCACHE => false,
LENGTH => 2147483647, TTL => FOREVER, BLOOMFILTER => NONE}, {NAME =>
'column_b', VERSIONS => 3, COMPRESSION => 'NONE', IN_MEMORY => false,
BLOCKCACHE => false, LENGTH => 2147483647, TTL => FOREVER, BLOOMFILTER =>
NONE}]}}, SERVER => '140.211.11.106:42051', STARTCODE => 1214455795906
[junit] 2008-06-26 04:50:16,104 DEBUG [RegionManager.metaScanner]
master.BaseScanner(337): TestUpgrade,,1199736362468 no longer has references to
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:16,112 DEBUG [RegionManager.metaScanner]
master.BaseScanner(337): TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 no longer has references to
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:16,130 INFO [RegionManager.metaScanner]
master.BaseScanner(271): Deleting region TestUpgrade,,1199736332093 because
daughter splits no longer hold references
[junit] 2008-06-26 04:50:16,131 DEBUG [RegionManager.metaScanner]
regionserver.HRegion(2088): DELETING region
hdfs://localhost:41945/user/hudson/hbase/TestUpgrade/1396626490
[junit] 2008-06-26 04:50:16,141 INFO [main] util.TestMigrate(169):
Iterated over 12000 rows.
[junit] 2008-06-26 04:50:16,190 INFO [RegionManager.metaScanner]
master.BaseScanner(229): RegionManager.metaScanner scan of meta region
{regionname: .META.,,1, startKey: <>, server: 140.211.11.106:42051} complete
[junit] 2008-06-26 04:50:16,190 INFO [RegionManager.metaScanner]
master.MetaScanner(138): all meta regions scanned
[junit] 2008-06-26 04:50:16,397 DEBUG [IPC Server handler 0 on 60000]
master.ServerManager(570): Total Load: 4, Num Servers: 1, Avg Load: 4.0
[junit] 2008-06-26 04:50:16,503 INFO [main] util.TestMigrate(169):
Iterated over 13000 rows.
[junit] 2008-06-26 04:50:16,624 INFO [main] util.TestMigrate(169):
Iterated over 14000 rows.
[junit] 2008-06-26 04:50:16,795 INFO [main] util.TestMigrate(169):
Iterated over 15000 rows.
[junit] 2008-06-26 04:50:16,903 INFO [main] util.TestMigrate(169):
Iterated over 16000 rows.
[junit] 2008-06-26 04:50:17,079 INFO [main] util.TestMigrate(169):
Iterated over 17000 rows.
[junit] 2008-06-26 04:50:17,106 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(987): Started memcache flush for region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468. Current region memcache size 0.0
[junit] 2008-06-26 04:50:17,111 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(1058): Finished memcache flush for region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 in 5ms,
sequence id=35195, 0.0
[junit] 2008-06-26 04:50:17,111 DEBUG [RegionServer:0.cacheFlusher]
regionserver.CompactSplitThread(120): Compaction requested for region:
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,112 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(987): Started memcache flush for region
TestUpgrade,,1199736362468. Current region memcache size 0.0
[junit] 2008-06-26 04:50:17,112 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,117 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(1058): Finished memcache flush for region
TestUpgrade,,1199736362468 in 5ms, sequence id=35196, 0.0
[junit] 2008-06-26 04:50:17,117 DEBUG [RegionServer:0.cacheFlusher]
regionserver.CompactSplitThread(120): Compaction requested for region:
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,117 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 in 0sec
[junit] 2008-06-26 04:50:17,156 INFO [RegionManager.rootScanner]
master.BaseScanner(149): RegionManager.rootScanner scanning meta region
{regionname: -ROOT-,,0, startKey: <>, server: 140.211.11.106:42051}
[junit] 2008-06-26 04:50:17,173 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,194 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region
TestUpgrade,,1199736362468 in 0sec
[junit] 2008-06-26 04:50:17,247 DEBUG [RegionManager.rootScanner]
master.BaseScanner(175): RegionManager.rootScannerREGION => {NAME =>
'.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE =>
{NAME => '.META.', FAMILIES => [{NAME => 'info', VERSIONS => 1, COMPRESSION =>
'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH => 2147483647, TTL =>
FOREVER, BLOOMFILTER => NONE}, {NAME => 'historian', VERSIONS => 2147483647,
COMPRESSION => 'NONE', IN_MEMORY => false, BLOCKCACHE => false, LENGTH =>
2147483647, TTL => FOREVER, BLOOMFILTER => NONE}]}}, SERVER =>
'140.211.11.106:42051', STARTCODE => 1214455795906
[junit] 2008-06-26 04:50:17,247 INFO [RegionManager.rootScanner]
master.BaseScanner(229): RegionManager.rootScanner scan of meta region
{regionname: -ROOT-,,0, startKey: <>, server: 140.211.11.106:42051} complete
[junit] 2008-06-26 04:50:17,339 DEBUG [main]
client.HTable$ClientScanner(1330): Advancing forward from region REGION =>
{NAME => 'TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468',
STARTKEY => '[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|', ENDKEY => '', ENCODED =>
1971203659, TABLE => {NAME => 'TestUpgrade', FAMILIES => [{NAME => 'column_a',
VERSIONS => 3, COMPRESSION => 'NONE', IN_MEMORY => false, BLOCKCACHE => false,
LENGTH => 2147483647, TTL => FOREVER, BLOOMFILTER => NONE}, {NAME =>
'column_b', VERSIONS => 3, COMPRESSION => 'NONE', IN_MEMORY => false,
BLOCKCACHE => false, LENGTH => 2147483647, TTL => FOREVER, BLOOMFILTER =>
NONE}]}
[junit] 2008-06-26 04:50:17,342 DEBUG [main] hbase.LocalHBaseCluster(253):
Shutting down HBase Cluster
[junit] 2008-06-26 04:50:17,342 INFO [main] master.HMaster(571): Cluster
shutdown requested. Starting to quiesce servers
[junit] 2008-06-26 04:50:17,438 INFO [RegionServer:0]
regionserver.HRegionServer(320): MSG_REGIONSERVER_QUIESCE
[junit] 2008-06-26 04:50:17,439 INFO [RegionServer:0.worker]
regionserver.HRegionServer$Worker(795): MSG_REGIONSERVER_QUIESCE
[junit] 2008-06-26 04:50:17,440 DEBUG [RegionServer:0.worker]
regionserver.HRegionServer(973): closing region TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,440 DEBUG [RegionServer:0.worker]
regionserver.HRegion(560): Compactions and cache flushes disabled for region
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,441 DEBUG [RegionServer:0.worker]
regionserver.HRegion(575): Updates and scanners disabled for region
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,441 DEBUG [RegionServer:0.worker]
regionserver.HRegion(590): No more active scanners for region
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,441 DEBUG [RegionServer:0.worker]
regionserver.HRegion(596): No more row locks outstanding on region
TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,441 DEBUG [RegionServer:0.worker]
regionserver.HRegion(987): Started memcache flush for region
TestUpgrade,,1199736362468. Current region memcache size 0.0
[junit] 2008-06-26 04:50:17,441 DEBUG [RegionServer:0.worker]
regionserver.HRegion(1058): Finished memcache flush for region
TestUpgrade,,1199736362468 in 0ms, sequence id=35199, 0.0
[junit] 2008-06-26 04:50:17,444 DEBUG [RegionServer:0.worker]
regionserver.HStore(651): closed 341377241/column_b
[junit] 2008-06-26 04:50:17,445 DEBUG [RegionServer:0.worker]
regionserver.HStore(651): closed 341377241/column_a
[junit] 2008-06-26 04:50:17,446 INFO [RegionServer:0.worker]
regionserver.HRegion(609): closed TestUpgrade,,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegionServer(973): closing region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegion(560): Compactions and cache flushes disabled for region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegion(575): Updates and scanners disabled for region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegion(590): No more active scanners for region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegion(596): No more row locks outstanding on region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:17,446 DEBUG [RegionServer:0.worker]
regionserver.HRegion(987): Started memcache flush for region TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468. Current region memcache size 0.0
[junit] 2008-06-26 04:50:17,447 DEBUG [RegionServer:0.worker]
regionserver.HRegion(1058): Finished memcache flush for region
TestUpgrade,[EMAIL PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468 in 1ms,
sequence id=35200, 0.0
[junit] 2008-06-26 04:50:17,448 DEBUG [RegionServer:0.worker]
regionserver.HStore(651): closed 1971203659/column_b
[junit] 2008-06-26 04:50:17,452 DEBUG [RegionServer:0.worker]
regionserver.HStore(651): closed 1971203659/column_a
[junit] 2008-06-26 04:50:17,452 INFO [RegionServer:0.worker]
regionserver.HRegion(609): closed TestUpgrade,[EMAIL
PROTECTED]&*()-_+=:;',.<>/?[]{}|,1199736362468
[junit] 2008-06-26 04:50:18,126 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(987): Started memcache flush for region -ROOT-,,0. Current
region memcache size 0.0
[junit] 2008-06-26 04:50:18,128 DEBUG [RegionServer:0.cacheFlusher]
regionserver.HRegion(1058): Finished memcache flush for region -ROOT-,,0 in
2ms, sequence id=35201, 0.0
[junit] 2008-06-26 04:50:18,128 DEBUG [RegionServer:0.cacheFlusher]
regionserver.CompactSplitThread(120): Compaction requested for region: -ROOT-,,0
[junit] 2008-06-26 04:50:18,128 INFO [RegionServer:0.compactor]
regionserver.HRegion(868): starting compaction on region -ROOT-,,0
[junit] 2008-06-26 04:50:18,129 INFO [RegionServer:0.compactor]
regionserver.HRegion(880): compaction completed on region -ROOT-,,0 in 0sec
[junit] 2008-06-26 04:50:18,446 INFO [IPC Server handler 2 on 60000]
master.ServerManager(165): Region server 140.211.11.106:42051 quiesced
[junit] 2008-06-26 04:50:18,447 INFO [IPC Server handler 2 on 60000]
master.ServerManager(174): All user tables quiesced. Proceeding with shutdown
[junit] 2008-06-26 04:50:18,447 DEBUG [IPC Server handler 2 on 60000]
master.RegionManager(453): telling root scanner to stop
[junit] 2008-06-26 04:50:18,448 DEBUG [IPC Server handler 2 on 60000]
master.RegionManager(457): telling meta scanner to stop
[junit] 2008-06-26 04:50:18,448 INFO [RegionManager.rootScanner]
hbase.Chore(74): RegionManager.rootScanner exiting
[junit] 2008-06-26 04:50:18,450 DEBUG [IPC Server handler 2 on 60000]
master.RegionManager(461): meta and root scanners notified
[junit] 2008-06-26 04:50:18,450 INFO [RegionManager.metaScanner]
hbase.Chore(74): RegionManager.metaScanner exiting
[junit] 2008-06-26 04:50:18,450 INFO [RegionServer:0]
regionserver.HRegionServer(320): MSG_REGIONSERVER_STOP
[junit] 2008-06-26 04:50:18,451 DEBUG [RegionServer:0]
hbase.RegionHistorian(316): Offlined
[junit] 2008-06-26 04:50:18,456 INFO [RegionServer:0.worker]
regionserver.HRegionServer$Worker(849): worker thread exiting
[junit] 2008-06-26 04:50:18,457 DEBUG [RegionServer:0]
regionserver.HRegionServer(939): closing region .META.,,1
[junit] 2008-06-26 04:50:18,457 INFO [RegionServer:0.compactor]
regionserver.CompactSplitThread(113): RegionServer:0.compactor exiting
[junit] 2008-06-26 04:50:18,457 DEBUG [RegionServer:0]
regionserver.HRegion(560): Compactions and cache flushes disabled for region
.META.,,1
[junit] 2008-06-26 04:50:18,458 DEBUG [RegionServer:0]
regionserver.HRegion(575): Updates and scanners disabled for region .META.,,1
[junit] 2008-06-26 04:50:18,458 DEBUG [RegionServer:0]
regionserver.HRegion(590): No more active scanners for region .META.,,1
[junit] 2008-06-26 04:50:18,458 DEBUG [RegionServer:0]
regionserver.HRegion(596): No more row locks outstanding on region .META.,,1
[junit] 2008-06-26 04:50:18,458 DEBUG [RegionServer:0]
regionserver.HRegion(987): Started memcache flush for region .META.,,1. Current
region memcache size 860.0
[junit] 2008-06-26 04:50:18,460 INFO [RegionServer:0.cacheFlusher]
regionserver.Flusher(111): RegionServer:0.cacheFlusher exiting
[junit] 2008-06-26 04:50:18,461 INFO [RegionServer:0.logRoller]
regionserver.LogRoller(93): LogRoller exiting.
[junit] 2008-06-26 04:50:18,808 DEBUG [RegionServer:0]
regionserver.HStore(756): Added
/user/hudson/hbase/.META./1028785192/historian/mapfiles/6597473069040346308
with 4 entries, sequence id 35202, data size 420.0, file size 586.0
[junit] 2008-06-26 04:50:18,982 DEBUG [RegionServer:0]
regionserver.HStore(756): Added
/user/hudson/hbase/.META./1028785192/info/mapfiles/6932496064497146730 with 5
entries, sequence id 35202, data size 318.0, file size 498.0
[junit] 2008-06-26 04:50:18,982 DEBUG [RegionServer:0]
regionserver.HRegion(1058): Finished memcache flush for region .META.,,1 in
524ms, sequence id=35202, 122.0
[junit] 2008-06-26 04:50:18,983 DEBUG [RegionServer:0]
regionserver.HStore(651): closed 1028785192/historian
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HStore(651): closed 1028785192/info
[junit] 2008-06-26 04:50:18,985 INFO [RegionServer:0]
regionserver.HRegion(609): closed .META.,,1
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegionServer(939): closing region -ROOT-,,0
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegion(560): Compactions and cache flushes disabled for region
-ROOT-,,0
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegion(575): Updates and scanners disabled for region -ROOT-,,0
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegion(590): No more active scanners for region -ROOT-,,0
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegion(596): No more row locks outstanding on region -ROOT-,,0
[junit] 2008-06-26 04:50:18,985 DEBUG [RegionServer:0]
regionserver.HRegion(987): Started memcache flush for region -ROOT-,,0. Current
region memcache size 0.0
[junit] 2008-06-26 04:50:18,986 DEBUG [RegionServer:0]
regionserver.HRegion(1058): Finished memcache flush for region -ROOT-,,0 in
1ms, sequence id=35203, 0.0
[junit] 2008-06-26 04:50:18,987 DEBUG [RegionServer:0]
regionserver.HStore(651): closed 70236052/info
[junit] 2008-06-26 04:50:18,987 INFO [RegionServer:0]
regionserver.HRegion(609): closed -ROOT-,,0
[junit] 2008-06-26 04:50:18,987 DEBUG [RegionServer:0]
regionserver.HLog(337): closing log writer in
hdfs://localhost:41945/user/hudson/hbase/log_140.211.11.106_1214455795906_42051
[junit] 2008-06-26 04:50:19,256 INFO [RegionServer:0]
regionserver.HRegionServer(462): telling master that region server is shutting
down at: 140.211.11.106:42051
[junit] 2008-06-26 04:50:19,257 INFO [IPC Server handler 4 on 60000]
master.ServerManager(524): Cancelling lease for 140.211.11.106:42051
[junit] 2008-06-26 04:50:19,258 INFO [IPC Server handler 4 on 60000]
master.ServerManager(242): Region server 140.211.11.106:42051:
MSG_REPORT_EXITING -- lease cancelled
[junit] 2008-06-26 04:50:19,258 INFO [RegionServer:0]
regionserver.HRegionServer(469): stopping server at: 140.211.11.106:42051
[junit] 2008-06-26 04:50:19,258 INFO [RegionServer:0]
regionserver.HRegionServer(473): RegionServer:0 exiting
[junit] 2008-06-26 04:50:19,326 INFO [RegionServer:0.leaseChecker]
hbase.Leases(115): RegionServer:0.leaseChecker closing leases
[junit] 2008-06-26 04:50:19,326 INFO [RegionServer:0.leaseChecker]
hbase.Leases(122): RegionServer:0.leaseChecker closed leases
[junit] 2008-06-26 04:50:19,386 DEBUG [HMaster] master.RegionManager(453):
telling root scanner to stop
[junit] 2008-06-26 04:50:19,386 DEBUG [HMaster] master.RegionManager(457):
telling meta scanner to stop
[junit] 2008-06-26 04:50:19,386 DEBUG [HMaster] master.RegionManager(461):
meta and root scanners notified
[junit] 2008-06-26 04:50:19,386 DEBUG [HMaster] hbase.RegionHistorian(316):
Offlined
[junit] 2008-06-26 04:50:19,388 INFO [HMaster] hbase.Leases(115): HMaster
closing leases
[junit] 2008-06-26 04:50:19,389 INFO [HMaster] hbase.Leases(122): HMaster
closed leases
[junit] 2008-06-26 04:50:19,389 INFO [HMaster] master.HMaster(385):
HMaster main thread exiting
[junit] 2008-06-26 04:50:20,146 INFO [ServerManager.leaseChecker]
hbase.Leases(115): ServerManager.leaseChecker closing leases
[junit] 2008-06-26 04:50:20,146 INFO [ServerManager.leaseChecker]
hbase.Leases(122): ServerManager.leaseChecker closed leases
[junit] 2008-06-26 04:50:20,266 INFO [main] hbase.LocalHBaseCluster(282):
Shutdown HMaster 1 region server(s)
[junit] 2008-06-26 04:50:20,266 INFO [main] hbase.HBaseTestCase(614):
Shutting down FileSystem
[junit] 2008-06-26 04:50:21,236 INFO [main] hbase.HBaseTestCase(621):
Shutting down Mini DFS
[junit] Shutting down the Mini HDFS Cluster
[junit] Shutting down DataNode 1
[junit] 2008-06-26 04:50:21,240 INFO [main] http.SocketListener(212):
Stopped SocketListener on 127.0.0.1:41952
[junit] 2008-06-26 04:50:21,343 INFO [main] util.Container(156): Stopped
HttpContext[/static,/static]
[junit] 2008-06-26 04:50:21,344 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:21,483 INFO [main] util.Container(156): Stopped
WebApplicationContext[/,/]
[junit] 2008-06-26 04:50:21,484 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] Shutting down DataNode 0
[junit] 2008-06-26 04:50:22,487 INFO [Acceptor
ServerSocket[addr=localhost/127.0.0.1,port=0,localport=41950]]
util.ThreadedServer$Acceptor(656): Stopping Acceptor
ServerSocket[addr=localhost/127.0.0.1,port=0,localport=41950]
[junit] 2008-06-26 04:50:22,487 INFO [main] http.SocketListener(212):
Stopped SocketListener on 127.0.0.1:41950
[junit] 2008-06-26 04:50:22,721 INFO [main] util.Container(156): Stopped
HttpContext[/static,/static]
[junit] 2008-06-26 04:50:22,721 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:22,805 INFO [main] util.Container(156): Stopped
WebApplicationContext[/,/]
[junit] 2008-06-26 04:50:22,805 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:24,069 INFO [Acceptor
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=41946]]
util.ThreadedServer$Acceptor(656): Stopping Acceptor
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=41946]
[junit] 2008-06-26 04:50:24,069 INFO [main] http.SocketListener(212):
Stopped SocketListener on 0.0.0.0:41946
[junit] 2008-06-26 04:50:24,206 INFO [main] util.Container(156): Stopped
HttpContext[/static,/static]
[junit] 2008-06-26 04:50:24,206 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:24,406 INFO [main] util.Container(156): Stopped
WebApplicationContext[/,/]
[junit] 2008-06-26 04:50:24,406 INFO [main] util.Container(156): Stopped
[EMAIL PROTECTED]
[junit] 2008-06-26 04:50:24,407 WARN [EMAIL PROTECTED]
dfs.FSNamesystem$ReplicationMonitor(2263): ReplicationMonitor thread received
InterruptedException.java.lang.InterruptedException: sleep interrupted
[junit] 2008-06-26 04:50:24,409 ERROR [EMAIL PROTECTED]
dfs.FSNamesystem$ResolutionMonitor(1950): java.lang.InterruptedException
[junit] at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
[junit] at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1934)
[junit] at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
[junit] at
org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor.run(FSNamesystem.java:1927)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit]
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 61.017 sec
[junit] 2008-06-26 04:50:24,451 INFO [Thread-278]
regionserver.HRegionServer$ShutdownThread(180): Starting shutdown thread.
[junit] 2008-06-26 04:50:24,451 INFO [Thread-278]
regionserver.HRegionServer$ShutdownThread(185): Shutdown thread complete
[junit] Running org.apache.hadoop.hbase.util.TestRootPath
[junit] 2008-06-26 04:50:25,145 INFO [main] util.TestRootPath(60): Got
expected exception when checking invalid path:
[junit] java.io.IOException: Root directory does not contain a scheme
[junit] at
org.apache.hadoop.hbase.util.FSUtils.validateRootPath(FSUtils.java:161)
[junit] at
org.apache.hadoop.hbase.util.TestRootPath.testRootPath(TestRootPath.java:56)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:154)
[junit] at junit.framework.TestCase.runBare(TestCase.java:127)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
[junit] at junit.framework.TestResult.run(TestResult.java:109)
[junit] at junit.framework.TestCase.run(TestCase.java:118)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:208)
[junit] at junit.framework.TestSuite.run(TestSuite.java:203)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:421)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:912)
[junit] at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:766)
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.062 sec
[junit] Running org.onelab.test.TestFilter
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.07 sec
BUILD FAILED
/zonestorage/hudson/home/hudson/hudson/jobs/HBase-Patch/workspace/trunk/build.xml:389:
Tests failed!
Total time: 33 minutes 22 seconds
Recording test results
