Clint, Don't dig too deep, META edits will always be missed until we can flush every edits.
J-D On Thu, Jun 4, 2009 at 12:31 PM, Clint Morgan <[email protected]> wrote: > I the meta edits did not get applied correctly. The only changes were > region->server assignments, and after replay it still had the old > assignments. > > I had made one explicit change to another table before the hard restart, and > that change did not get recovered either. > > I'll dig deeper today... > -clint > > On Wed, Jun 3, 2009 at 7:22 PM, stack <[email protected]> wrote: > >> On Wed, Jun 3, 2009 at 5:53 PM, Clint Morgan <[email protected]> wrote: >> >> > Which seems weird because the hlog path points to a table's directory >> which >> > is a different table than the given region. Does this seem right? >> > >> > >> No. It seems badly wrong. I took a quick look at HLog#splitLog and how >> its >> making up name of the file to write to. I don't see anything obvious. Its >> using the HLogKey it got out of the commit log. This in turn looks like it >> has right table name going in -- it gets it from region whose commit is >> being logged. >> >> I see in below that it made queues for -ROOT- and .META. Did these edits >> get split into the right location? >> >> St.Ack >> >> >> >> > A larger portion of the log is below. >> > >> > [03/06/09 16:35:34] 530059 [ HMaster] INFO >> > hadoop.hbase.regionserver.HLog - Splitting 1 log(s) in >> > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802 >> > [03/06/09 16:35:34] 530060 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Splitting hlog 1 of 1: >> > >> > >> file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802/hlog.dat.1244071612170, >> > length=45056 >> > [03/06/09 16:35:35] 530106 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for -ROOT-,,0 >> > [03/06/09 16:35:35] 530110 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for .META.,,1 >> > [03/06/09 16:35:35] 530120 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for >> > TransactionStatus,,1244059490418 >> > [03/06/09 16:35:35] 530121 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for >> > net_troove_coin_project_Project,,1244059485284 >> > [03/06/09 16:35:35] 530122 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for >> > net_troove_security_ApplicationUser,,1244059486102 >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG >> > hadoop.hbase.regionserver.HLog - Adding queue for >> > >> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 >> > [03/06/09 16:35:35] 530133 [pool-1-thread-1] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 2 to process >> > [03/06/09 16:35:35] 530137 [pool-1-thread-3] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 7 to process >> > [03/06/09 16:35:35] 530137 [pool-1-thread-2] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 220 to process >> > [03/06/09 16:35:35] 530144 [pool-1-thread-4] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 1 to process >> > [03/06/09 16:35:35] 530149 [pool-1-thread-5] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 18 to process >> > [03/06/09 16:35:35] 530157 [pool-1-thread-6] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process >> > [03/06/09 16:35:35] 530209 [pool-1-thread-7] DEBUG >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process >> > [03/06/09 16:35:35] 530325 [pool-1-thread-3] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/228384672/oldlogfile.log >> > and region TransactionStatus,,1244059490418 >> > [03/06/09 16:35:35] 530326 [pool-1-thread-3] DEBUG >> > hadoop.hbase.regionserver.HLog - Applied 7 total edits to >> > TransactionStatus,,1244059490418 in 187ms >> > [03/06/09 16:35:35] 530328 [pool-1-thread-1] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/70236052/oldlogfile.log >> > and region -ROOT-,,0 >> > [03/06/09 16:35:35] 530328 [pool-1-thread-5] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/767391690/oldlogfile.log >> > and region net_troove_coin_timeexpense_BaseLog,,1244059487257 >> > [03/06/09 16:35:35] 530329 [pool-1-thread-5] DEBUG >> > hadoop.hbase.regionserver.HLog - Applied 18 total edits to >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 in 173ms >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/364454454/oldlogfile.log >> > and region >> > >> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG >> > hadoop.hbase.regionserver.HLog - Applied 10 total edits to >> > >> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520 >> > in 154ms >> > [03/06/09 16:35:35] 530330 [pool-1-thread-2] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/1028785192/oldlogfile.log >> > and region .META.,,1 >> > [03/06/09 16:35:35] 530328 [pool-1-thread-4] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/124273138/oldlogfile.log >> > and region net_troove_coin_project_Project,,1244059485284 >> > [03/06/09 16:35:35] 530331 [pool-1-thread-4] DEBUG >> > hadoop.hbase.regionserver.HLog - Applied 1 total edits to >> > net_troove_coin_project_Project,,1244059485284 in 187ms >> > [03/06/09 16:35:35] 530330 [pool-1-thread-7] DEBUG >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer for path >> > >> > >> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/395668855/oldlogfile.log >> > and region net_troove_security_ApplicationUser,,1244059486102 >> > >> >
