Ok, I tracked down the issue:
HLog:803
queue.push(new HLogEntry(val, key));
The same val and key references are used in the iterator, so they keep
changing their contents as we .next() through the loop.
Thus, when we are done, all the HLogEntries in logEntries will have the same
HLogEdits and HLogKeys! I don't think this would have ever worked unless
there is only one region in the log being split.
I guess the solution is to deep copy those val and key contents...
-clint
On Thu, Jun 4, 2009 at 10:38 AM, stack <[email protected]> wrote:
> If you can figure whats up, that'd be great Clint.
> St.Ack
>
> On Thu, Jun 4, 2009 at 10:15 AM, Clint Morgan <[email protected]
> >wrote:
>
> > I did not properly close the log, just did a kill -9 on the regionserver.
> > But I was using local filesystem so I thought the WAL appends would get
> > through. It appears some did...
> >
> > I'm not worried about the META.
> >
> > The debug log does sound like it sees my changes in the WAL (to the
> BaseLog
> > table, and its index), but it does not apply them appropriately...
> >
> >
> > On Thu, Jun 4, 2009 at 10:04 AM, stack <[email protected]> wrote:
> >
> > > Clint, as J-D suggests, you sure the edit made into a properly closed
> log
> > > first (I'm guessing you did)? If not, you can make it happen by tuning
> > > down
> > > the time between log flushes -- its new config
> > > hbase.regionserver.logroll.period.
> > > St.Ack
> > >
> > > On Thu, Jun 4, 2009 at 9:35 AM, Jean-Daniel Cryans <
> [email protected]
> > > >wrote:
> > >
> > > > Clint,
> > > >
> > > > Don't dig too deep, META edits will always be missed until we can
> > > > flush every edits.
> > > >
> > > > J-D
> > > >
> > > > On Thu, Jun 4, 2009 at 12:31 PM, Clint Morgan <[email protected]>
> > > > wrote:
> > > > > I the meta edits did not get applied correctly. The only changes
> were
> > > > > region->server assignments, and after replay it still had the old
> > > > > assignments.
> > > > >
> > > > > I had made one explicit change to another table before the hard
> > > restart,
> > > > and
> > > > > that change did not get recovered either.
> > > > >
> > > > > I'll dig deeper today...
> > > > > -clint
> > > > >
> > > > > On Wed, Jun 3, 2009 at 7:22 PM, stack <[email protected]> wrote:
> > > > >
> > > > >> On Wed, Jun 3, 2009 at 5:53 PM, Clint Morgan <
> [email protected]>
> > > > wrote:
> > > > >>
> > > > >> > Which seems weird because the hlog path points to a table's
> > > directory
> > > > >> which
> > > > >> > is a different table than the given region. Does this seem
> right?
> > > > >> >
> > > > >> >
> > > > >> No. It seems badly wrong. I took a quick look at HLog#splitLog
> and
> > > how
> > > > >> its
> > > > >> making up name of the file to write to. I don't see anything
> > obvious.
> > > > Its
> > > > >> using the HLogKey it got out of the commit log. This in turn
> looks
> > > like
> > > > it
> > > > >> has right table name going in -- it gets it from region whose
> commit
> > > is
> > > > >> being logged.
> > > > >>
> > > > >> I see in below that it made queues for -ROOT- and .META. Did
> these
> > > > edits
> > > > >> get split into the right location?
> > > > >>
> > > > >> St.Ack
> > > > >>
> > > > >>
> > > > >>
> > > > >> > A larger portion of the log is below.
> > > > >> >
> > > > >> > [03/06/09 16:35:34] 530059 [ HMaster] INFO
> > > > >> > hadoop.hbase.regionserver.HLog - Splitting 1 log(s) in
> > > > >> >
> > file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802
> > > > >> > [03/06/09 16:35:34] 530060 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Splitting hlog 1 of 1:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/log_10.0.1.196_1244071611670_7802/hlog.dat.1244071612170,
> > > > >> > length=45056
> > > > >> > [03/06/09 16:35:35] 530106 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for -ROOT-,,0
> > > > >> > [03/06/09 16:35:35] 530110 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for .META.,,1
> > > > >> > [03/06/09 16:35:35] 530120 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for
> > > > >> > TransactionStatus,,1244059490418
> > > > >> > [03/06/09 16:35:35] 530121 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for
> > > > >> > net_troove_coin_project_Project,,1244059485284
> > > > >> > [03/06/09 16:35:35] 530122 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for
> > > > >> > net_troove_security_ApplicationUser,,1244059486102
> > > > >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for
> > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257
> > > > >> > [03/06/09 16:35:35] 530124 [ HMaster] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Adding queue for
> > > > >> >
> > > > >>
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > >> > [03/06/09 16:35:35] 530133 [pool-1-thread-1] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 2 to process
> > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-3] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 7 to process
> > > > >> > [03/06/09 16:35:35] 530137 [pool-1-thread-2] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 220 to process
> > > > >> > [03/06/09 16:35:35] 530144 [pool-1-thread-4] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 1 to process
> > > > >> > [03/06/09 16:35:35] 530149 [pool-1-thread-5] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 18 to process
> > > > >> > [03/06/09 16:35:35] 530157 [pool-1-thread-6] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process
> > > > >> > [03/06/09 16:35:35] 530209 [pool-1-thread-7] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Thread got 10 to process
> > > > >> > [03/06/09 16:35:35] 530325 [pool-1-thread-3] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/228384672/oldlogfile.log
> > > > >> > and region TransactionStatus,,1244059490418
> > > > >> > [03/06/09 16:35:35] 530326 [pool-1-thread-3] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Applied 7 total edits to
> > > > >> > TransactionStatus,,1244059490418 in 187ms
> > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-1] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/70236052/oldlogfile.log
> > > > >> > and region -ROOT-,,0
> > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-5] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/767391690/oldlogfile.log
> > > > >> > and region net_troove_coin_timeexpense_BaseLog,,1244059487257
> > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-5] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Applied 18 total edits to
> > > > >> > net_troove_coin_timeexpense_BaseLog,,1244059487257 in 173ms
> > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/364454454/oldlogfile.log
> > > > >> > and region
> > > > >> >
> > > > >>
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > >> > [03/06/09 16:35:35] 530329 [pool-1-thread-6] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Applied 10 total edits to
> > > > >> >
> > > > >>
> > > >
> > >
> >
> net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING,,1244059487520
> > > > >> > in 154ms
> > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-2] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/1028785192/oldlogfile.log
> > > > >> > and region .META.,,1
> > > > >> > [03/06/09 16:35:35] 530328 [pool-1-thread-4] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/124273138/oldlogfile.log
> > > > >> > and region net_troove_coin_project_Project,,1244059485284
> > > > >> > [03/06/09 16:35:35] 530331 [pool-1-thread-4] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Applied 1 total edits to
> > > > >> > net_troove_coin_project_Project,,1244059485284 in 187ms
> > > > >> > [03/06/09 16:35:35] 530330 [pool-1-thread-7] DEBUG
> > > > >> > hadoop.hbase.regionserver.HLog - Creating new hlog file writer
> > for
> > > > path
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> file:/opt/troove-data/hadoop/hbase/net_troove_coin_timeexpense_BaseLog-Base_Employee_ASCENDING/395668855/oldlogfile.log
> > > > >> > and region net_troove_security_ApplicationUser,,1244059486102
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
