By "lost tables" i mean no user table definitions were listed in interface
60010 and my queries got me TableNotFoundExceptions. Routine BaseScanner
logged like
# RegionManager.metaScanner scan of 0 row(s) of meta region ... complete
so i guess my .META. was empty. But unfortunately I don't know if any
regions were stuck in transition, i'll be sure to check this out while i try
to reproduce.
I rechecked the logs, specifically after the splitting completed and it
contains lines like "Current assignment is not valid..." so i guess this is
something unexpected. But in hopes of some configuration error you may spot,
whole log goes like that (stripped to be more readable):
# hlog file splitting completed in 80372 millis
# Log split complete, meta reassignment and scanning:
# ProcessServerShutdown reassigning ROOT region
# -ROOT- region unset (but not set to be reassigned)
# ROOT inserted into regionsInTransition
# Assigning for serverName=C... total nregions to assign=1, regions to give
other servers than this=0, isMetaAssign=true
# Assigning serverName=C... 1 regions
# Assigning region -ROOT-... to C...
# Processing MSG_REPORT_OPEN: -ROOT-... from C...; 1 of 1
# RegionManager.rootScanner scanning meta region {server: C ...
# Current assignment of .META.... is not valid; serverAddress=A,
startCode=blah unknown; checking once more!
# Current assignment of .META.... is not valid; serverAddress=A,
startCode=blah unknown.
# RegionManager.rootScanner scan of 1 row(s) of meta region {server: C ...
complete
# RegionManager.rootScanner scanning meta region {server: C ...
# RegionManager.rootScanner scan of 1 row(s) of meta region {server: C ...
complete
# Assigning for serverName=E... total nregions to assign=-1, regions to
give other servers than this=2, isMetaAssign=true
# Assigning serverName=E... -1 regions
# Assigning region .META.... to E...
# Processing MSG_REPORT_OPEN: .META.... from E...; 1 of 1
# Processing todo: PendingOpenOperation from E...
# .META.... open on E
# Updated row .META.... in region -ROOT- ...
# Adding to onlineMetaRegions: {server: E ...
# RegionManager.metaScanner scanning meta region {server: E ...
# RegionManager.metaScanner scan of 0 row(s) of meta region {server: E ...
# All 1 .META. region(s) scanned
10 secs later
# Processing todo: ProcessServerShutdown of F
# Process shutdown of server F...: logSplit: true, rootRescanned: false,
numberOfMetaRegions: 1, onlineMetaRegions.size(): 1
# Log split complete, meta reassignment and scanning
# Process server shutdown scanning root region on C
# Process server shutdown scanning root region on C finished master
# process server shutdown scanning .META.,,1 on E
# process server shutdown finished scanning .META.,,1 on E
# Removed F... from deadservers Map
Thanks again.
On Mon, Nov 1, 2010 at 6:58 PM, Jean-Daniel Cryans <[email protected]>wrote:
> The fact that the tables are "revived" is a clue here IMO, but let's
> go back to more basic questions...
>
> So when you say that during step 1 you lost tables, what do you mean
> by "lost"? Were the rows of those tables still in .META.? Were the
> regions stuck in transition (in the shell, do: status 'detailed')? Or
> when you tried to query them you just got a TableNotFoundException?
>
> Also, the fact that only -ROOT- and not .META. was on this region
> server means that if there was any data lost, it would be .META.'s
> location and it would have been assigned somewhere else (E), but still
> stayed assigned on A. Since the data is in the memstore, recent data
> couldn't be read by this second assignment of .META. but... it could
> also be reassigned for a "normal" reason like rebalancing. The best
> way to confirm that is when the -ROOT- region gets reassigned at the
> end of step 1 (so this is after the message that goes like "...file
> splitting completed in 80372..."), do so see something like this in
> the master's log: "Current assignment of .META.,,some_timestamp is
> not valid; serverAddress=blah, startcode=blah unknown."? If so, then
> it seems that data was lost and this is really unexpected.
>
> J-D
>
> On Mon, Nov 1, 2010 at 1:36 AM, Erdem Agaoglu <[email protected]>
> wrote:
> > Hi again,
> >
> > I have re-checked our configuration to confirm that we have
> > dfs.support.append enabled and saw "Using syncFs -- HDFS-200" in logs. I
> > inspected logs around log splits to find something, but i'm not sure
> that's
> > what we are looking for. In the first step of the scenario i mentioned
> > before (where we kill -9ed everything on the node that hosts the ROOT
> > region), HLog says (stripping hdfs:// prefixes and hostnames for clarity)
> >
> > # Splitting 7 hlog(s) in .logs/F,60020,1287491528908
> >
> > Then it goes over every single one like
> >
> > # Splitting hlog 1 of 7
> > # Splitting hlog 2 of 7
> > # ...
> > # Splitting hlog 7 of 7
> >
> > On the 7th hlog it WARNs with two lines
> >
> > # File .logs/F,60020,1287491528908/10.1.10.229%3A60020.1288021443546
> might
> > be still open, length is 0
> > # Could not open .logs/F,60020,1287491528908/10.1.10.229
> %3A60020.1288021443546
> > for reading. File is emptyjava.io.EOFException
> >
> > And completes with
> >
> > # log file splitting completed in 80372 millis for
> > .logs/F,60020,1287491528908
> >
> > This might be it, but on the sixth step (where we kill -9ed the
> RegionServer
> > that hosts the only META region), it splits 2 hlogs without any empty
> file
> > problems nor any log above INFO, but as i told before, our testtable
> still
> > got lost.
> >
> > I'll try to reproduce the problem in a cleaner way, but in the meantime,
> any
> > kind of pointers to any problems we might have is greatly appreciated.
> >
> >
> > On Fri, Oct 29, 2010 at 9:25 AM, Erdem Agaoglu <[email protected]
> >wrote:
> >
> >> Thanks for the answer.
> >>
> >> I am pretty sure we have dfs.support.append enabled. I remember both the
> >> conf file and the logs, and don't recall seeing any errors on 60010. I
> >> crawled through logs all yesterday but don't remember anything
> indicating a
> >> specific error too. But i'm not sure about that. Let me check that and
> get
> >> back here on monday.
> >>
> >> On Thu, Oct 28, 2010 at 7:30 PM, Jean-Daniel Cryans <
> [email protected]>wrote:
> >>
> >>> First thing I'd check is if your configuration has dfs.support.append,
> >>> you can confirm this by looking at your region server logs. When a RS
> >>> starts, it creates an HLog and will print out: "Using syncFs --
> >>> HDFS-200" if it's configured, else you'll see "syncFs -- HDFS-200 --
> >>> not available, dfs.support.append=false". Also the master web ui (on
> >>> port 60010) will print an error message regarding that.
> >>>
> >>> If it's all ok, then you should take a look at the master log when it
> >>> does the log splitting and see if it contains any obvious errors.
> >>>
> >>> J-D
> >>>
> >>> On Thu, Oct 28, 2010 at 12:58 AM, Erdem Agaoglu <
> [email protected]>
> >>> wrote:
> >>> > Hi all,
> >>> >
> >>> > We have a testing cluster of 6 nodes which we try to run an
> >>> HBase/MapReduce
> >>> > application on. In order to simulate a power failure we kill -9ed all
> >>> things
> >>> > hadoop related on one of the slave nodes (DataNode, RegionServer,
> >>> > TaskTracker, ZK quorum peer and i think SecondaryNameNode was on this
> >>> node
> >>> > too). We were expecting a smooth transition on all services but were
> >>> unable
> >>> > to get on HBase end. While our regions seemed intact (not confirmed),
> we
> >>> > lost table definitions that pointed some kind of META region fail. So
> >>> our
> >>> > application failed with several TableNotFoundExceptions. Simulation
> was
> >>> > conducted with no-load and extremely small data (like 10 rows in 3
> >>> tables).
> >>> >
> >>> > On our setup, HBase is 0.89.20100924, r1001068 while Hadoop
> >>> > runs 0.20.3-append-r964955-1240, r960957. Most of the configuration
> >>> > parameters are in default.
> >>> >
> >>> > If we did something wrong up to this point, please ignore the rest of
> >>> the
> >>> > message as i'll try to explain what we did to reproduce it and might
> be
> >>> > irrelevant.
> >>> >
> >>> > Say the machines are named A, B, C, D, E, F; where A is master-like
> >>> node,
> >>> > others are slaves and power fail is on F. Since we have little data,
> we
> >>> have
> >>> > one ROOT and only one META region. I'll try to sum up the whole
> >>> scenario.
> >>> >
> >>> > A: NN, DN, JT, TT, HM, RS
> >>> > B: DN, TT, RS, ZK
> >>> > C: DN, TT, RS, ZK
> >>> > D: DN, TT, RS, ZK
> >>> > E: DN, TT, RS, ZK
> >>> > F: SNN, DN, TT, RS, ZK
> >>> >
> >>> > 0. Initial state -> ROOT: F, META: A
> >>> > 1. Power fail on F -> ROOT: C, META: E -> lost tables, waited for
> about
> >>> half
> >>> > an hour to get nothing BTW
> >>> > 2. Put F back online -> No effect
> >>> > 3. Create a table 'testtable' to see if we lose it
> >>> > 4. Kill -9ed DataNode on F -> No effect -> Start it again
> >>> > 5. Kill -9ed RegionServer on F -> No effect -> Start it again
> >>> > 6. Kill -9ed RegionServer on E -> ROOT: C, META: A -> We lost
> >>> 'testtable'
> >>> > but get our tables from before the simulation. It seemed like because
> A
> >>> had
> >>> > META before the simulation, the table definitions were revived.
> >>> > 7. Restarted the whole cluster -> ROOT: A, META: F -> We lost 2 out
> of
> >>> our
> >>> > original 6 tables, 'testtable' revived. That small data seems
> corrupted
> >>> too
> >>> > as our Scans don't finish.
> >>> > 8. Run to mailing-list.
> >>> >
> >>> > First of all thanks for reading up to this point. From what we are
> now,
> >>> we
> >>> > are not even sure if this is the expected behavior, like if ROOT or
> META
> >>> > region dies we lose data and must do sth like hbck, or if we are
> missing
> >>> a
> >>> > configuration, or if this is a bug. No need to mention that we are
> >>> > relatively new to HBase so the last possibility is that if we didn't
> >>> > understand it at all.
> >>> >
> >>> > Thanks in advance for any ideas.
> >>> >
> >>> > --
> >>> > erdem agaoglu
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> erdem agaoglu
> >>
> >
> >
> >
> > --
> > erdem agaoglu
> >
>
--
erdem agaoglu
