On Thu, Aug 6, 2009 at 10:25 AM, Andrew Purtell <[email protected]> wrote:
> I updated the roadmap up on the wiki: > > > * Data integrity > * Insure that proper append() support in HDFS actually closes the > WAL last block write hole > * HBase-FSCK (HBASE-7) -- Suggest making this a blocker for 0.21 > > I have had several recent conversations on my travels with people in > Fortune 100 companies (based on this list: > http://www.wageproject.org/content/fortune/index.php). I like that links' topic matter. The question is I think does the above align with project goals. > Making HBase-FSCK a blocker will probably knock something someone > wants for the 0.21 timeframe off the list. > I think topic of integrity is a good one to raise at this time. Its about time for a (re)visit. Is there enough information in the filesystem for an hbasck tool to do its reconstruction work? Regions now have .regioninfo files written to them on creation with regioninfo written to them, the hfiles have first, last, and sequenceids as metadata in them. What else do we need to fully-reconstruct tables when, ${deity} forbid (<-I like this one), there is a catastrophic crash? A requirement of any hbsfck is that it finish promptly (MR job?). It should not be one of those tools that chew for hours on end spinning disks while a progress bar crawls to completion. One area that for sure could do with review is logsplitting and then replay of edits on region redeploy. We've not given this the attention it deserves ensuring we are not dropping edits mostly because we've just presumed loss because up to this there has been no working flush/append. Another interesting question I was asked recently was, if a regionserver crashes during an upload, how do I know what has been lost? From where do I restart the upload? St.Ack
