On Wed, Aug 25, 2010 at 11:22 AM, Stuart Smith <[email protected]> wrote: > Just curious, though, (if it happens again) - assume the regions were invalid > - I don't know, maybe it was halfway through splitting something and died - > but say they're invalid. >
(See if a failed MR task associated with the bad region. You could also tgz' the bad region and we can take a look at it for you.) > Would the best thing to do in that case be a manual deletion of the hdfs > directories containing the invalid regions? What hbase handle that OK? > If its a 'bad' region, should be fine. There'd be no holes in loaded table. But if its not... > And a side question that ties a lot of my issues together - I finally have a > (somewhat) clean interface that moves the occasional too big file into hdfs, > and stores everything else into hbase - I built this up as a layer in java > with a metadata/filestore split in hbase (all file metadata is in hbase, > files are directed to hbase/hdfs based on size). > > Is there another project that does this? It seems too handy to be the first > time someone did this... Or does something like this always end up needing > domain-specific tweaks & interfaces? > I haven't heard of a project like this (though as you say, you can't be the first... maybe you are though?) > Because once you have huge cells in hbase, it really seems to be unhappy. > Especially when a good chunk of your tasks are done as M/R tasks or some > layer on top of M/R. > Yeah, I'd imagine so. At least default configuration is set for cells in the 0-50k or so size. I'd imagine they'd need to be pulled around some if cells are MBs. > Or would this be a good project to open-source? Or pointless to do so? > Do it on github as Ted suggests. It'll either flourish and then you'll have to figure out how to support it or it'll wither when you move on (add it to supporting projects on wiki so its easier for folks to find?) > I guess in the long-run hbase could absorb these requirements with some > tweaks of the file format, but I thought it could be nice to do this with a > little library layer on top. > You are a good man Stu, St.Ack > --- On Mon, 8/23/10, Stack <[email protected]> wrote: > >> From: Stack <[email protected]> >> Subject: Re: WARN add_table: Missing .regioninfo:.. No server address.. what >> to do? >> To: [email protected] >> Date: Monday, August 23, 2010, 6:08 PM >> On Mon, Aug 23, 2010 at 1:35 PM, >> Stuart Smith <[email protected]> >> wrote: >> > >> > Hmm... AFAICT, if the regioninfo files is gone from a >> region directory (and I looked on hdfs, and it is gone), the >> region is hosed. >> >> Is it a legit region? Its wholesome looking with >> hfiles that make >> sense (non-zero)? My guess is that the regions are >> incompletes and >> loadtable is not smart enough recognizing them as so. >> If you grep >> your master log for the region encoded name, do you find >> anything? >> Maybe this way you can figure its provenance? >> >> St.Ack >> > > > > >
