Hey, Awesome. Well, this is a research project for work, so I have to ask the powers that be if it's OK to publish the plumbing parts.
It's really just plumbing though, so from the techy perspective it's not the "interesting" part. So hopefully I can sell it as such (selling my work to the boss as not interesting.. hmm... ;) ). We'll see. I'm not an expert Java coder either, but, hopefully I can get it up and stimulate something... Take care, -stu --- On Thu, 8/26/10, Stack <[email protected]> wrote: > From: Stack <[email protected]> > Subject: Re: WARN add_table: Missing .regioninfo:.. No server address.. what > to do? > To: [email protected] > Date: Thursday, August 26, 2010, 2:11 AM > On Wed, Aug 25, 2010 at 11:22 AM, > Stuart Smith <[email protected]> > wrote: > > Just curious, though, (if it happens again) - assume > the regions were invalid - I don't know, maybe it was > halfway through splitting something and died - but say > they're invalid. > > > > (See if a failed MR task associated with the bad > region. You could > also tgz' the bad region and we can take a look at it for > you.) > > > Would the best thing to do in that case be a manual > deletion of the hdfs directories containing the invalid > regions? What hbase handle that OK? > > > > If its a 'bad' region, should be fine. There'd be no > holes in loaded > table. But if its not... > > > And a side question that ties a lot of my issues > together - I finally have a (somewhat) clean interface that > moves the occasional too big file into hdfs, and stores > everything else into hbase - I built this up as a layer in > java with a metadata/filestore split in hbase (all file > metadata is in hbase, files are directed to hbase/hdfs based > on size). > > > > Is there another project that does this? It seems too > handy to be the first time someone did this... Or does > something like this always end up needing domain-specific > tweaks & interfaces? > > > > I haven't heard of a project like this (though as you say, > you can't > be the first... maybe you are though?) > > > Because once you have huge cells in hbase, it really > seems to be unhappy. Especially when a good chunk of your > tasks are done as M/R tasks or some layer on top of M/R. > > > > Yeah, I'd imagine so. At least default configuration > is set for cells > in the 0-50k or so size. I'd imagine they'd need to > be pulled around > some if cells are MBs. > > > Or would this be a good project to open-source? Or > pointless to do so? > > > > Do it on github as Ted suggests. It'll either > flourish and then > you'll have to figure out how to support it or it'll wither > when you > move on (add it to supporting projects on wiki so its > easier for folks > to find?) > > > I guess in the long-run hbase could absorb these > requirements with some tweaks of the file format, but I > thought it could be nice to do this with a little library > layer on top. > > > > You are a good man Stu, > St.Ack > > > > --- On Mon, 8/23/10, Stack <[email protected]> > wrote: > > > >> From: Stack <[email protected]> > >> Subject: Re: WARN add_table: Missing > .regioninfo:.. No server address.. what to do? > >> To: [email protected] > >> Date: Monday, August 23, 2010, 6:08 PM > >> On Mon, Aug 23, 2010 at 1:35 PM, > >> Stuart Smith <[email protected]> > >> wrote: > >> > > >> > Hmm... AFAICT, if the regioninfo files is > gone from a > >> region directory (and I looked on hdfs, and it is > gone), the > >> region is hosed. > >> > >> Is it a legit region? Its wholesome looking > with > >> hfiles that make > >> sense (non-zero)? My guess is that the regions > are > >> incompletes and > >> loadtable is not smart enough recognizing them as > so. > >> If you grep > >> your master log for the region encoded name, do > you find > >> anything? > >> Maybe this way you can figure its provenance? > >> > >> St.Ack > >> > > > > > > > > > > >
