Stuart: If you have separated domain-specific tweaks out of this layer, definitely you can put your code on github.
Cheers On Wed, Aug 25, 2010 at 11:22 AM, Stuart Smith <[email protected]> wrote: > > Hello St.Ack, > > Hmm.. I actually just dropped everything in the database & rebuilt - along > with some very much needed cleanups, improvements, code-refactoring, etc. > > Just curious, though, (if it happens again) - assume the regions were > invalid - I don't know, maybe it was halfway through splitting something and > died - but say they're invalid. > > Would the best thing to do in that case be a manual deletion of the hdfs > directories containing the invalid regions? What hbase handle that OK? > > And a side question that ties a lot of my issues together - I finally have > a (somewhat) clean interface that moves the occasional too big file into > hdfs, and stores everything else into hbase - I built this up as a layer in > java with a metadata/filestore split in hbase (all file metadata is in > hbase, files are directed to hbase/hdfs based on size). > > Is there another project that does this? It seems too handy to be the first > time someone did this... Or does something like this always end up needing > domain-specific tweaks & interfaces? > > Because once you have huge cells in hbase, it really seems to be unhappy. > Especially when a good chunk of your tasks are done as M/R tasks or some > layer on top of M/R. > > Or would this be a good project to open-source? Or pointless to do so? > > I guess in the long-run hbase could absorb these requirements with some > tweaks of the file format, but I thought it could be nice to do this with a > little library layer on top. > > Take care, > -stu > > > --- On Mon, 8/23/10, Stack <[email protected]> wrote: > > > From: Stack <[email protected]> > > Subject: Re: WARN add_table: Missing .regioninfo:.. No server address.. > what to do? > > To: [email protected] > > Date: Monday, August 23, 2010, 6:08 PM > > On Mon, Aug 23, 2010 at 1:35 PM, > > Stuart Smith <[email protected]> > > wrote: > > > > > > Hmm... AFAICT, if the regioninfo files is gone from a > > region directory (and I looked on hdfs, and it is gone), the > > region is hosed. > > > > Is it a legit region? Its wholesome looking with > > hfiles that make > > sense (non-zero)? My guess is that the regions are > > incompletes and > > loadtable is not smart enough recognizing them as so. > > If you grep > > your master log for the region encoded name, do you find > > anything? > > Maybe this way you can figure its provenance? > > > > St.Ack > > > > > > >
