HBase is not developed or tested against Hadoop's S3FS. Have you considered using HDFS backed by EBS volumes instead of S3? HDFS on top of EBS insures the filesystem semantics that HBase expects.
Getting back to the first pastebin, I see this > 2009-10-26 11:38:12,282 INFO org.apache.hadoop.hbase.master.HMaster: Found log folder : ip-10-244-9-171.ec2.internal,60020,1255635341245 > 2009-10-26 11:38:12,282 INFO org.apache.hadoop.hbase.master.HMaster: Log folder doesn't belong to a known region server, splitting On a clean shutdown, logs will be deleted. And in your filesystem listing, there are some pretty big reconstruction logs: > -rwxrwxrwx 1 148161186 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/oldlogfile.log Was this a clean shutdown of HBase and Hadoop? (Stop services with stop-hbase.sh etc. then shut down, not just ec2-terminate-instance.)Did some region server crash? Any errors in the region server logs during shutdown? - Andy ________________________________ From: Ananth T. Sarathy <[email protected]> To: [email protected] Sent: Mon, October 26, 2009 10:58:56 AM Subject: Re: None of my tables are showing up Here is the output... drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT- drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052 -rwxrwxrwx 1 622 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/.regioninfo drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info -rwxrwxrwx 1 841 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/1996912273062753577 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/2193779228149112035 -rwxrwxrwx 1 462 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/2351329094215113700 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/246199753790950739 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/2525006410357531294 -rwxrwxrwx 1 462 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/2791558595254923243 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/4021322725584207875 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/4849111854392651007 -rwxrwxrwx 1 462 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/5363712926256617749 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/5611302436720701967 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/6831054531189082272 -rwxrwxrwx 1 462 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/68808045018098702 -rwxrwxrwx 1 462 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/7253796415415350972 -rwxrwxrwx 1 461 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info/7757549506442648999 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META. drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192 -rwxrwxrwx 1 932 1969-12-31 19:00 /hbasedata/.META./1028785192/.regioninfo drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192/historian -rwxrwxrwx 1 911 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/1968569179310304585 -rwxrwxrwx 1 907 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/2158222582121734940 -rwxrwxrwx 1 907 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/2646558927756874624 -rwxrwxrwx 1 907 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/3553325538317585634 -rwxrwxrwx 1 2037 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/4092877814999091772 -rwxrwxrwx 1 1185 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/4380516119952135238 -rwxrwxrwx 1 2437 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/6276910425096618885 -rwxrwxrwx 1 911 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/7295248711127991783 -rwxrwxrwx 1 911 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/7481877460145101553 -rwxrwxrwx 1 911 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/752086821053742201 -rwxrwxrwx 1 907 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/7817013763537032897 -rwxrwxrwx 1 1173 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/8114421049620083173 -rwxrwxrwx 1 1563 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/8322527460340654381 -rwxrwxrwx 1 911 1969-12-31 19:00 /hbasedata/.META./1028785192/historian/8999997006964561665 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192/info -rwxrwxrwx 1 694 1969-12-31 19:00 /hbasedata/.META./1028785192/info/1512108120879032405 -rwxrwxrwx 1 3971 1969-12-31 19:00 /hbasedata/.META./1028785192/info/2775950188169392903 -rwxrwxrwx 1 646 1969-12-31 19:00 /hbasedata/.META./1028785192/info/3098482905740093325 -rwxrwxrwx 1 696 1969-12-31 19:00 /hbasedata/.META./1028785192/info/3098738884562255978 -rwxrwxrwx 1 1561 1969-12-31 19:00 /hbasedata/.META./1028785192/info/383466394636378400 -rwxrwxrwx 1 694 1969-12-31 19:00 /hbasedata/.META./1028785192/info/4974231550598739590 -rwxrwxrwx 1 694 1969-12-31 19:00 /hbasedata/.META./1028785192/info/5034205868382579974 -rwxrwxrwx 1 694 1969-12-31 19:00 /hbasedata/.META./1028785192/info/5137021646723327707 -rwxrwxrwx 1 696 1969-12-31 19:00 /hbasedata/.META./1028785192/info/5626349865156916725 -rwxrwxrwx 1 696 1969-12-31 19:00 /hbasedata/.META./1028785192/info/6707659272909568633 -rwxrwxrwx 1 696 1969-12-31 19:00 /hbasedata/.META./1028785192/info/7346625667936340286 -rwxrwxrwx 1 696 1969-12-31 19:00 /hbasedata/.META./1028785192/info/7438651887567350087 -rwxrwxrwx 1 5699 1969-12-31 19:00 /hbasedata/.META./1028785192/info/7671471165225263042 -rwxrwxrwx 1 1202 1969-12-31 19:00 /hbasedata/.META./1028785192/info/8586093328550564694 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./compaction.dir drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-242-71-191.ec2.internal,60020,1256571494907 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1255635341245 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1256571493935 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-245-82-160.ec2.internal,58439,1255623467258 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1255977369779 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1256571494768 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711 -rwxrwxrwx 1 2632 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/.regioninfo drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Application -rwxrwxrwx 1 13303446 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Application/2124252833220218245 -rwxrwxrwx 1 26149 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Application/5509287308900349804 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryData -rwxrwxrwx 1 267219 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryData/2009456488953384412 -rwxrwxrwx 1 66582 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryData/4704251150705174118 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryRetrieval -rwxrwxrwx 1 1325402 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryRetrieval/1389090912532859839 -rwxrwxrwx 1 3093 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/BinaryRetrieval/9178892710156256966 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Files -rwxrwxrwx 1 69548974 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Files/6175860477346559387 -rwxrwxrwx 1 157418897 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Files/7670651200769000729 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Info -rwxrwxrwx 1 71844 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Info/1715357090325080119 -rwxrwxrwx 1 36052800 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Info/7204941744454360171 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Network -rwxrwxrwx 1 42060118 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/Network/3039242992290063476 -rwxrwxrwx 1 148161186 1969-12-31 19:00 /hbasedata/GS_Applications/1264304711/oldlogfile.log drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1732076772 -rwxrwxrwx 1 2484 1969-12-31 19:00 /hbasedata/GS_Applications/1732076772/.regioninfo drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139 -rwxrwxrwx 1 2780 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/.regioninfo drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Application -rwxrwxrwx 1 27428283 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Application/7122413424433440685 -rwxrwxrwx 1 47203 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Application/8284859005131013460 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/BinaryData -rwxrwxrwx 1 311872 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/BinaryData/8301036036223997974 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/BinaryRetrieval -rwxrwxrwx 1 6368 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/BinaryRetrieval/1610935246237259262 -rwxrwxrwx 1 2738315 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/BinaryRetrieval/8651784529191787005 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Files -rwxrwxrwx 1 127201108 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Files/7247655344177400794 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Info -rwxrwxrwx 1 74860411 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Info/514906809998228574 -rwxrwxrwx 1 162245 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Info/8300037259597514515 drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Network -rwxrwxrwx 1 228 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/Network/2748465749080635338 -rwxrwxrwx 1 32857683 1969-12-31 19:00 /hbasedata/GS_Applications/1888816139/oldlogfile.log drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications/compaction.dir drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_EventLists drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_EventLists/453741364 -rwxrwxrwx 1 1191 1969-12-31 19:00 /hbasedata/GS_EventLists/453741364/.regioninfo -rwxrwxrwx 1 3 1969-12-31 19:00 /hbasedata/hbase.version Ananth T Sarathy On Mon, Oct 26, 2009 at 1:56 PM, stack <[email protected]> wrote: > If you open the .regioninfo file inside in each region, you can see its > content? You might be able to make loadtable.rb work by making it go via > jets3s but would take some tinkering on your part. > > As to why the regions were lost from .META. in first place, I'm not sure > why; was there no flush of .META.? Usually this is frequent so you > should > have some of your table info. > > We're about hdfs here in this channel, usually, and speaking for myself, > I've played with hadoop on s3 but not hbase so I'm not much good to you I'm > afraid. > > St.Ack > > > On Mon, Oct 26, 2009 at 10:42 AM, Ananth T. Sarathy < > [email protected]> wrote: > > > yeah I see a folder with my table name. and inside that I see 3 other > > folders named with a bunch of numbers (which i assume are the regions). > > > > Ananth T Sarathy > > > > > > On Mon, Oct 26, 2009 at 1:38 PM, stack <[email protected]> wrote: > > > > > If you list the hbase homedir in s3, do you see tables and then under > > that > > > regions? > > > > > > The loadtable.rb script won't work against a fs that is other than hdfs > > > unfortunately. > > > St.Ack > > > > > > On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy < > > > [email protected]> wrote: > > > > > > > well where i have hadoop installed (not running) since not using it > for > > > > hdfs > > > > > > > > bin/hadoop fs -lsr /hbase.rootdir > > > > lsr: Cannot access /hbase.rootdir: No such file or directory. > > > > > > > > > > > > > > > > Ananth T Sarathy > > > > > > > > > > > > On Mon, Oct 26, 2009 at 1:24 PM, stack <[email protected]> wrote: > > > > > > > > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens? > > > > > St.Ack > > > > > > > > > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy < > > > > > [email protected]> wrote: > > > > > > > > > > > @ the command line? I get a command not found. > > > > > > Ananth T Sarathy > > > > > > > > > > > > > > > > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <[email protected]> wrote: > > > > > > > > > > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon). > > > > > > > > > > > > > > What happens if you do a lsr /hbase? > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > I don't have a loadtable.rb > > > > > > > > > > > > > > > > is that in 0.20.0? > > > > > > > > This is what i have in bin > > > > > > > > > > > > > > > > Formatter.rb hbase-config.sh regionservers.sh > > zookeepers.sh > > > > > > > > HBase.rb hbase-daemon.sh rename_table.rb > > > > > > > > copy_table.rb hbase-daemons.sh start-hbase.sh > > > > > > > > hbase hirb.rb stop-hbase.sh > > > > > > > > > > > > > > > > Ananth T Sarathy > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > Your log would seem to say that there are no tables in > hbase: > > > > > > > > > > > > > > > > > > 2009-10-26 11:40:13,984 INFO > > > > > > > org.apache.hadoop.hbase.master.BaseScanner: > > > > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta region > > > > {server: > > > > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} > > > > complete > > > > > > > > > > > > > > > > > > Do as Jon suggests. Do you see listing of regions? If so, > > it > > > > > would > > > > > > > seem > > > > > > > > > that edits to .META. table are not persisting on your s3 > > hdfs. > > > > > You > > > > > > > > might > > > > > > > > > be able to add in all tables using the bin/loadtable.rb > > script; > > > > it > > > > > > > reads > > > > > > > > > the > > > > > > > > > .regioninfo files in all regions and per region adds to > > .META. > > > an > > > > > > > entry. > > > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > I am confused , why would I need a hadoop home if I am > > using > > > s3 > > > > > and > > > > > > > the > > > > > > > > > > jets3t package to write to s3? > > > > > > > > > > Ananth T Sarathy > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Not S3, HDFS. Can you checkout the web ui or using the > > > > > > > command-line > > > > > > > > > > > interface? > > > > > > > > > > > > > > > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase > > > > > > > > > > > > > > > > > > > > > > ...would be a good start > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ananth T. Sarathy wrote: > > > > > > > > > > > > > > > > > > > > > >> i see all my blocks in my s3 bucket. > > > > > > > > > > >> Ananth T Sarathy > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray < > > > > > > > [email protected]> > > > > > > > > > > >> wrote: > > > > > > > > > > >> > > > > > > > > > > >> Do you see the files/blocks in HDFS? > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> Ananth T. Sarathy wrote: > > > > > > > > > > >>> > > > > > > > > > > >>> I just restarted Hbase and when I go into the shell > > and > > > > type > > > > > > > list, > > > > > > > > > > none > > > > > > > > > > >>>> of > > > > > > > > > > >>>> my tables are listed, but I see all the data/blocks > in > > > s3. > > > > > > > > > > >>>> > > > > > > > > > > >>>> here is the master log when it's restarted > > > > > > > > > > >>>> > > > > > > > > > > >>>> http://pastebin.com/m1ebb7217 > > > > > > > > > > >>>> > > > > > > > > > > >>>> this happened once before, but we just started over > > > since > > > > it > > > > > > was > > > > > > > > > early > > > > > > > > > > >>>> in > > > > > > > > > > >>>> our process. This time we have a lot of data, and > need > > > to > > > > > keep > > > > > > > it. > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> Ananth T Sarathy > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
