Ok, but with those caveats, does it work at all? Or is it just 100% broken?
-- Mark Corner On Sunday, November 27, 2011 at 11:04 PM, Andrew Purtell wrote: > A couple points off the top of my head: > > - Consistency (or lack thereof). When HBase writes a file it expects to read > it back, not get a 500 response or a 404, depending. S3N makes this worse by > breaking files into chunks and adding a layer of indirection. > > - Streaming versus random reads: Unlike Hive, HBase reads are at best a mix > of short random reads and streaming reads of large files. At worst, mostly > the former. With HDFS we can ask for a few KB from the middle of a file and > get only that. With S3 the whole block must be read, of 64 MB, just for those > few bytes. Every time. > > - Response latency: With HDFS on the order of milliseconds. With S3, on the > order of seconds in practice. > > I could go on but pondering the above should be sufficient. > > Just because the Hadoop FS layer abstracts HDFS and S3 doesn't mean they are > remotely equivalent, and you should not expect to drop one in place of the > other without radically changing the resulting architecture. > > Best regards, > > - Andy > > > On Nov 27, 2011, at 7:24 PM, Mark Corner <[email protected] > (mailto:[email protected])> wrote: > > > Hrm, that sounds ominous...care to expand? It seems perfectly usable under > > Hive (we have done it). There are certainly indications in various forums > > and books that s3 is a slower, but perfectly usable DFS for HBase. > > > > I don't see anything in JIRA for HBase that says that s3 is broken, should > > I file a bug? > > > > -- > > Mark Corner > > > > > > On Sunday, November 27, 2011 at 10:05 PM, Andrew Purtell wrote: > > > > > Don't do this. It will never work. > > > > > > Best regards, > > > > > > - Andy > > > > > > > > > On Nov 27, 2011, at 7:21 AM, Mark Corner <[email protected] > > > (mailto:[email protected])> wrote: > > > > > > > I am trying to get hbase to run directly over s3n (much preferable) or > > > > s3 (less preferable). I ran into a few different problems that I > > > > solved(such as getting jets3t installed), but now i am stuck!! Sorry > > > > for the long post, but I wanted to include some error messages. > > > > > > > > I am on 0.90.3 installed on a Mac using homebrew. I have hadoop, but > > > > that doesn't come in to play as I am not using HDFS. > > > > > > > > The configuration I am using is below (a few replacements of keys for > > > > anonymity). > > > > > > > > <configuration> > > > > > > > > <property> > > > > <name>hbase.cluster.distributed</name> > > > > <value>true</value> > > > > </property> > > > > > > > > <property> > > > > <name>hbase.rootdir</name> > > > > <value>s3n://mcorner-hbase/</value> (note below, I used different > > > > values for this) > > > > </property> > > > > > > > > <property> > > > > <name>fs.s3n.awsAccessKeyId</name> > > > > <value>mykeyid</value> > > > > </property> > > > > > > > > <property> > > > > <name>fs.s3n.awsSecretAccessKey</name> > > > > <value>mysecret</value> > > > > </property> > > > > </configuration> > > > > > > > > > > > > > > > > > > > > > > > > 1) If I use this hbase.rootdir with s3n (this seems to be the preferred > > > > format…) > > > > <value>s3n://mcorner-hbase.s3.aws.com:80/data > > > > (http://mcorner-hbase.s3.aws.com:80/data)</value> > > > > > > > > then I get this failure. > > > > 2011-11-27 10:03:23,572 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:03:22 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:R0zegLCQbQyGxLVjPbzUobQMqHE=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 0BCBF072804E621A, x-amz-id-2: > > > > KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:03:22 GMT, Server: AmazonS3] > > > > 2011-11-27 10:03:23,573 FATAL org.apache.hadoop.hbase.master.HMaster: > > > > Unhandled exception. Starting shutdown. > > > > org.apache.hadoop.fs.s3.S3Exception: > > > > org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on > > > > Host 'mcorner-hbase.s3.aws.com.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 > > > > 15:03:22 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, > > > > RequestId: 0BCBF072804E621A, HostId: > > > > KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm > > > > at > > > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:122) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > > > > > > > So it looks like it is appending other things to the URL: > > > > "mcorner-hbase.s3.aws.com.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)" > > > > > > > > 2) If I replace the rootdir with: > > > > <value>s3n://mcorner-hbase:80/data</value> > > > > same problem: > > > > 2011-11-27 10:18:21,848 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:18:20 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:3tlyWfqbSkZB9lKQUCbQ8DW2VKk=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 2AFB15695EF68E13, x-amz-id-2: > > > > grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:18:21 GMT, Server: AmazonS3] > > > > 2011-11-27 10:18:21,849 FATAL org.apache.hadoop.hbase.master.HMaster: > > > > Unhandled exception. Starting shutdown. > > > > org.apache.hadoop.fs.s3.S3Exception: > > > > org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on > > > > Host 'mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:18:21 > > > > GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: > > > > 2AFB15695EF68E13, HostId: > > > > grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy > > > > > > > > > > > > 3) If I switch to s3 (and change the key properties to match), with > > > > s3n://mcorner-hbase:80/data, then I get: > > > > > > > > 2011-11-27 10:08:55,951 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:08:54 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:nKXZEBy/WIinzNByPX7JKBasSVw=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: B8A5A618F8F86B87, x-amz-id-2: > > > > 7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:08:55 GMT, Server: AmazonS3] > > > > 2011-11-27 10:08:55,958 FATAL org.apache.hadoop.hbase.master.HMaster: > > > > Unhandled exception. Starting shutdown. > > > > org.apache.hadoop.fs.s3.S3Exception: > > > > org.jets3t.service.S3ServiceException: S3 Error Message. GET '/%2Fdata' > > > > on Host 'mcorner-hbase.s3.aws.com.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 > > > > 15:08:55 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML > > > > Error Message: <?xml version="1.0" > > > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The > > > > specified bucket does not > > > > exist</Message><BucketName>mcorner-hbase.s3.aws.com > > > > (http://mcorner-hbase.s3.aws.com)</BucketName><RequestId>B8A5A618F8F86B87</RequestId><HostId>7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B</HostId></Error> > > > > > > > > > > > > > > > > 4) The closest I get is with <value>s3://mcorner-hbase/data</value>. > > > > Then I get what appears to be more progress, but it appears to be > > > > creating files (I can see a file called "block_-222…." in my bucket, > > > > and a directory with a blank name, but the bootstrapping process > > > > eventually bombs out as you can se. > > > > > > > > 2011-11-27 10:11:12,793 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:11 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:rRR32CQb1xmqLv9PgEwOIVaTLMQ=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 2296D45E1390890C, x-amz-id-2: > > > > ZT9IpXdAIAqWu+owdtilWQWj+9nRQmH48IhOFoWajFhuRSEOcTplMjekzZa8wkXb, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:11 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:12,905 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2F' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:12 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:Be5+Q5QRILCtbJzUA6HcHEP9YZ8=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 6B6159CBC16B4742, x-amz-id-2: > > > > 4mBHDq4YdtrIBQS63x8zKjjpt2md2raM+LcXpIaGwybGrRPUjxOM53ZuYBab37FO, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:11 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:13,332 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:xPkvn6VtlerDOhZL604gwM3848c=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 46EC664A7B09DC6B, x-amz-id-2: > > > > 9mb0kcVcdjm86x69r9QDbN6cATm3nUKSPIfcxs7rs79M5CZ527z25F2Y3c6CzRdW, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:12 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:13,604 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2Fhbase.version' -- ResponseCode: 404, ResponseStatus: > > > > Not Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 > > > > 15:11:13 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:6iBF8bkt+roA4N5/1RvL7X4qP9M=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 640F58D7AAF536E3, x-amz-id-2: > > > > QY3YqAO4+652mNCPaIIqMe3tKTigUs+YlOj/vzKuRPJXiCD7DXEmziqdX/145Xov, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:12 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:14,023 DEBUG org.apache.hadoop.hbase.util.FSUtils: > > > > Created version file at s3://mcorner-hbase/data set its version at:7 > > > > 2011-11-27 10:11:14,281 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/block_-2255577201702834920' -- ResponseCode: 404, ResponseStatus: > > > > Not Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 > > > > 15:11:14 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:gICpmb7gELgfm2sW02O2OBQGSPw=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: DE383F7CAFC6856E, x-amz-id-2: > > > > y3ffN4iTYIkR+iOJI9uJkW8xntsdBZHq4miEsO1qz1YLdJlAlpBjNM30n2ASx9bs, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:13 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:14,836 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:14 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:XbfWC0XVWS53IAEHGTNvkBZ9gQk=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 25AE3E24162F7850, x-amz-id-2: > > > > Q2oOyCd2NDQPiY2FjJPuI8pvhTid/dLyhFGQymF3SDvHkT0krDh79JerjxBt4FGY, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:14 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:14,838 INFO > > > > org.apache.hadoop.hbase.master.MasterFileSystem: BOOTSTRAP: creating > > > > ROOT and first META regions > > > > 2011-11-27 10:11:15,462 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-' -- ResponseCode: 404, ResponseStatus: Not > > > > Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 > > > > 15:11:15 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:GdG0M+6O3+Cbls3DzXSfEIrISDw=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: AEEB8EFE863A9B5A, x-amz-id-2: > > > > P+wk8B2p3sRb/MR3uwG1jjH8BFwT6xHM3GXmoXjRwt/sc01NVNrJVUpr48j3Df4a, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:14 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:15,735 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:15 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:qgFz5nvPmwHYLIKudvclcGJK2gw=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 7856EA5209B7D4A8, x-amz-id-2: > > > > lAQR13CX8ZfwSMKjl4SfErVtQPUwss5zCrcPg2tV5CpJ1bEmixNOpACIEVzwW/jf, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:14 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:16,024 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:15 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:go7R49fCUX01Gn5thKP4nl2emh0=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 9AB9F6F7CE9012CD, x-amz-id-2: > > > > GifVpi0k+F/se1gjESuUyFzUtSY93GyvLggqpWn6ZgxTMhwEw0YMa5jG1FTHhA33, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:14 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:17,636 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:16 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:pUj2QwlQa7ypigvoUs4LHikhDd0=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 5120CAE72D290EB7, x-amz-id-2: > > > > aCkMJu4F5FrvRAYRHac/CTxGsgSN3VgBfAEo/jVW0OeXaQNvZ09y6Qu8moV03fvp, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:16 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:17,926 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:17 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:8fAMMGghzJs6ICOfqy5Euf0nuC8=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: BEDBB9E931FEE069, x-amz-id-2: > > > > R9sMP4IkMOXsUCCzZErJYt9qGHhadkKstadB7pcpo0XKX9l4L+Clna4SPfZhNzyp, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:17 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:20,130 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404, > > > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, > > > > 27 Nov 2011 15:11:19 GMT, Authorization: AWS > > > > AKIAJSHPUUTMSSRCOTBA:Hg1cqsBmprgya4GXSeAOVhVoju0=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 8752AD561C46847E, x-amz-id-2: > > > > 6bA9vjM4kLhQoL0uaY2hG30atmaYKYAXJaIb3utehSqXlIlH8UoN1KG0lnwe4UfR, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:18 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:20,307 INFO > > > > org.apache.hadoop.hbase.regionserver.wal.HLog: HLog configuration: > > > > blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1, > > > > optionallogflushinternal=1000ms > > > > 2011-11-27 10:11:20,403 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' -- > > > > ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > > > [Content-Type: , Date: Sun, 27 Nov 2011 15:11:20 GMT, Authorization: > > > > AWS AKIAJSHPUUTMSSRCOTBA:IaE+/XqX0aUmfo0LJw9O7pCsV1A=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: F84C1DC251EB6F09, x-amz-id-2: > > > > viG+7saKo8ZiPro1oWyNNqh8kAnve/5nddl35mETpf6udDScbbEY3ZCqV0pB7uJR, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:19 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:21,578 INFO > > > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs > > > > -- HDFS-200 -- not available, dfs.support.append=false > > > > 2011-11-27 10:11:21,837 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' -- > > > > ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > > > [Content-Type: , Date: Sun, 27 Nov 2011 15:11:21 GMT, Authorization: > > > > AWS AKIAJSHPUUTMSSRCOTBA:5l0MMGpPTR3OCqmgji3W3bWoLgQ=, User-Agent: > > > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > > > mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 9B1502504A01DCF4, x-amz-id-2: > > > > Kdzu/QRSHRgrrURTYiwijDgEIHcdZZZzGhleYsgsuLYlSEQ5D5WKCrhYhcDuZwHw, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:11:20 GMT, Server: AmazonS3] > > > > 2011-11-27 10:11:21,839 ERROR > > > > org.apache.hadoop.hbase.master.MasterFileSystem: bootstrap > > > > java.io.FileNotFoundException: > > > > s3://mcorner-hbase/data/-ROOT-/70236052/.logs/hlog.1322406680307: No > > > > such file or directory. > > > > at > > > > org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334) > > > > at > > > > org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:486) > > > > at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:374) > > > > > > > > 5) I have also tried creating the data directory in my s3 bucket, and > > > > that doesn't help. > > > > 2011-11-27 10:20:18,003 WARN > > > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: > > > > HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:20:15 GMT, > > > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:AqeMf6wmBZGJye7qwiYFrG6ozSA=, > > > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM > > > > 1.6.0_26), Host: mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > > > [x-amz-request-id: 168138BCA8FCBD1D, x-amz-id-2: > > > > o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko, > > > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, > > > > 27 Nov 2011 15:20:17 GMT, Server: AmazonS3] > > > > 2011-11-27 10:20:18,004 FATAL org.apache.hadoop.hbase.master.HMaster: > > > > Unhandled exception. Starting shutdown. > > > > org.apache.hadoop.fs.s3.S3Exception: > > > > org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on > > > > Host 'mcorner-hbase.s3.amazonaws.com > > > > (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:20:17 > > > > GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: > > > > 168138BCA8FCBD1D, HostId: > > > > o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko > > > > > > > > > > > > > > > > What is the magic combination here?? Do you need to create the > > > > directory? What are you using for the rootdir url?? > > > > > > > > > > > > Mark Corner > > > > VP Engineering > > > > Mobile: +1.413.330.9601 > > > > Fiksu (http://www.fiksu.com) | LinkedIn > > > > (http://www.linkedin.com/in/mcorner) | Research > > > > (http://www.cs.umass.edu/~mcorner) > > > > > > > > > > > > > > > > >
