Hrm, that sounds ominous...care to expand? It seems perfectly usable under Hive (we have done it). There are certainly indications in various forums and books that s3 is a slower, but perfectly usable DFS for HBase.
I don't see anything in JIRA for HBase that says that s3 is broken, should I file a bug? -- Mark Corner On Sunday, November 27, 2011 at 10:05 PM, Andrew Purtell wrote: > Don't do this. It will never work. > > Best regards, > > - Andy > > > On Nov 27, 2011, at 7:21 AM, Mark Corner <[email protected] > (mailto:[email protected])> wrote: > > > I am trying to get hbase to run directly over s3n (much preferable) or s3 > > (less preferable). I ran into a few different problems that I solved(such > > as getting jets3t installed), but now i am stuck!! Sorry for the long post, > > but I wanted to include some error messages. > > > > I am on 0.90.3 installed on a Mac using homebrew. I have hadoop, but that > > doesn't come in to play as I am not using HDFS. > > > > The configuration I am using is below (a few replacements of keys for > > anonymity). > > > > <configuration> > > > > <property> > > <name>hbase.cluster.distributed</name> > > <value>true</value> > > </property> > > > > <property> > > <name>hbase.rootdir</name> > > <value>s3n://mcorner-hbase/</value> (note below, I used different values > > for this) > > </property> > > > > <property> > > <name>fs.s3n.awsAccessKeyId</name> > > <value>mykeyid</value> > > </property> > > > > <property> > > <name>fs.s3n.awsSecretAccessKey</name> > > <value>mysecret</value> > > </property> > > </configuration> > > > > > > > > > > > > 1) If I use this hbase.rootdir with s3n (this seems to be the preferred > > format…) > > <value>s3n://mcorner-hbase.s3.aws.com:80/data > > (http://mcorner-hbase.s3.aws.com:80/data)</value> > > > > then I get this failure. > > 2011-11-27 10:03:23,572 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD > > '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:03:22 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:R0zegLCQbQyGxLVjPbzUobQMqHE=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.aws.com.s3.amazonaws.com > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 0BCBF072804E621A, x-amz-id-2: > > KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:03:22 GMT, Server: AmazonS3] > > 2011-11-27 10:03:23,573 FATAL org.apache.hadoop.hbase.master.HMaster: > > Unhandled exception. Starting shutdown. > > org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: > > Request Error. HEAD '/data' on Host > > 'mcorner-hbase.s3.aws.com.s3.amazonaws.com > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 > > 15:03:22 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: > > 0BCBF072804E621A, HostId: > > KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm > > at > > org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:122) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > > So it looks like it is appending other things to the URL: > > "mcorner-hbase.s3.aws.com.s3.amazonaws.com > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)" > > > > 2) If I replace the rootdir with: > > <value>s3n://mcorner-hbase:80/data</value> > > same problem: > > 2011-11-27 10:18:21,848 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD > > '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:18:20 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:3tlyWfqbSkZB9lKQUCbQ8DW2VKk=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 2AFB15695EF68E13, x-amz-id-2: > > grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:18:21 GMT, Server: AmazonS3] > > 2011-11-27 10:18:21,849 FATAL org.apache.hadoop.hbase.master.HMaster: > > Unhandled exception. Starting shutdown. > > org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: > > Request Error. HEAD '/data' on Host 'mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:18:21 GMT' > > -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: > > 2AFB15695EF68E13, HostId: > > grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy > > > > > > 3) If I switch to s3 (and change the key properties to match), with > > s3n://mcorner-hbase:80/data, then I get: > > > > 2011-11-27 10:08:55,951 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:08:54 GMT, > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:nKXZEBy/WIinzNByPX7JKBasSVw=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: B8A5A618F8F86B87, x-amz-id-2: > > 7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:08:55 GMT, Server: AmazonS3] > > 2011-11-27 10:08:55,958 FATAL org.apache.hadoop.hbase.master.HMaster: > > Unhandled exception. Starting shutdown. > > org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: > > S3 Error Message. GET '/%2Fdata' on Host > > 'mcorner-hbase.s3.aws.com.s3.amazonaws.com > > (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 > > 15:08:55 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML Error > > Message: <?xml version="1.0" > > encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified > > bucket does not exist</Message><BucketName>mcorner-hbase.s3.aws.com > > (http://mcorner-hbase.s3.aws.com)</BucketName><RequestId>B8A5A618F8F86B87</RequestId><HostId>7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B</HostId></Error> > > > > > > > > 4) The closest I get is with <value>s3://mcorner-hbase/data</value>. Then I > > get what appears to be more progress, but it appears to be creating files > > (I can see a file called "block_-222…." in my bucket, and a directory with > > a blank name, but the bootstrapping process eventually bombs out as you can > > se. > > > > 2011-11-27 10:11:12,793 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:11 GMT, > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:rRR32CQb1xmqLv9PgEwOIVaTLMQ=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 2296D45E1390890C, x-amz-id-2: > > ZT9IpXdAIAqWu+owdtilWQWj+9nRQmH48IhOFoWajFhuRSEOcTplMjekzZa8wkXb, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:11 GMT, Server: AmazonS3] > > 2011-11-27 10:11:12,905 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2F' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:11:12 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:Be5+Q5QRILCtbJzUA6HcHEP9YZ8=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 6B6159CBC16B4742, x-amz-id-2: > > 4mBHDq4YdtrIBQS63x8zKjjpt2md2raM+LcXpIaGwybGrRPUjxOM53ZuYBab37FO, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:11 GMT, Server: AmazonS3] > > 2011-11-27 10:11:13,332 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request > > Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT, > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:xPkvn6VtlerDOhZL604gwM3848c=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 46EC664A7B09DC6B, x-amz-id-2: > > 9mb0kcVcdjm86x69r9QDbN6cATm3nUKSPIfcxs7rs79M5CZ527z25F2Y3c6CzRdW, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:12 GMT, Server: AmazonS3] > > 2011-11-27 10:11:13,604 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2Fhbase.version' -- ResponseCode: 404, ResponseStatus: Not Found, > > Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT, > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:6iBF8bkt+roA4N5/1RvL7X4qP9M=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 640F58D7AAF536E3, x-amz-id-2: > > QY3YqAO4+652mNCPaIIqMe3tKTigUs+YlOj/vzKuRPJXiCD7DXEmziqdX/145Xov, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:12 GMT, Server: AmazonS3] > > 2011-11-27 10:11:14,023 DEBUG org.apache.hadoop.hbase.util.FSUtils: Created > > version file at s3://mcorner-hbase/data set its version at:7 > > 2011-11-27 10:11:14,281 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/block_-2255577201702834920' -- ResponseCode: 404, ResponseStatus: Not > > Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:14 > > GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:gICpmb7gELgfm2sW02O2OBQGSPw=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: DE383F7CAFC6856E, x-amz-id-2: > > y3ffN4iTYIkR+iOJI9uJkW8xntsdBZHq4miEsO1qz1YLdJlAlpBjNM30n2ASx9bs, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:13 GMT, Server: AmazonS3] > > 2011-11-27 10:11:14,836 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus: Not > > Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:14 > > GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:XbfWC0XVWS53IAEHGTNvkBZ9gQk=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 25AE3E24162F7850, x-amz-id-2: > > Q2oOyCd2NDQPiY2FjJPuI8pvhTid/dLyhFGQymF3SDvHkT0krDh79JerjxBt4FGY, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:14 GMT, Server: AmazonS3] > > 2011-11-27 10:11:14,838 INFO > > org.apache.hadoop.hbase.master.MasterFileSystem: BOOTSTRAP: creating ROOT > > and first META regions > > 2011-11-27 10:11:15,462 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-' -- ResponseCode: 404, ResponseStatus: Not Found, > > Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:15 GMT, > > Authorization: AWS AKIAJSHPUUTMSSRCOTBA:GdG0M+6O3+Cbls3DzXSfEIrISDw=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: AEEB8EFE863A9B5A, x-amz-id-2: > > P+wk8B2p3sRb/MR3uwG1jjH8BFwT6xHM3GXmoXjRwt/sc01NVNrJVUpr48j3Df4a, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:14 GMT, Server: AmazonS3] > > 2011-11-27 10:11:15,735 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus: Not > > Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:15 > > GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:qgFz5nvPmwHYLIKudvclcGJK2gw=, > > User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), > > Host: mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)], Response Headers: > > [x-amz-request-id: 7856EA5209B7D4A8, x-amz-id-2: > > lAQR13CX8ZfwSMKjl4SfErVtQPUwss5zCrcPg2tV5CpJ1bEmixNOpACIEVzwW/jf, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:14 GMT, Server: AmazonS3] > > 2011-11-27 10:11:16,024 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404, > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27 > > Nov 2011 15:11:15 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:go7R49fCUX01Gn5thKP4nl2emh0=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 9AB9F6F7CE9012CD, x-amz-id-2: > > GifVpi0k+F/se1gjESuUyFzUtSY93GyvLggqpWn6ZgxTMhwEw0YMa5jG1FTHhA33, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:14 GMT, Server: AmazonS3] > > 2011-11-27 10:11:17,636 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404, > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27 > > Nov 2011 15:11:16 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:pUj2QwlQa7ypigvoUs4LHikhDd0=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 5120CAE72D290EB7, x-amz-id-2: > > aCkMJu4F5FrvRAYRHac/CTxGsgSN3VgBfAEo/jVW0OeXaQNvZ09y6Qu8moV03fvp, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:16 GMT, Server: AmazonS3] > > 2011-11-27 10:11:17,926 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404, > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27 > > Nov 2011 15:11:17 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:8fAMMGghzJs6ICOfqy5Euf0nuC8=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: BEDBB9E931FEE069, x-amz-id-2: > > R9sMP4IkMOXsUCCzZErJYt9qGHhadkKstadB7pcpo0XKX9l4L+Clna4SPfZhNzyp, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:17 GMT, Server: AmazonS3] > > 2011-11-27 10:11:20,130 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404, > > ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27 > > Nov 2011 15:11:19 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:Hg1cqsBmprgya4GXSeAOVhVoju0=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 8752AD561C46847E, x-amz-id-2: > > 6bA9vjM4kLhQoL0uaY2hG30atmaYKYAXJaIb3utehSqXlIlH8UoN1KG0lnwe4UfR, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:18 GMT, Server: AmazonS3] > > 2011-11-27 10:11:20,307 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: > > HLog configuration: blocksize=32 MB, rollsize=30.4 MB, enabled=true, > > flushlogentries=1, optionallogflushinternal=1000ms > > 2011-11-27 10:11:20,403 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' -- > > ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:11:20 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:IaE+/XqX0aUmfo0LJw9O7pCsV1A=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: F84C1DC251EB6F09, x-amz-id-2: > > viG+7saKo8ZiPro1oWyNNqh8kAnve/5nddl35mETpf6udDScbbEY3ZCqV0pB7uJR, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:19 GMT, Server: AmazonS3] > > 2011-11-27 10:11:21,578 INFO > > org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs -- > > HDFS-200 -- not available, dfs.support.append=false > > 2011-11-27 10:11:21,837 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET > > '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' -- > > ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:11:21 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:5l0MMGpPTR3OCqmgji3W3bWoLgQ=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 9B1502504A01DCF4, x-amz-id-2: > > Kdzu/QRSHRgrrURTYiwijDgEIHcdZZZzGhleYsgsuLYlSEQ5D5WKCrhYhcDuZwHw, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:11:20 GMT, Server: AmazonS3] > > 2011-11-27 10:11:21,839 ERROR > > org.apache.hadoop.hbase.master.MasterFileSystem: bootstrap > > java.io.FileNotFoundException: > > s3://mcorner-hbase/data/-ROOT-/70236052/.logs/hlog.1322406680307: No such > > file or directory. > > at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334) > > at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:486) > > at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:374) > > > > 5) I have also tried creating the data directory in my s3 bucket, and that > > doesn't help. > > 2011-11-27 10:20:18,003 WARN > > org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD > > '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers: > > [Content-Type: , Date: Sun, 27 Nov 2011 15:20:15 GMT, Authorization: AWS > > AKIAJSHPUUTMSSRCOTBA:AqeMf6wmBZGJye7qwiYFrG6ozSA=, User-Agent: > > JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host: > > mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)], > > Response Headers: [x-amz-request-id: 168138BCA8FCBD1D, x-amz-id-2: > > o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko, > > Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27 > > Nov 2011 15:20:17 GMT, Server: AmazonS3] > > 2011-11-27 10:20:18,004 FATAL org.apache.hadoop.hbase.master.HMaster: > > Unhandled exception. Starting shutdown. > > org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: > > Request Error. HEAD '/data' on Host 'mcorner-hbase.s3.amazonaws.com > > (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:20:17 GMT' > > -- ResponseCode: 404, ResponseStatus: Not Found, RequestId: > > 168138BCA8FCBD1D, HostId: > > o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko > > > > > > > > What is the magic combination here?? Do you need to create the directory? > > What are you using for the rootdir url?? > > > > > > Mark Corner > > VP Engineering > > Mobile: +1.413.330.9601 > > Fiksu (http://www.fiksu.com) | LinkedIn > > (http://www.linkedin.com/in/mcorner) | Research > > (http://www.cs.umass.edu/~mcorner) > > > > >
