A couple points off the top of my head:
- Consistency (or lack thereof). When HBase writes a file it expects to read it
back, not get a 500 response or a 404, depending. S3N makes this worse by
breaking files into chunks and adding a layer of indirection.
- Streaming versus random reads: Unlike Hive, HBase reads are at best a mix of
short random reads and streaming reads of large files. At worst, mostly the
former. With HDFS we can ask for a few KB from the middle of a file and get
only that. With S3 the whole block must be read, of 64 MB, just for those few
bytes. Every time.
- Response latency: With HDFS on the order of milliseconds. With S3, on the
order of seconds in practice.
I could go on but pondering the above should be sufficient.
Just because the Hadoop FS layer abstracts HDFS and S3 doesn't mean they are
remotely equivalent, and you should not expect to drop one in place of the
other without radically changing the resulting architecture.
Best regards,
- Andy
On Nov 27, 2011, at 7:24 PM, Mark Corner <[email protected]> wrote:
> Hrm, that sounds ominous...care to expand? It seems perfectly usable under
> Hive (we have done it). There are certainly indications in various forums
> and books that s3 is a slower, but perfectly usable DFS for HBase.
>
> I don't see anything in JIRA for HBase that says that s3 is broken, should I
> file a bug?
>
> --
> Mark Corner
>
>
> On Sunday, November 27, 2011 at 10:05 PM, Andrew Purtell wrote:
>
>> Don't do this. It will never work.
>>
>> Best regards,
>>
>> - Andy
>>
>>
>> On Nov 27, 2011, at 7:21 AM, Mark Corner <[email protected]
>> (mailto:[email protected])> wrote:
>>
>>> I am trying to get hbase to run directly over s3n (much preferable) or s3
>>> (less preferable). I ran into a few different problems that I solved(such
>>> as getting jets3t installed), but now i am stuck!! Sorry for the long post,
>>> but I wanted to include some error messages.
>>>
>>> I am on 0.90.3 installed on a Mac using homebrew. I have hadoop, but that
>>> doesn't come in to play as I am not using HDFS.
>>>
>>> The configuration I am using is below (a few replacements of keys for
>>> anonymity).
>>>
>>> <configuration>
>>>
>>> <property>
>>> <name>hbase.cluster.distributed</name>
>>> <value>true</value>
>>> </property>
>>>
>>> <property>
>>> <name>hbase.rootdir</name>
>>> <value>s3n://mcorner-hbase/</value> (note below, I used different values
>>> for this)
>>> </property>
>>>
>>> <property>
>>> <name>fs.s3n.awsAccessKeyId</name>
>>> <value>mykeyid</value>
>>> </property>
>>>
>>> <property>
>>> <name>fs.s3n.awsSecretAccessKey</name>
>>> <value>mysecret</value>
>>> </property>
>>> </configuration>
>>>
>>>
>>>
>>>
>>>
>>> 1) If I use this hbase.rootdir with s3n (this seems to be the preferred
>>> format…)
>>> <value>s3n://mcorner-hbase.s3.aws.com:80/data
>>> (http://mcorner-hbase.s3.aws.com:80/data)</value>
>>>
>>> then I get this failure.
>>> 2011-11-27 10:03:23,572 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD
>>> '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:03:22 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:R0zegLCQbQyGxLVjPbzUobQMqHE=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 0BCBF072804E621A, x-amz-id-2:
>>> KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:03:22 GMT, Server: AmazonS3]
>>> 2011-11-27 10:03:23,573 FATAL org.apache.hadoop.hbase.master.HMaster:
>>> Unhandled exception. Starting shutdown.
>>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
>>> Request Error. HEAD '/data' on Host
>>> 'mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011
>>> 15:03:22 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>> 0BCBF072804E621A, HostId:
>>> KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm
>>> at
>>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:122)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>
>>>
>>> So it looks like it is appending other things to the URL:
>>> "mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)"
>>>
>>> 2) If I replace the rootdir with:
>>> <value>s3n://mcorner-hbase:80/data</value>
>>> same problem:
>>> 2011-11-27 10:18:21,848 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD
>>> '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:18:20 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:3tlyWfqbSkZB9lKQUCbQ8DW2VKk=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 2AFB15695EF68E13, x-amz-id-2:
>>> grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:18:21 GMT, Server: AmazonS3]
>>> 2011-11-27 10:18:21,849 FATAL org.apache.hadoop.hbase.master.HMaster:
>>> Unhandled exception. Starting shutdown.
>>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
>>> Request Error. HEAD '/data' on Host 'mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:18:21 GMT'
>>> -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>> 2AFB15695EF68E13, HostId:
>>> grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy
>>>
>>>
>>> 3) If I switch to s3 (and change the key properties to match), with
>>> s3n://mcorner-hbase:80/data, then I get:
>>>
>>> 2011-11-27 10:08:55,951 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:08:54 GMT,
>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:nKXZEBy/WIinzNByPX7JKBasSVw=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: B8A5A618F8F86B87, x-amz-id-2:
>>> 7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:08:55 GMT, Server: AmazonS3]
>>> 2011-11-27 10:08:55,958 FATAL org.apache.hadoop.hbase.master.HMaster:
>>> Unhandled exception. Starting shutdown.
>>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
>>> S3 Error Message. GET '/%2Fdata' on Host
>>> 'mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011
>>> 15:08:55 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML Error
>>> Message: <?xml version="1.0"
>>> encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified
>>> bucket does not exist</Message><BucketName>mcorner-hbase.s3.aws.com
>>> (http://mcorner-hbase.s3.aws.com)</BucketName><RequestId>B8A5A618F8F86B87</RequestId><HostId>7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B</HostId></Error>
>>>
>>>
>>>
>>> 4) The closest I get is with <value>s3://mcorner-hbase/data</value>. Then I
>>> get what appears to be more progress, but it appears to be creating files
>>> (I can see a file called "block_-222…." in my bucket, and a directory with
>>> a blank name, but the bootstrapping process eventually bombs out as you can
>>> se.
>>>
>>> 2011-11-27 10:11:12,793 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:11 GMT,
>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:rRR32CQb1xmqLv9PgEwOIVaTLMQ=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 2296D45E1390890C, x-amz-id-2:
>>> ZT9IpXdAIAqWu+owdtilWQWj+9nRQmH48IhOFoWajFhuRSEOcTplMjekzZa8wkXb,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:11 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:12,905 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2F' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:11:12 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:Be5+Q5QRILCtbJzUA6HcHEP9YZ8=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 6B6159CBC16B4742, x-amz-id-2:
>>> 4mBHDq4YdtrIBQS63x8zKjjpt2md2raM+LcXpIaGwybGrRPUjxOM53ZuYBab37FO,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:11 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:13,332 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT,
>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:xPkvn6VtlerDOhZL604gwM3848c=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 46EC664A7B09DC6B, x-amz-id-2:
>>> 9mb0kcVcdjm86x69r9QDbN6cATm3nUKSPIfcxs7rs79M5CZ527z25F2Y3c6CzRdW,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:12 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:13,604 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2Fhbase.version' -- ResponseCode: 404, ResponseStatus: Not Found,
>>> Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT,
>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:6iBF8bkt+roA4N5/1RvL7X4qP9M=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 640F58D7AAF536E3, x-amz-id-2:
>>> QY3YqAO4+652mNCPaIIqMe3tKTigUs+YlOj/vzKuRPJXiCD7DXEmziqdX/145Xov,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:12 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:14,023 DEBUG org.apache.hadoop.hbase.util.FSUtils: Created
>>> version file at s3://mcorner-hbase/data set its version at:7
>>> 2011-11-27 10:11:14,281 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/block_-2255577201702834920' -- ResponseCode: 404, ResponseStatus: Not
>>> Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:14
>>> GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:gICpmb7gELgfm2sW02O2OBQGSPw=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: DE383F7CAFC6856E, x-amz-id-2:
>>> y3ffN4iTYIkR+iOJI9uJkW8xntsdBZHq4miEsO1qz1YLdJlAlpBjNM30n2ASx9bs,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:13 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:14,836 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus: Not
>>> Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:14
>>> GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:XbfWC0XVWS53IAEHGTNvkBZ9gQk=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 25AE3E24162F7850, x-amz-id-2:
>>> Q2oOyCd2NDQPiY2FjJPuI8pvhTid/dLyhFGQymF3SDvHkT0krDh79JerjxBt4FGY,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:14,838 INFO
>>> org.apache.hadoop.hbase.master.MasterFileSystem: BOOTSTRAP: creating ROOT
>>> and first META regions
>>> 2011-11-27 10:11:15,462 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-' -- ResponseCode: 404, ResponseStatus: Not Found,
>>> Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:15 GMT,
>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:GdG0M+6O3+Cbls3DzXSfEIrISDw=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: AEEB8EFE863A9B5A, x-amz-id-2:
>>> P+wk8B2p3sRb/MR3uwG1jjH8BFwT6xHM3GXmoXjRwt/sc01NVNrJVUpr48j3Df4a,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:15,735 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus: Not
>>> Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:15
>>> GMT, Authorization: AWS AKIAJSHPUUTMSSRCOTBA:qgFz5nvPmwHYLIKudvclcGJK2gw=,
>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>> Host: mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>> [x-amz-request-id: 7856EA5209B7D4A8, x-amz-id-2:
>>> lAQR13CX8ZfwSMKjl4SfErVtQPUwss5zCrcPg2tV5CpJ1bEmixNOpACIEVzwW/jf,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:16,024 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404,
>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27
>>> Nov 2011 15:11:15 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:go7R49fCUX01Gn5thKP4nl2emh0=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 9AB9F6F7CE9012CD, x-amz-id-2:
>>> GifVpi0k+F/se1gjESuUyFzUtSY93GyvLggqpWn6ZgxTMhwEw0YMa5jG1FTHhA33,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:17,636 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404,
>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27
>>> Nov 2011 15:11:16 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:pUj2QwlQa7ypigvoUs4LHikhDd0=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 5120CAE72D290EB7, x-amz-id-2:
>>> aCkMJu4F5FrvRAYRHac/CTxGsgSN3VgBfAEo/jVW0OeXaQNvZ09y6Qu8moV03fvp,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:16 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:17,926 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404,
>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27
>>> Nov 2011 15:11:17 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:8fAMMGghzJs6ICOfqy5Euf0nuC8=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: BEDBB9E931FEE069, x-amz-id-2:
>>> R9sMP4IkMOXsUCCzZErJYt9qGHhadkKstadB7pcpo0XKX9l4L+Clna4SPfZhNzyp,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:17 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:20,130 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404,
>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun, 27
>>> Nov 2011 15:11:19 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:Hg1cqsBmprgya4GXSeAOVhVoju0=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 8752AD561C46847E, x-amz-id-2:
>>> 6bA9vjM4kLhQoL0uaY2hG30atmaYKYAXJaIb3utehSqXlIlH8UoN1KG0lnwe4UfR,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:18 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:20,307 INFO org.apache.hadoop.hbase.regionserver.wal.HLog:
>>> HLog configuration: blocksize=32 MB, rollsize=30.4 MB, enabled=true,
>>> flushlogentries=1, optionallogflushinternal=1000ms
>>> 2011-11-27 10:11:20,403 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' --
>>> ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:11:20 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:IaE+/XqX0aUmfo0LJw9O7pCsV1A=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: F84C1DC251EB6F09, x-amz-id-2:
>>> viG+7saKo8ZiPro1oWyNNqh8kAnve/5nddl35mETpf6udDScbbEY3ZCqV0pB7uJR,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:19 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:21,578 INFO
>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs --
>>> HDFS-200 -- not available, dfs.support.append=false
>>> 2011-11-27 10:11:21,837 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: GET
>>> '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' --
>>> ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:11:21 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:5l0MMGpPTR3OCqmgji3W3bWoLgQ=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 9B1502504A01DCF4, x-amz-id-2:
>>> Kdzu/QRSHRgrrURTYiwijDgEIHcdZZZzGhleYsgsuLYlSEQ5D5WKCrhYhcDuZwHw,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:11:20 GMT, Server: AmazonS3]
>>> 2011-11-27 10:11:21,839 ERROR
>>> org.apache.hadoop.hbase.master.MasterFileSystem: bootstrap
>>> java.io.FileNotFoundException:
>>> s3://mcorner-hbase/data/-ROOT-/70236052/.logs/hlog.1322406680307: No such
>>> file or directory.
>>> at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
>>> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:486)
>>> at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:374)
>>>
>>> 5) I have also tried creating the data directory in my s3 bucket, and that
>>> doesn't help.
>>> 2011-11-27 10:20:18,003 WARN
>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response: HEAD
>>> '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:20:15 GMT, Authorization: AWS
>>> AKIAJSHPUUTMSSRCOTBA:AqeMf6wmBZGJye7qwiYFrG6ozSA=, User-Agent:
>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>> Response Headers: [x-amz-request-id: 168138BCA8FCBD1D, x-amz-id-2:
>>> o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko,
>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>> Nov 2011 15:20:17 GMT, Server: AmazonS3]
>>> 2011-11-27 10:20:18,004 FATAL org.apache.hadoop.hbase.master.HMaster:
>>> Unhandled exception. Starting shutdown.
>>> org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
>>> Request Error. HEAD '/data' on Host 'mcorner-hbase.s3.amazonaws.com
>>> (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:20:17 GMT'
>>> -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>> 168138BCA8FCBD1D, HostId:
>>> o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko
>>>
>>>
>>>
>>> What is the magic combination here?? Do you need to create the directory?
>>> What are you using for the rootdir url??
>>>
>>>
>>> Mark Corner
>>> VP Engineering
>>> Mobile: +1.413.330.9601
>>> Fiksu (http://www.fiksu.com) | LinkedIn
>>> (http://www.linkedin.com/in/mcorner) | Research
>>> (http://www.cs.umass.edu/~mcorner)
>>>
>>
>>
>>
>
>
