Broken is the wrong word.
Incompatible is a more accurate summary.
Best regards,
- Andy
On Nov 27, 2011, at 8:16 PM, Mark Corner <[email protected]> wrote:
> Ok, but with those caveats, does it work at all?
>
> Or is it just 100% broken?
>
> --
> Mark Corner
>
>
> On Sunday, November 27, 2011 at 11:04 PM, Andrew Purtell wrote:
>
>> A couple points off the top of my head:
>>
>> - Consistency (or lack thereof). When HBase writes a file it expects to read
>> it back, not get a 500 response or a 404, depending. S3N makes this worse by
>> breaking files into chunks and adding a layer of indirection.
>>
>> - Streaming versus random reads: Unlike Hive, HBase reads are at best a mix
>> of short random reads and streaming reads of large files. At worst, mostly
>> the former. With HDFS we can ask for a few KB from the middle of a file and
>> get only that. With S3 the whole block must be read, of 64 MB, just for
>> those few bytes. Every time.
>>
>> - Response latency: With HDFS on the order of milliseconds. With S3, on the
>> order of seconds in practice.
>>
>> I could go on but pondering the above should be sufficient.
>>
>> Just because the Hadoop FS layer abstracts HDFS and S3 doesn't mean they are
>> remotely equivalent, and you should not expect to drop one in place of the
>> other without radically changing the resulting architecture.
>>
>> Best regards,
>>
>> - Andy
>>
>>
>> On Nov 27, 2011, at 7:24 PM, Mark Corner <[email protected]
>> (mailto:[email protected])> wrote:
>>
>>> Hrm, that sounds ominous...care to expand? It seems perfectly usable under
>>> Hive (we have done it). There are certainly indications in various forums
>>> and books that s3 is a slower, but perfectly usable DFS for HBase.
>>>
>>> I don't see anything in JIRA for HBase that says that s3 is broken, should
>>> I file a bug?
>>>
>>> --
>>> Mark Corner
>>>
>>>
>>> On Sunday, November 27, 2011 at 10:05 PM, Andrew Purtell wrote:
>>>
>>>> Don't do this. It will never work.
>>>>
>>>> Best regards,
>>>>
>>>> - Andy
>>>>
>>>>
>>>> On Nov 27, 2011, at 7:21 AM, Mark Corner <[email protected]
>>>> (mailto:[email protected])> wrote:
>>>>
>>>>> I am trying to get hbase to run directly over s3n (much preferable) or s3
>>>>> (less preferable). I ran into a few different problems that I solved(such
>>>>> as getting jets3t installed), but now i am stuck!! Sorry for the long
>>>>> post, but I wanted to include some error messages.
>>>>>
>>>>> I am on 0.90.3 installed on a Mac using homebrew. I have hadoop, but that
>>>>> doesn't come in to play as I am not using HDFS.
>>>>>
>>>>> The configuration I am using is below (a few replacements of keys for
>>>>> anonymity).
>>>>>
>>>>> <configuration>
>>>>>
>>>>> <property>
>>>>> <name>hbase.cluster.distributed</name>
>>>>> <value>true</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>hbase.rootdir</name>
>>>>> <value>s3n://mcorner-hbase/</value> (note below, I used different values
>>>>> for this)
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>fs.s3n.awsAccessKeyId</name>
>>>>> <value>mykeyid</value>
>>>>> </property>
>>>>>
>>>>> <property>
>>>>> <name>fs.s3n.awsSecretAccessKey</name>
>>>>> <value>mysecret</value>
>>>>> </property>
>>>>> </configuration>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 1) If I use this hbase.rootdir with s3n (this seems to be the preferred
>>>>> format…)
>>>>> <value>s3n://mcorner-hbase.s3.aws.com:80/data
>>>>> (http://mcorner-hbase.s3.aws.com:80/data)</value>
>>>>>
>>>>> then I get this failure.
>>>>> 2011-11-27 10:03:23,572 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:03:22 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:R0zegLCQbQyGxLVjPbzUobQMqHE=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 0BCBF072804E621A, x-amz-id-2:
>>>>> KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:03:22 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:03:23,573 FATAL org.apache.hadoop.hbase.master.HMaster:
>>>>> Unhandled exception. Starting shutdown.
>>>>> org.apache.hadoop.fs.s3.S3Exception:
>>>>> org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on
>>>>> Host 'mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011
>>>>> 15:03:22 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>>>> 0BCBF072804E621A, HostId:
>>>>> KHIjDEAcpDjGCXuEQ0c/V7hGr/iH5vqo8utMokZp+YCIw7U6SbVNem3RvtmBESCm
>>>>> at
>>>>> org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:122)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>
>>>>>
>>>>> So it looks like it is appending other things to the URL:
>>>>> "mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)"
>>>>>
>>>>> 2) If I replace the rootdir with:
>>>>> <value>s3n://mcorner-hbase:80/data</value>
>>>>> same problem:
>>>>> 2011-11-27 10:18:21,848 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:18:20 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:3tlyWfqbSkZB9lKQUCbQ8DW2VKk=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 2AFB15695EF68E13, x-amz-id-2:
>>>>> grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:18:21 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:18:21,849 FATAL org.apache.hadoop.hbase.master.HMaster:
>>>>> Unhandled exception. Starting shutdown.
>>>>> org.apache.hadoop.fs.s3.S3Exception:
>>>>> org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on
>>>>> Host 'mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:18:21
>>>>> GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>>>> 2AFB15695EF68E13, HostId:
>>>>> grz/Sfl7OWZEjSFd+MiiPe8he43g/LfZWrtt6zT3oOpg7DlWxqtyDq7vlCLUlcOy
>>>>>
>>>>>
>>>>> 3) If I switch to s3 (and change the key properties to match), with
>>>>> s3n://mcorner-hbase:80/data, then I get:
>>>>>
>>>>> 2011-11-27 10:08:55,951 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:08:54 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:nKXZEBy/WIinzNByPX7JKBasSVw=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: B8A5A618F8F86B87, x-amz-id-2:
>>>>> 7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:08:55 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:08:55,958 FATAL org.apache.hadoop.hbase.master.HMaster:
>>>>> Unhandled exception. Starting shutdown.
>>>>> org.apache.hadoop.fs.s3.S3Exception:
>>>>> org.jets3t.service.S3ServiceException: S3 Error Message. GET '/%2Fdata'
>>>>> on Host 'mcorner-hbase.s3.aws.com.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.aws.com.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011
>>>>> 15:08:55 GMT' -- ResponseCode: 404, ResponseStatus: Not Found, XML Error
>>>>> Message: <?xml version="1.0"
>>>>> encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified
>>>>> bucket does not exist</Message><BucketName>mcorner-hbase.s3.aws.com
>>>>> (http://mcorner-hbase.s3.aws.com)</BucketName><RequestId>B8A5A618F8F86B87</RequestId><HostId>7EbeGYqAaIC2XApd66aGsEHEq8cW4r2D1YycYSkLLk9WmnIX7HDPYhWpl+pm4u4B</HostId></Error>
>>>>>
>>>>>
>>>>>
>>>>> 4) The closest I get is with <value>s3://mcorner-hbase/data</value>. Then
>>>>> I get what appears to be more progress, but it appears to be creating
>>>>> files (I can see a file called "block_-222…." in my bucket, and a
>>>>> directory with a blank name, but the bootstrapping process eventually
>>>>> bombs out as you can se.
>>>>>
>>>>> 2011-11-27 10:11:12,793 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:11 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:rRR32CQb1xmqLv9PgEwOIVaTLMQ=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 2296D45E1390890C, x-amz-id-2:
>>>>> ZT9IpXdAIAqWu+owdtilWQWj+9nRQmH48IhOFoWajFhuRSEOcTplMjekzZa8wkXb,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:11 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:12,905 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2F' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:12 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:Be5+Q5QRILCtbJzUA6HcHEP9YZ8=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 6B6159CBC16B4742, x-amz-id-2:
>>>>> 4mBHDq4YdtrIBQS63x8zKjjpt2md2raM+LcXpIaGwybGrRPUjxOM53ZuYBab37FO,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:11 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:13,332 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:xPkvn6VtlerDOhZL604gwM3848c=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 46EC664A7B09DC6B, x-amz-id-2:
>>>>> 9mb0kcVcdjm86x69r9QDbN6cATm3nUKSPIfcxs7rs79M5CZ527z25F2Y3c6CzRdW,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:12 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:13,604 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2Fhbase.version' -- ResponseCode: 404, ResponseStatus: Not
>>>>> Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:13
>>>>> GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:6iBF8bkt+roA4N5/1RvL7X4qP9M=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 640F58D7AAF536E3, x-amz-id-2:
>>>>> QY3YqAO4+652mNCPaIIqMe3tKTigUs+YlOj/vzKuRPJXiCD7DXEmziqdX/145Xov,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:12 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:14,023 DEBUG org.apache.hadoop.hbase.util.FSUtils:
>>>>> Created version file at s3://mcorner-hbase/data set its version at:7
>>>>> 2011-11-27 10:11:14,281 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/block_-2255577201702834920' -- ResponseCode: 404, ResponseStatus:
>>>>> Not Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011
>>>>> 15:11:14 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:gICpmb7gELgfm2sW02O2OBQGSPw=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: DE383F7CAFC6856E, x-amz-id-2:
>>>>> y3ffN4iTYIkR+iOJI9uJkW8xntsdBZHq4miEsO1qz1YLdJlAlpBjNM30n2ASx9bs,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:13 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:14,836 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus:
>>>>> Not Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011
>>>>> 15:11:14 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:XbfWC0XVWS53IAEHGTNvkBZ9gQk=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 25AE3E24162F7850, x-amz-id-2:
>>>>> Q2oOyCd2NDQPiY2FjJPuI8pvhTid/dLyhFGQymF3SDvHkT0krDh79JerjxBt4FGY,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:14,838 INFO
>>>>> org.apache.hadoop.hbase.master.MasterFileSystem: BOOTSTRAP: creating ROOT
>>>>> and first META regions
>>>>> 2011-11-27 10:11:15,462 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-' -- ResponseCode: 404, ResponseStatus: Not Found,
>>>>> Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:11:15 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:GdG0M+6O3+Cbls3DzXSfEIrISDw=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: AEEB8EFE863A9B5A, x-amz-id-2:
>>>>> P+wk8B2p3sRb/MR3uwG1jjH8BFwT6xHM3GXmoXjRwt/sc01NVNrJVUpr48j3Df4a,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:15,735 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052' -- ResponseCode: 404, ResponseStatus:
>>>>> Not Found, Request Headers: [Content-Type: , Date: Sun, 27 Nov 2011
>>>>> 15:11:15 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:qgFz5nvPmwHYLIKudvclcGJK2gw=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 7856EA5209B7D4A8, x-amz-id-2:
>>>>> lAQR13CX8ZfwSMKjl4SfErVtQPUwss5zCrcPg2tV5CpJ1bEmixNOpACIEVzwW/jf,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:16,024 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404,
>>>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun,
>>>>> 27 Nov 2011 15:11:15 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:go7R49fCUX01Gn5thKP4nl2emh0=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 9AB9F6F7CE9012CD, x-amz-id-2:
>>>>> GifVpi0k+F/se1gjESuUyFzUtSY93GyvLggqpWn6ZgxTMhwEw0YMa5jG1FTHhA33,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:14 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:17,636 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs' -- ResponseCode: 404,
>>>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun,
>>>>> 27 Nov 2011 15:11:16 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:pUj2QwlQa7ypigvoUs4LHikhDd0=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 5120CAE72D290EB7, x-amz-id-2:
>>>>> aCkMJu4F5FrvRAYRHac/CTxGsgSN3VgBfAEo/jVW0OeXaQNvZ09y6Qu8moV03fvp,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:16 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:17,926 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404,
>>>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun,
>>>>> 27 Nov 2011 15:11:17 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:8fAMMGghzJs6ICOfqy5Euf0nuC8=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: BEDBB9E931FEE069, x-amz-id-2:
>>>>> R9sMP4IkMOXsUCCzZErJYt9qGHhadkKstadB7pcpo0XKX9l4L+Clna4SPfZhNzyp,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:17 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:20,130 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.oldlogs' -- ResponseCode: 404,
>>>>> ResponseStatus: Not Found, Request Headers: [Content-Type: , Date: Sun,
>>>>> 27 Nov 2011 15:11:19 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:Hg1cqsBmprgya4GXSeAOVhVoju0=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 8752AD561C46847E, x-amz-id-2:
>>>>> 6bA9vjM4kLhQoL0uaY2hG30atmaYKYAXJaIb3utehSqXlIlH8UoN1KG0lnwe4UfR,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:18 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:20,307 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.HLog: HLog configuration:
>>>>> blocksize=32 MB, rollsize=30.4 MB, enabled=true, flushlogentries=1,
>>>>> optionallogflushinternal=1000ms
>>>>> 2011-11-27 10:11:20,403 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' --
>>>>> ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:11:20 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:IaE+/XqX0aUmfo0LJw9O7pCsV1A=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: F84C1DC251EB6F09, x-amz-id-2:
>>>>> viG+7saKo8ZiPro1oWyNNqh8kAnve/5nddl35mETpf6udDScbbEY3ZCqV0pB7uJR,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:19 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:21,578 INFO
>>>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: syncFs --
>>>>> HDFS-200 -- not available, dfs.support.append=false
>>>>> 2011-11-27 10:11:21,837 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> GET '/%2Fdata%2F-ROOT-%2F70236052%2F.logs%2Fhlog.1322406680307' --
>>>>> ResponseCode: 404, ResponseStatus: Not Found, Request Headers:
>>>>> [Content-Type: , Date: Sun, 27 Nov 2011 15:11:21 GMT, Authorization: AWS
>>>>> AKIAJSHPUUTMSSRCOTBA:5l0MMGpPTR3OCqmgji3W3bWoLgQ=, User-Agent:
>>>>> JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26), Host:
>>>>> mcorner-hbase.s3.amazonaws.com (http://mcorner-hbase.s3.amazonaws.com)],
>>>>> Response Headers: [x-amz-request-id: 9B1502504A01DCF4, x-amz-id-2:
>>>>> Kdzu/QRSHRgrrURTYiwijDgEIHcdZZZzGhleYsgsuLYlSEQ5D5WKCrhYhcDuZwHw,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:11:20 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:11:21,839 ERROR
>>>>> org.apache.hadoop.hbase.master.MasterFileSystem: bootstrap
>>>>> java.io.FileNotFoundException:
>>>>> s3://mcorner-hbase/data/-ROOT-/70236052/.logs/hlog.1322406680307: No such
>>>>> file or directory.
>>>>> at
>>>>> org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:334)
>>>>> at org.apache.hadoop.hbase.regionserver.wal.HLog.rollWriter(HLog.java:486)
>>>>> at org.apache.hadoop.hbase.regionserver.wal.HLog.<init>(HLog.java:374)
>>>>>
>>>>> 5) I have also tried creating the data directory in my s3 bucket, and
>>>>> that doesn't help.
>>>>> 2011-11-27 10:20:18,003 WARN
>>>>> org.jets3t.service.impl.rest.httpclient.RestS3Service: Error Response:
>>>>> HEAD '/data' -- ResponseCode: 404, ResponseStatus: Not Found, Request
>>>>> Headers: [Content-Type: , Date: Sun, 27 Nov 2011 15:20:15 GMT,
>>>>> Authorization: AWS AKIAJSHPUUTMSSRCOTBA:AqeMf6wmBZGJye7qwiYFrG6ozSA=,
>>>>> User-Agent: JetS3t/0.7.4-dev (Mac OS X/10.7.2; x86_64; en; JVM 1.6.0_26),
>>>>> Host: mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)], Response Headers:
>>>>> [x-amz-request-id: 168138BCA8FCBD1D, x-amz-id-2:
>>>>> o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko,
>>>>> Content-Type: application/xml, Transfer-Encoding: chunked, Date: Sun, 27
>>>>> Nov 2011 15:20:17 GMT, Server: AmazonS3]
>>>>> 2011-11-27 10:20:18,004 FATAL org.apache.hadoop.hbase.master.HMaster:
>>>>> Unhandled exception. Starting shutdown.
>>>>> org.apache.hadoop.fs.s3.S3Exception:
>>>>> org.jets3t.service.S3ServiceException: Request Error. HEAD '/data' on
>>>>> Host 'mcorner-hbase.s3.amazonaws.com
>>>>> (http://mcorner-hbase.s3.amazonaws.com)' @ 'Sun, 27 Nov 2011 15:20:17
>>>>> GMT' -- ResponseCode: 404, ResponseStatus: Not Found, RequestId:
>>>>> 168138BCA8FCBD1D, HostId:
>>>>> o4gIuMOBc1Jf+Gw2NI2rWYCNBf/5uDVk6bL9TURfhkrTYGJ5JaZQc4J0Lq9k0jko
>>>>>
>>>>>
>>>>>
>>>>> What is the magic combination here?? Do you need to create the directory?
>>>>> What are you using for the rootdir url??
>>>>>
>>>>>
>>>>> Mark Corner
>>>>> VP Engineering
>>>>> Mobile: +1.413.330.9601
>>>>> Fiksu (http://www.fiksu.com) | LinkedIn
>>>>> (http://www.linkedin.com/in/mcorner) | Research
>>>>> (http://www.cs.umass.edu/~mcorner)
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
