Hi, I am trying to use hbase with s3, using s3n, but I get the below errors, when starting the master. I am testing this in a pseudo distributed mode on my laptop. I've also set hbase.rootdir to s3n://kdiaa-hbase.s3-us-west-2.amazonaws.com:80/root, where the corresponding bucket and directory are already created on s3. I've also set fs.s3n.awsAccessKeyId, and fs.s3n.awsSecretAccessKey to the appropriate values in hbase-site.xml
So, I must be missing something. Any advice is appreciated. 2014-10-31 16:47:15,312 WARN [master:172.16.209.239:60000] httpclient.RestS3Service: Response '/root' - Unexpected response code 404, expected 200 2014-10-31 16:47:15,349 WARN [master:172.16.209.239:60000] httpclient.RestS3Service: Response '/root_%24folder%24' - Unexpected response code 404, expected 200 2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000] httpclient.RestS3Service: Response '/' - Unexpected response code 404, expected 200 2014-10-31 16:47:15,420 WARN [master:172.16.209.239:60000] httpclient.RestS3Service: Response '/' - Received error response with XML message 2014-10-31 16:47:15,601 FATAL [master:172.16.209.239:60000] master.HMaster: Unhandled exception. Starting shutdown. org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>kdiaa-hbase.s3-us-west-2.amazonaws.com</BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error> at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleServiceException(Jets3tNativeFileSystemStore.java:245) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:181) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:158) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:151) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at org.apache.hadoop.fs.s3native.$Proxy9.list(Unknown Source) at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:432) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1397) at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:439) at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:147) at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:128) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:802) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:615) at java.lang.Thread.run(Thread.java:695) Caused by: org.jets3t.service.S3ServiceException: S3 GET failed for '/' XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>kdiaa-hbase.s3-us-west-2.amazonaws.com</BucketName><RequestId>1589CC5DB70ED750</RequestId><HostId>cb2ZGGlNkxtf5fredweXt/wxJlAHLkioUJC86pkh0JxQfBJ1CMYoZuxHU1g+CnTB</HostId></Error> at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:424) at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRestGet(RestS3Service.java:686) at org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsInternal(RestS3Service.java:1083) at org.jets3t.service.impl.rest.httpclient.RestS3Service.listObjectsChunkedImpl(RestS3Service.java:1053) at org.jets3t.service.S3Service.listObjectsChunked(S3Service.java:1333) at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.list(Jets3tNativeFileSystemStore.java:168) ... 17 more 2014-10-31 16:47:15,604 INFO [master:172.16.209.239:60000] master.HMaster: Aborting 2014-10-31 16:47:15,612 DEBUG [master:172.16.209.239:60000] master.HMaster: Stopping service threads 2014-10-31 16:47:15,612 INFO [master:172.16.209.239:60000] ipc.RpcServer: Stopping server on 60000 2014-10-31 16:47:15,612 INFO [RpcServer.listener,port=60000] ipc.RpcServer: RpcServer.listener,port=60000: stopping 2014-10-31 16:47:15,619 INFO [master:172.16.209.239:60000] master.HMaster: Stopping infoServer 2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped 2014-10-31 16:47:15,633 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping 2014-10-31 16:47:15,660 INFO [master:172.16.209.239:60000] mortbay.log: Stopped [email protected]:60010 2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000] zookeeper.ZooKeeper: Session: 0x149689a7dd80000 closed 2014-10-31 16:47:15,804 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2014-10-31 16:47:15,804 INFO [master:172.16.209.239:60000] master.HMaster: HMaster main thread exiting 2014-10-31 16:47:15,804 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2803)
