I have reported a bug - https://issues.apache.org/jira/browse/CLOUDSTACK-1807
Now, when I create a bucket using the S3 AWS API on cloudstack I get > error as - OutOfStorageException: Unable to create /s3storage/test123 > for bucket test123 Stacktrace is- > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header x-amz-date:Tue, 26 Mar 2013 > 11:13:42 GMT > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header > host:torvm-cloudstack-mgmt.sigmasys.net:7080 > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header content-length:0 > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header connection:Keep-Alive > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) - End of request - > 2013-03-26 07:13:47,800 DEBUG [bridge.service.UserContext] > (catalina-exec-int-22:null) initializing a new [anonymous] UserContext! > 2013-03-26 07:13:47,815 ERROR [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Unexpected exception Unable to create > /s3storage/test123 for bucket test123 > com.cloud.bridge.service.exception.OutOfStorageException: Unable to create > /s3storage/test123 for bucket test123 > at > com.cloud.bridge.io.S3FileSystemBucketAdapter.createContainer(S3FileSystemBucketAdapter.java:55) > at > com.cloud.bridge.service.core.s3.S3Engine.allocBucketStorageHost(S3Engine.java:1466) > at > com.cloud.bridge.service.core.s3.S3Engine.handleRequest(S3Engine.java:210) > at > com.cloud.bridge.service.controller.s3.S3BucketAction.executePutBucket(S3BucketAction.java:837) > at > com.cloud.bridge.service.controller.s3.S3BucketAction.execute(S3BucketAction.java:159) > at > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet.java:181) > at com.cloud.bridge.service.S3RestServlet.doPut(S3RestServlet.java:97) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:640) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > What could be the reason? Thanks and Regards. Asmita -----Original Message----- From: Sebastien Goasguen [mailto:run...@gmail.com] Sent: 26 March 2013 PM 04:57 To: Asmita Vagyani Subject: Re: CS4 AWS S3 support On Mar 26, 2013, at 7:19 AM, Asmita Vagyani <asmita.vagy...@sigma-systems.com> wrote: > There is no bucket created on my NFS mount. > This is not an error specific to S3 browser tool.Even if I use this url in > the IE browser, it gives 400-Bad request error. > > When I give url as > http://torvm-cloudstack-mgmt.sigmasys.net:7080/awsapi/rest/AmazonS3/te > st2342 > It gives error like : > <Error> > <Code>NoSuchBucket</Code> > <Message>The specified bucket does not exist</Message> > <BucketName>test2342</BucketName> <RequestId>1DEADBEEF9</RequestId> > <HostId>abCdeFgHiJ1k2LmN3op4q56r7st89</HostId> > </Error> > > This is a normal behavior. Same behavior I should get when I don't pass any > name of bucket. >>>>>>>>>>>>>> 1.....Please check if this is a bug in CS4. if you think it's a bug, then please open a bug in jira: https://issues.apache.org/jira/browse/CLOUDSTACK > > Now, when I create a bucket using the S3 AWS API on cloudstack I get > error as - OutOfStorageException: Unable to create /s3storage/test123 > for bucket test123 Stacktrace is- > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header x-amz-date:Tue, 26 Mar 2013 > 11:13:42 GMT > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header > host:torvm-cloudstack-mgmt.sigmasys.net:7080 > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header content-length:0 > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Request header connection:Keep-Alive > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) - End of request - > 2013-03-26 07:13:47,800 DEBUG [bridge.service.UserContext] > (catalina-exec-int-22:null) initializing a new [anonymous] UserContext! > 2013-03-26 07:13:47,815 ERROR [bridge.service.S3RestServlet] > (catalina-exec-int-22:null) Unexpected exception Unable to create > /s3storage/test123 for bucket test123 > com.cloud.bridge.service.exception.OutOfStorageException: Unable to create > /s3storage/test123 for bucket test123 > at > com.cloud.bridge.io.S3FileSystemBucketAdapter.createContainer(S3FileSystemBucketAdapter.java:55) > at > com.cloud.bridge.service.core.s3.S3Engine.allocBucketStorageHost(S3Engine.java:1466) > at > com.cloud.bridge.service.core.s3.S3Engine.handleRequest(S3Engine.java:210) > at > com.cloud.bridge.service.controller.s3.S3BucketAction.executePutBucket(S3BucketAction.java:837) > at > com.cloud.bridge.service.controller.s3.S3BucketAction.execute(S3BucketAction.java:159) > at > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet.java:181) > at com.cloud.bridge.service.S3RestServlet.doPut(S3RestServlet.java:97) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:640) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > > > What could be the reason? > > > > Thanks and Regards. > > Asmita > > > -----Original Message----- > From: Sebastien Goasguen [mailto:run...@gmail.com] > Sent: 26 March 2013 PM 04:33 > To: Asmita Vagyani > Subject: Re: CS4 AWS S3 support > > > On Mar 26, 2013, at 6:56 AM, Asmita Vagyani > <asmita.vagy...@sigma-systems.com> wrote: > >> Hi Sebastien, >> >> Thanks , after giving the FQDN instead of IP address, it works. >> Now, atleast the S3 browser is able to invoke the cloudstack S3 AWS REST API. >> >> I saw in the documentation >> https://cwiki.apache.org/CLOUDSTACK/s3-api-in-cloudstack.html >> It says to List All Buckets, simply make a call to : >> REST GET request, http://myhost:8080/awsapi/rest/AmazonS3 >> >> However, when I invoke >> http://torvm-cloudstack-mgmt.sigmasys.net:7080/awsapi/rest/AmazonS3 >> I get the 400 Bad request error on browser. >> I get the exception at backend when I checked awsapi.log: >> > > I have never used the S3 browser but it looks like you may have some existing > buckets in there that cause the error. > > >> 2013-03-26 06:49:32,080 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Request servletPath: /rest/AmazonS3 >> 2013-03-26 06:49:32,080 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Request header user-agent:S3 Browser3-8-1 >> 2013-03-26 06:49:32,081 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Request header authorization:AWS >> Qh69lnbtbr0H68i_Q7im7eZNhoFYUyHS8eDU9nyL2k3OP0gBGN0gNHfHWeKQo1EZq2r0M >> M ZK-23U9_wF6PMojg:RWAiwif28JxZ/le9mDkQskfPAd8= >> 2013-03-26 06:49:32,081 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Request header x-amz-date:Tue, 26 Mar >> 2013 >> 10:49:26 GMT >> 2013-03-26 06:49:32,081 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Request header >> host:torvm-cloudstack-mgmt.sigmasys.net:7080 >> 2013-03-26 06:49:32,081 INFO [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) - End of request - >> 2013-03-26 06:49:32,093 DEBUG [bridge.service.UserContext] >> (catalina-exec-int-20:null) initializing a new [anonymous] UserContext! >> 2013-03-26 06:49:32,093 ERROR [bridge.service.S3RestServlet] >> (catalina-exec-int-20:null) Unexpected exception is not between 3 >> and >> 255 characters long >> com.cloud.bridge.service.exception.InvalidBucketName: is not between 3 and >> 255 characters long >> at >> com.cloud.bridge.service.core.s3.S3Engine.verifyBucketName(S3Engine.java:1809) >> at >> com.cloud.bridge.service.S3RestServlet.routeRequest(S3RestServlet.java:440) >> at >> com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet.java:179) >> at com.cloud.bridge.service.S3RestServlet.doGet(S3RestServlet.java:84) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) >> at >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) >> at >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> at >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> at >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> at >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >> at >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> at >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) >> at >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> at >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> at >> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889) >> at >> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721) >> at >> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> at java.lang.Thread.run(Thread.java:722) >> >> Thanks and Regards. >> >> Asmita >> >> -----Original Message----- >> From: Sebastien Goasguen [mailto:run...@gmail.com] >> Sent: 26 March 2013 PM 04:06 >> To: Asmita Vagyani >> Subject: Re: CS4 AWS S3 support >> >> >> On Mar 26, 2013, at 6:25 AM, Asmita Vagyani >> <asmita.vagy...@sigma-systems.com> wrote: >> >>> Hi >>> Where do I register the DNS entry I mean do you mean a local DNS entry or >>> what? >> >> Local entry would do, as long as your "s3 browser" knows how to route >> <server.com> to the IP of the endpoint. >> >>> >>> Thanks and Regards. >>> >>> Asmita >>> >>> -----Original Message----- >>> From: Sebastien Goasguen [mailto:run...@gmail.com] >>> Sent: 26 March 2013 PM 03:34 >>> To: Asmita Vagyani >>> Subject: Re: CS4 AWS S3 support >>> >>> >>> On Mar 26, 2013, at 4:40 AM, Asmita Vagyani >>> <asmita.vagy...@sigma-systems.com> wrote: >>> >>>> Hi all, >>>> >>>> I tried to use a couple of freewares/tools like CyberDuck, DragonDisk, etc >>>> which interfaces the Cloudstack S3 APIs. >>>> I am stuck at one common point in all these tools. >>>> All tools say they support the alternative S3 compatible services, which >>>> in our case is our Cloudstack AWS S3. >>>> However, it asks me endpoint of the S3 compatible service. >>>> When I mention the endpoint as >>>> http://<cloudstack-server-IP:port>/awsapi/rest/AmazonS3 it doesnot accept >>>> it. >>>> It wants the server name as server.com. >>>> >>> >>> Asmita, I woud do two things: >>> >>> 1-Try to use boto like I explained in the slides, that should show you if >>> the S3 service is working properly or not. >>> 2-If the S3 tools that you are using do not take an IP, then you will need >>> to register a dns entry. >>> >>> note that the S3 support is really a tech preview. >>> >>> -sebastien >>> >>>> Is there any way I can get the S3 webservice endpoint the way these tools >>>> want. >>>> Is this because Cloudstack is not consistent with the EC2 endpoint. >>>> If yes, when are we planning to make it consistent. >>>> >>>> >>>> Thanks and Regards. >>>> >>>> Asmita >>>> >>>> -----Original Message----- >>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] >>>> Sent: 25 March 2013 PM 09:40 >>>> To: users@cloudstack.apache.org; 'cloudstack-us...@incubator.apache.org'; >>>> 'run...@gmail.com' >>>> Subject: RE: CS4 AWS S3 support >>>> >>>> Am not sure whether s3 browser allows user to configure the settings to >>>> access the cloudstack s3. >>>> If you can configure the settings the way we configure for boto then we >>>> can use s3 browser. >>>> >>>> Thanks >>>> Rajesh Battala >>>> >>>>> -----Original Message----- >>>>> From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] >>>>> Sent: Monday, March 25, 2013 7:09 PM >>>>> To: 'users@cloudstack.apache.org'; >>>>> 'cloudstack-us...@incubator.apache.org'; >>>>> 'run...@gmail.com' >>>>> Subject: RE: CS4 AWS S3 support >>>>> >>>>> Are there any other clients, non-python based. >>>>> Can I integrate this with S3 browser, such that the S3 browser >>>>> shows me buckets created on the NFS drive mounted using >>>>> cloud-bridge.properties? >>>>> User can just upload and download files in S3 browser and >>>>> ultimately on the NFS drive buckets. >>>>> >>>>> >>>>> Thanks and Regards. >>>>> >>>>> Asmita >>>>> >>>>> -----Original Message----- >>>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] >>>>> Sent: 25 March 2013 PM 04:42 >>>>> To: users@cloudstack.apache.org; >>>>> 'cloudstack-us...@incubator.apache.org'; >>>>> 'run...@gmail.com' >>>>> Subject: RE: CS4 AWS S3 support >>>>> >>>>> Asmita, >>>>> You can follow this link. >>>>> https://cwiki.apache.org/CLOUDSTACK/s3-api-in- >>>>> cloudstack.html >>>>> >>>>> You can use boto client which is python based to send s3 requests to CS. >>>>> >>>>> Thanks >>>>> Rajesh Battala >>>>> >>>>>> -----Original Message----- >>>>>> From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] >>>>>> Sent: Monday, March 25, 2013 4:13 PM >>>>>> To: 'users@cloudstack.apache.org'; >>>>>> 'cloudstack-us...@incubator.apache.org'; >>>>>> 'run...@gmail.com' >>>>>> Subject: RE: CS4 AWS S3 support >>>>>> >>>>>> Hi, >>>>>> >>>>>> I can see the entry of keys in the usercredential table in CS4 DB. >>>>>> When I access http://<server-ip>/awsapi link I get the servlet exception >>>>>> : >>>>>> >>>>>> java.lang.RuntimeException: javax.servlet.ServletException: >>>>>> Servlet execution threw an exception >>>>>> >>>>>> com.cloud.bridge.service.EC2MainServlet.doGetOrPost(EC2MainServle >>>>>> t >>>>>> .java:104) >>>>>> >>>>>> com.cloud.bridge.service.EC2MainServlet.doGet(EC2MainServlet.java: >>>>>> 7 >>>>>> 3) >>>>>> >>>>>> How can I run S3 requests? any links to documentation would be helpful. >>>>>> >>>>>> Thanks and Regards. >>>>>> >>>>>> Asmita >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] >>>>>> Sent: 25 March 2013 PM 03:35 >>>>>> To: users@cloudstack.apache.org; >>>>>> 'cloudstack-us...@incubator.apache.org'; >>>>>> 'run...@gmail.com' >>>>>> Subject: RE: CS4 AWS S3 support >>>>>> >>>>>> Hi Asmita, >>>>>> >>>>>> In the cloudbridge db, usercredentials table has the >>>>>> username/password values present? >>>>>> Generally, for an existing account, generate the apikey/secrect >>>>>> key and register them in awsapi. >>>>>> >>>>>> Were you able to run s3 requests.? Is awsap is running at 7080 port? >>>>>> >>>>>> Thanks >>>>>> Rajesh Battala >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] >>>>>>> Sent: Monday, March 25, 2013 3:24 PM >>>>>>> To: 'cloudstack-us...@incubator.apache.org'; 'run...@gmail.com' >>>>>>> Subject: CS4 AWS S3 support >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am trying to do following to have an implementation of S3 in >>>>>>> our >>>>> application. >>>>>>> Followed steps given in: >>>>>>> http://www.slideshare.net/sebastiengoasguen/cloudstack-s3 >>>>>>> >>>>>>> >>>>>>> 1) I have enabled S3 support and EC2 support flag in the CS4 global >>>>>> variables. >>>>>>> >>>>>>> 2) I have mounted the NFS store on the CS4 management server (I am >>>>> using >>>>>>> the NFS drive to store the buckets/files uploaded by user on >>>>>>> this >>>>>>> drive) >>>>>>> >>>>>>> 3) I have configured the cloud-bridge.properties appropriately >>>>>>> >>>>>>> host=http://localhost:7080/awsapi >>>>>>> >>>>>>> storage.root=/s3storage >>>>>>> >>>>>>> storage.multipartDir=__multipart__uploads__ >>>>>>> >>>>>>> bucket.dns=false >>>>>>> >>>>>>> serviceEndpoint=localhost:7080 >>>>>>> >>>>>>> >>>>>>> >>>>>>> 4) I made up a user account into CS4 client console, generated the >>>>>>> API key >>>>>>> and the Secret key. >>>>>>> >>>>>>> 5) I have registered the user by using cloudstack-aws-api-register >>>>>>> (refer - >>>>>>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.1- >>>>>>> incubating/html/Installation_Guide/aws-ec2-user-setup.html ) >>>>>>> >>>>>>> 6) I have downloaded the S3 browser. >>>>>>> >>>>>>> 7) I am trying to make an account with the same user API key and >>>>>>> Secret >>>>> key >>>>>>> which we create on Cloudstack. >>>>>>> >>>>>>> While creating an account it says - InvalidAccesskeyId >>>>>>> >>>>>>> Any hints if I am missing some step here? >>>>>>> >>>>>>> Thanks and Regards. >>>>>>> >>>>>>> Asmita Patil Vagyani. >>>>>>> >>>>>>> Tech Lead, Sigma Software Solutions, Pune, >>>>>>> >>>>>>> T: +91-20-25661992 / 93 / 94. Extn. : 286 >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > >
