By when do you think Cloudstack 4 will be supporting S3 AWS APIs fully? Which version of Cloudstack? And I have raised JIRA defects https://issues.apache.org/jira/browse/CLOUDSTACK-1821 https://issues.apache.org/jira/browse/CLOUDSTACK-1807
Can I get some fix for this or work-around. Thanks and Regards. Asmita -----Original Message----- From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] Sent: 27 March 2013 PM 01:24 To: users@cloudstack.apache.org Subject: RE: CS4 AWS S3 support I guess the request don't have the credential information so its failing. > -----Original Message----- > From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] > Sent: Wednesday, March 27, 2013 12:30 PM > To: 'users@cloudstack.apache.org' > Subject: RE: CS4 AWS S3 support > > Want to know , what is the expected output on the browser, if I try to > access > http://torvm-cloudstack-mgmt.sigmasys.net:7080/awsapi/rest/AmazonS3 > > I get the bad request 400 error on browser. Is this the expected behaviour? > Am I missing something in the S3 AWS setting on Cloudstack? > > awsapi.log gives error: > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header user-agent:Mozilla/5.0 (Windows NT 6.1; > WOW64) > AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 > Safari/537.22 > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header accept-encoding:gzip,deflate,sdch > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header accept-language:en-US,en;q=0.8 > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header > accept-charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3 > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header via:1.1 squid (squid/3.1.16) > 2013-03-27 02:58:33,221 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header x-forwarded-for:10.100.20.174 > 2013-03-27 02:58:33,222 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header cache-control:max-age=259200 > 2013-03-27 02:58:33,222 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Request header connection:keep-alive > 2013-03-27 02:58:33,222 INFO [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) - End of request - > 2013-03-27 02:58:33,222 DEBUG [bridge.service.UserContext] > (catalina-exec- > int-22:null) initializing a new [anonymous] UserContext! > 2013-03-27 02:58:33,222 ERROR [bridge.service.S3RestServlet] > (catalina-exec- > int-22:null) Unexpected exception null java.lang.NullPointerException > at > com.cloud.bridge.service.S3RestServlet.routeRequest(S3RestServlet.java:405) > at > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet.java:179) > at com.cloud.bridge.service.S3RestServlet.doGet(S3RestServlet.java:84) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli > cationFilte > rChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi > lterChain.j > ava:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa > lve > .java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa > lve.j > ava:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja > va:12 > 7) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv > e.jav > a:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor > .jav > a:889) > at > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.pro > ce > ss(Http11NioProtocol.java:721) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint > .java > :2268) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > ava:1 > 110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > 603) > at java.lang.Thread.run(Thread.java:722) > > > Thanks and Regards. > > Asmita > > -----Original Message----- > From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > Sent: 27 March 2013 AM 10:31 > To: users@cloudstack.apache.org > Subject: RE: CS4 AWS S3 support > > Asmita, > Are you using S3 from source or from the build? > Can you report when are you facing the issues with ACL's? > > Thanks > Rajesh Battala > > > -----Original Message----- > > From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] > > Sent: Wednesday, March 27, 2013 10:16 AM > > To: 'users@cloudstack.apache.org' > > Subject: RE: CS4 AWS S3 support > > > > Hi Rajesh, > > > > Is there any work-around to avoid the ACL permission issues? > > > > Thanks and Regards. > > > > Asmita > > > > -----Original Message----- > > From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > Sent: 26 March 2013 PM 07:47 > > To: users@cloudstack.apache.org; 'Sebastien Goasguen' > > Subject: RE: CS4 AWS S3 support > > > > Good to know the issue is resolve. As I had already told it's > > because of the permission issue. > > > > There are few bugs related to ACL's. feel free to file bugs against S3. > > I had run all the boto unit tests, most of them were working fine. > > > > > > Thanks > > Rajesh Battala > > > > > -----Original Message----- > > > From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] > > > Sent: Tuesday, March 26, 2013 6:52 PM > > > To: 'users@cloudstack.apache.org'; 'Sebastien Goasguen' > > > Subject: RE: CS4 AWS S3 support > > > > > > Issue is resolved now. > > > I used set gid on the /s3storage folder and then it allowed to create > > > buckets. > > > Now the bucket gets created in mount point folder, but the owner > > > of folder is nobody user. > > > > > > Now when I use: > > > http://<server-ip>:7080/awsapi/rest/AmazonS3/test12323 > > > This should give me details abt the bucket. > > > > > > I get following error on IE browser: Access denied - > > > com.cloud.bridge.service.exception.PermissionDeniedException: > > > Access Denied > > > - ACLs do not give user the required permission What is the issue? > > > > > > Logs show- > > > com.cloud.bridge.service.exception.PermissionDeniedException: > > > Access Denied > > > - ACLs do not give user the required permission > > > at > > > > > com.cloud.bridge.service.core.s3.S3Engine.accessAllowed(S3Engine.java: > > 1762) > > > at > > > com.cloud.bridge.service.core.s3.S3Engine.verifyAccess(S3Engine.java:1729) > > > at > > > com.cloud.bridge.service.core.s3.S3Engine.listBucketContents(S3Engine. > > > java:36 > > > 2) > > > at > > > com.cloud.bridge.service.controller.s3.S3BucketAction.executeGetBu > > > ck > > > et > > > (S3Bu > > > cketAction.java:578) > > > at > > > > > > com.cloud.bridge.service.controller.s3.S3BucketAction.execute(S3BucketAction. > > > java:202) > > > at > > > > > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet. > > ja > > va:181) > > > at > > > com.cloud.bridge.service.S3RestServlet.doGet(S3RestServlet.java:84) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > at > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A > > > pp > > > li > > > cationFilte > > > rChain.java:290) > > > at > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati > > > on > > > Fi > > > lterChain.j > > > ava:206) > > > at > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp > > > er > > > Va > > > lve > > > .java:233) > > > at > > > org.apache.catalina.core.StandardContextValve.invoke(StandardConte > > > xt > > > Va > > > lve.j > > > ava:191) > > > at > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve. > > > ja > > > va:12 > > > 7) > > > at > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve. > > ja > > va:102) > > > at > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:5 > 55) > > > at > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngine > > > Va > > > lv > > > e.jav > > > a:109) > > > at > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.ja > > va > > :298) > > > at > > > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProce > > > ss > > > or > > > .jav > > > a:889) > > > at > > > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler > > > .p > > > ro > > > ce > > > ss(Http11NioProtocol.java:721) > > > at > > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndp > > > oi > > > nt > > > .java > > > :2268) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut > > > or > > > .j > > > ava:1 > > > 110) > > > at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > > > 603) > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > Thanks and Regards. > > > > > > Asmita > > > > > > -----Original Message----- > > > From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > > Sent: 26 March 2013 PM 06:00 > > > To: users@cloudstack.apache.org; 'Sebastien Goasguen' > > > Subject: RE: CS4 AWS S3 support > > > > > > From the code, this exception will be thrown if the S3 is not > > > able to create the > > > folder(bucket) in that specified location. > > > Please check the permission and mount points. That is only issue. > > > > > > Thanks > > > Rajesh Battala > > > > > > > > > > > > > -----Original Message----- > > > > From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] > > > > Sent: Tuesday, March 26, 2013 5:31 PM > > > > To: 'users@cloudstack.apache.org'; 'Sebastien Goasguen' > > > > Subject: RE: CS4 AWS S3 support > > > > > > > > Hi , > > > > > > > > Yes, folder is created on root directory and mount this is the mount > > > > point. > > > > The ownership is assigned to the root group and cloud user. > > > > > > > > $mount > > > > > > > > /dev/mapper/RootVolGroup-RootVol on / type ext4 (rw) proc on > > > > /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on > > > > /dev/pts type devpts > > > > (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs > > > > (rw,rootcontext="system_u:object_r:tmpfs_t:s0") > > > > /dev/sda1 on /boot type ext4 (rw) > > > > /dev/mapper/RootVolGroup-DataVol on /data type ext4 (rw) > > > > /dev/mapper/RootVolGroup-VarVol on /var type ext4 (rw) none on > > > > /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on > > > > /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) nfsd on > > > > /proc/fs/nfsd type nfsd (rw) > > > > tornfs01sx.sigmasys.net:/data/cloudstack/s3storage > > > > on /s3storage type nfs > > > > (rw,rsize=8192,wsize=8192,timeo=14,intr,vers=4,addr=xx.xx.xx.xx, > > > > cl > > > > ie > > > > nt > > > > addr=xx > > > > .xx.xx.xx) > > > > > > > > > > > > Thanks and Regards. > > > > > > > > Asmita > > > > > > > > -----Original Message----- > > > > From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > > > Sent: 26 March 2013 PM 05:21 > > > > To: users@cloudstack.apache.org; 'Sebastien Goasguen' > > > > Subject: RE: CS4 AWS S3 support > > > > > > > > Is the storage point/ folder is created? > > > > You need to create the storage folder "/s3storage" in the server. > > > > > > > > Thanks > > > > Rajesh Battala > > > > > > > > > -----Original Message----- > > > > > From: Asmita Vagyani [mailto:asmita.vagy...@sigma-systems.com] > > > > > Sent: Tuesday, March 26, 2013 5:12 PM > > > > > To: 'Sebastien Goasguen'; 'users@cloudstack.apache.org' > > > > > Subject: RE: CS4 AWS S3 support > > > > > > > > > > I have reported a bug - > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK- > > > > > 1807 > > > > > > > > > > Now, when I create a bucket using the S3 AWS API on cloudstack > > > > > I get > > > > > > error as - OutOfStorageException: Unable to create > > > > > > /s3storage/test123 for bucket test123 Stacktrace is- > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header x-amz-date:Tue, > > > > > > 26 Mar > > > > > > 2013 > > > > > > 11:13:42 GMT > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header > > > > > > host:torvm-cloudstack-mgmt.sigmasys.net:7080 > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header content-length:0 > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header > > > > > > connection:Keep-Alive > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) - End of request - > > > > > > 2013-03-26 07:13:47,800 DEBUG [bridge.service.UserContext] > > > > > > (catalina-exec- > > > > > int-22:null) initializing a new [anonymous] UserContext! > > > > > > 2013-03-26 07:13:47,815 ERROR [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Unexpected exception Unable to > > > > > > create > > > > > > /s3storage/test123 for bucket test123 > > > > > > com.cloud.bridge.service.exception.OutOfStorageException: > > > > > > Unable to create > > > > > /s3storage/test123 for bucket test123 > > > > > > at > > > > > com.cloud.bridge.io.S3FileSystemBucketAdapter.createContainer( > > > > > S3 > > > > > Fi > > > > > le > > > > > Sy > > > > > stemB > > > > > ucketAdapter.java:55) > > > > > > at > > > > > com.cloud.bridge.service.core.s3.S3Engine.allocBucketStorageHo > > > > > st > > > > > (S > > > > > 3E > > > > > ng > > > > > ine.ja > > > > > va:1466) > > > > > > at > > > > > > com.cloud.bridge.service.core.s3.S3Engine.handleRequest(S3Engine.java: > > > > > 210) > > > > > > at > > > > > com.cloud.bridge.service.controller.s3.S3BucketAction.executeP > > > > > ut > > > > > Bu > > > > > ck > > > > > et > > > > > (S3Buc > > > > > ketAction.java:837) > > > > > > at > > > > > > > > > > > > > > > com.cloud.bridge.service.controller.s3.S3BucketAction.execute(S3BucketAction. > > > > > java:159) > > > > > > at > > > > > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet. > > > > > ja > > > > > va:181) > > > > > > at > > > com.cloud.bridge.service.S3RestServlet.doPut(S3RestServlet.java:97 > > > ) > > > > > > at > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:640) > > > > > > at > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > at > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > > > er > > > > > (A > > > > > pp > > > > > li > > > > > cationFilte > > > > > rChain.java:290) > > > > > > at > > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > > > ca > > > > > ti > > > > > on > > > > > Fi > > > > > lterChain.j > > > > > ava:206) > > > > > > at > > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > > > ra > > > > > pp > > > > > er > > > > > Va > > > > > lve > > > > > .java:233) > > > > > > at > > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > > > on > > > > > te > > > > > xt > > > > > Va > > > > > lve.j > > > > > ava:191) > > > > > > at > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve. > > > > > ja > > > > > va:12 > > > > > 7) > > > > > > at > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve. > > > > > ja > > > > > va:102) > > > > > > at > > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve. > > > > > ja > > > > > va > > > > > :5 > > > > > 55) > > > > > > at > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > > > gi > > > > > ne > > > > > Va > > > > > lv > > > > > e.jav > > > > > a:109) > > > > > > at > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter. > > > > > ja > > > > > va > > > > > :298) > > > > > > at > > > > > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioP > > > > > ro > > > > > ce > > > > > ss > > > > > or > > > > > .jav > > > > > a:889) > > > > > > at > > > > > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHan > > > > > dl > > > > > er > > > > > .p > > > > > ro > > > > > ce > > > > > ss(Http11NioProtocol.java:721) > > > > > > at > > > > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(Nio > > > > > En > > > > > dp > > > > > oi > > > > > nt > > > > > .java > > > > > :2268) > > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > > > > > ec > > > > > ut > > > > > or > > > > > .j > > > > > ava:1 > > > > > 110) > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > > > > > 603) > > > > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > > > > > > > > > > What could be the reason? > > > > > > > > > > > > > > > Thanks and Regards. > > > > > > > > > > Asmita > > > > > > > > > > -----Original Message----- > > > > > From: Sebastien Goasguen [mailto:run...@gmail.com] > > > > > Sent: 26 March 2013 PM 04:57 > > > > > To: Asmita Vagyani > > > > > Subject: Re: CS4 AWS S3 support > > > > > > > > > > > > > > > On Mar 26, 2013, at 7:19 AM, Asmita Vagyani > > > > > <Asmita.Vagyani@sigma- systems.com> wrote: > > > > > > > > > > > There is no bucket created on my NFS mount. > > > > > > This is not an error specific to S3 browser tool.Even if I > > > > > > use this url in the IE > > > > > browser, it gives 400-Bad request error. > > > > > > > > > > > > When I give url as > > > > > > http://torvm-cloudstack-mgmt.sigmasys.net:7080/awsapi/rest/A > > > > > > ma > > > > > > zo > > > > > > nS > > > > > > 3/ > > > > > > te > > > > > > st2342 > > > > > > It gives error like : > > > > > > <Error> > > > > > > <Code>NoSuchBucket</Code> > > > > > > <Message>The specified bucket does not exist</Message> > > > > > > <BucketName>test2342</BucketName> > > > > > <RequestId>1DEADBEEF9</RequestId> > > > > > > <HostId>abCdeFgHiJ1k2LmN3op4q56r7st89</HostId> > > > > > > </Error> > > > > > > > > > > > > This is a normal behavior. Same behavior I should get when I > > > > > > don't pass any > > > > > name of bucket. > > > > > >>>>>>>>>>>>>> 1.....Please check if this is a bug in CS4. > > > > > > > > > > if you think it's a bug, then please open a bug in jira: > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK > > > > > > > > > > > > > > > > > Now, when I create a bucket using the S3 AWS API on > > > > > > cloudstack I get error as - OutOfStorageException: Unable to > > > > > > create > > > > > > /s3storage/test123 for bucket test123 Stacktrace is- > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header x-amz-date:Tue, > > > > > > 26 Mar > > > > > > 2013 > > > > > > 11:13:42 GMT > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header > > > > > > host:torvm-cloudstack-mgmt.sigmasys.net:7080 > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header content-length:0 > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Request header > > > > > > connection:Keep-Alive > > > > > > 2013-03-26 07:13:47,796 INFO [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) - End of request - > > > > > > 2013-03-26 07:13:47,800 DEBUG [bridge.service.UserContext] > > > > > > (catalina-exec- > > > > > int-22:null) initializing a new [anonymous] UserContext! > > > > > > 2013-03-26 07:13:47,815 ERROR [bridge.service.S3RestServlet] > > > > > > (catalina-exec-int-22:null) Unexpected exception Unable to > > > > > > create > > > > > > /s3storage/test123 for bucket test123 > > > > > > com.cloud.bridge.service.exception.OutOfStorageException: > > > > > > Unable to create > > > > > /s3storage/test123 for bucket test123 > > > > > > at > > > > > com.cloud.bridge.io.S3FileSystemBucketAdapter.createContainer( > > > > > S3 > > > > > Fi > > > > > le > > > > > Sy > > > > > stemB > > > > > ucketAdapter.java:55) > > > > > > at > > > > > com.cloud.bridge.service.core.s3.S3Engine.allocBucketStorageHo > > > > > st > > > > > (S > > > > > 3E > > > > > ng > > > > > ine.ja > > > > > va:1466) > > > > > > at > > > > > > com.cloud.bridge.service.core.s3.S3Engine.handleRequest(S3Engine.java: > > > > > 210) > > > > > > at > > > > > com.cloud.bridge.service.controller.s3.S3BucketAction.executeP > > > > > ut > > > > > Bu > > > > > ck > > > > > et > > > > > (S3Buc > > > > > ketAction.java:837) > > > > > > at > > > > > > > > > > > > > > > com.cloud.bridge.service.controller.s3.S3BucketAction.execute(S3BucketAction. > > > > > java:159) > > > > > > at > > > > > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet. > > > > > ja > > > > > va:181) > > > > > > at > > > com.cloud.bridge.service.S3RestServlet.doPut(S3RestServlet.java:97 > > > ) > > > > > > at > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:640) > > > > > > at > > > > > > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > at > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > > > er > > > > > (A > > > > > pp > > > > > li > > > > > cationFilte > > > > > rChain.java:290) > > > > > > at > > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > > > ca > > > > > ti > > > > > on > > > > > Fi > > > > > lterChain.j > > > > > ava:206) > > > > > > at > > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > > > ra > > > > > pp > > > > > er > > > > > Va > > > > > lve > > > > > .java:233) > > > > > > at > > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > > > on > > > > > te > > > > > xt > > > > > Va > > > > > lve.j > > > > > ava:191) > > > > > > at > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve. > > > > > ja > > > > > va:12 > > > > > 7) > > > > > > at > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve. > > > > > ja > > > > > va:102) > > > > > > at > > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve. > > > > > ja > > > > > va > > > > > :5 > > > > > 55) > > > > > > at > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > > > gi > > > > > ne > > > > > Va > > > > > lv > > > > > e.jav > > > > > a:109) > > > > > > at > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter. > > > > > ja > > > > > va > > > > > :298) > > > > > > at > > > > > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioP > > > > > ro > > > > > ce > > > > > ss > > > > > or > > > > > .jav > > > > > a:889) > > > > > > at > > > > > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHan > > > > > dl > > > > > er > > > > > .p > > > > > ro > > > > > ce > > > > > ss(Http11NioProtocol.java:721) > > > > > > at > > > > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(Nio > > > > > En > > > > > dp > > > > > oi > > > > > nt > > > > > .java > > > > > :2268) > > > > > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > > > > > ec > > > > > ut > > > > > or > > > > > .j > > > > > ava:1 > > > > > 110) > > > > > > at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > > > > > 603) > > > > > > at java.lang.Thread.run(Thread.java:722) > > > > > > > > > > > > > > > > > > What could be the reason? > > > > > > > > > > > > > > > > > > > > > > > > Thanks and Regards. > > > > > > > > > > > > Asmita > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Sebastien Goasguen [mailto:run...@gmail.com] > > > > > > Sent: 26 March 2013 PM 04:33 > > > > > > To: Asmita Vagyani > > > > > > Subject: Re: CS4 AWS S3 support > > > > > > > > > > > > > > > > > > On Mar 26, 2013, at 6:56 AM, Asmita Vagyani > > > > > > <Asmita.Vagyani@sigma- > > > > > systems.com> wrote: > > > > > > > > > > > >> Hi Sebastien, > > > > > >> > > > > > >> Thanks , after giving the FQDN instead of IP address, it works. > > > > > >> Now, atleast the S3 browser is able to invoke the > > > > > >> cloudstack > > > > > >> S3 AWS REST > > > > > API. > > > > > >> > > > > > >> I saw in the documentation > > > > > >> https://cwiki.apache.org/CLOUDSTACK/s3-api-in-cloudstack.ht > > > > > >> ml It says to List All Buckets, simply make a call to : > > > > > >> REST GET request, http://myhost:8080/awsapi/rest/AmazonS3 > > > > > >> > > > > > >> However, when I invoke > > > > > >> http://torvm-cloudstack-mgmt.sigmasys.net:7080/awsapi/rest/ > > > > > >> Am > > > > > >> az > > > > > >> on > > > > > >> S3 I get the 400 Bad request error on browser. > > > > > >> I get the exception at backend when I checked awsapi.log: > > > > > >> > > > > > > > > > > > > I have never used the S3 browser but it looks like you may > > > > > > have some existing > > > > > buckets in there that cause the error. > > > > > > > > > > > > > > > > > >> 2013-03-26 06:49:32,080 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Request servletPath: > > > > > >> /rest/AmazonS3 > > > > > >> 2013-03-26 06:49:32,080 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Request header user-agent:S3 > > > > > >> Browser3-8-1 > > > > > >> 2013-03-26 06:49:32,081 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Request header > > > > > >> authorization:AWS > > > > > >> > > > > > > > > > > > > > > > Qh69lnbtbr0H68i_Q7im7eZNhoFYUyHS8eDU9nyL2k3OP0gBGN0gNHfHWeKQo1 > > > > > EZq2r0M > > > > > >> M ZK-23U9_wF6PMojg:RWAiwif28JxZ/le9mDkQskfPAd8= > > > > > >> 2013-03-26 06:49:32,081 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Request header x-amz-date:Tue, > > > > > >> 26 Mar > > > > > >> 2013 > > > > > >> 10:49:26 GMT > > > > > >> 2013-03-26 06:49:32,081 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Request header > > > > > >> host:torvm-cloudstack-mgmt.sigmasys.net:7080 > > > > > >> 2013-03-26 06:49:32,081 INFO > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) - End of request - > > > > > >> 2013-03-26 06:49:32,093 DEBUG [bridge.service.UserContext] > > > > > >> (catalina- > > > > > exec-int-20:null) initializing a new [anonymous] UserContext! > > > > > >> 2013-03-26 06:49:32,093 ERROR > > > > > >> [bridge.service.S3RestServlet] > > > > > >> (catalina-exec-int-20:null) Unexpected exception is not > > > > > >> between > > > > > >> 3 and > > > > > >> 255 characters long > > > > > >> com.cloud.bridge.service.exception.InvalidBucketName: is > > > > > >> not between 3 > > > > > and 255 characters long > > > > > >> at > > > > > com.cloud.bridge.service.core.s3.S3Engine.verifyBucketName(S3Engine. > > > > > ja > > > > > va:18 > > > > > 09) > > > > > >> at > > > > > com.cloud.bridge.service.S3RestServlet.routeRequest(S3RestServlet. > > > > > ja > > > > > va > > > > > :440) > > > > > >> at > > > > > com.cloud.bridge.service.S3RestServlet.processRequest(S3RestServlet. > > > > > ja > > > > > va:179) > > > > > >> at > > > com.cloud.bridge.service.S3RestServlet.doGet(S3RestServlet.java:84 > > > ) > > > > > >> at > > > > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > > > > >> at > > > > > >> javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > >> at > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilt > > > > > er > > > > > (A > > > > > pp > > > > > li > > > > > cationFilte > > > > > rChain.java:290) > > > > > >> at > > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(Appli > > > > > ca > > > > > ti > > > > > on > > > > > Fi > > > > > lterChain.j > > > > > ava:206) > > > > > >> at > > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardW > > > > > ra > > > > > pp > > > > > er > > > > > Va > > > > > lve > > > > > .java:233) > > > > > >> at > > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardC > > > > > on > > > > > te > > > > > xt > > > > > Va > > > > > lve.j > > > > > ava:191) > > > > > >> at > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve. > > > > > ja > > > > > va:12 > > > > > 7) > > > > > >> at > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve. > > > > > ja > > > > > va:102) > > > > > >> at > > > > > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve. > > > > > ja > > > > > va > > > > > :5 > > > > > 55) > > > > > >> at > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEn > > > > > gi > > > > > ne > > > > > Va > > > > > lv > > > > > e.jav > > > > > a:109) > > > > > >> at > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter. > > > > > ja > > > > > va > > > > > :298) > > > > > >> at > > > > > org.apache.coyote.http11.Http11NioProcessor.process(Http11NioP > > > > > ro > > > > > ce > > > > > ss > > > > > or > > > > > .jav > > > > > a:889) > > > > > >> at > > > > > org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHan > > > > > dl > > > > > er > > > > > .p > > > > > ro > > > > > ce > > > > > ss(Http11NioProtocol.java:721) > > > > > >> at > > > > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(Nio > > > > > En > > > > > dp > > > > > oi > > > > > nt > > > > > .java > > > > > :2268) > > > > > >> at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx > > > > > ec > > > > > ut > > > > > or > > > > > .j > > > > > ava:1 > > > > > 110) > > > > > >> at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: > > > > > 603) > > > > > >> at java.lang.Thread.run(Thread.java:722) > > > > > >> > > > > > >> Thanks and Regards. > > > > > >> > > > > > >> Asmita > > > > > >> > > > > > >> -----Original Message----- > > > > > >> From: Sebastien Goasguen [mailto:run...@gmail.com] > > > > > >> Sent: 26 March 2013 PM 04:06 > > > > > >> To: Asmita Vagyani > > > > > >> Subject: Re: CS4 AWS S3 support > > > > > >> > > > > > >> > > > > > >> On Mar 26, 2013, at 6:25 AM, Asmita Vagyani > > > > > >> <Asmita.Vagyani@sigma- > > > > > systems.com> wrote: > > > > > >> > > > > > >>> Hi > > > > > >>> Where do I register the DNS entry I mean do you mean a > > > > > >>> local DNS entry or > > > > > what? > > > > > >> > > > > > >> Local entry would do, as long as your "s3 browser" knows > > > > > >> how to route > > > > > <server.com> to the IP of the endpoint. > > > > > >> > > > > > >>> > > > > > >>> Thanks and Regards. > > > > > >>> > > > > > >>> Asmita > > > > > >>> > > > > > >>> -----Original Message----- > > > > > >>> From: Sebastien Goasguen [mailto:run...@gmail.com] > > > > > >>> Sent: 26 March 2013 PM 03:34 > > > > > >>> To: Asmita Vagyani > > > > > >>> Subject: Re: CS4 AWS S3 support > > > > > >>> > > > > > >>> > > > > > >>> On Mar 26, 2013, at 4:40 AM, Asmita Vagyani > > > > > >>> <Asmita.Vagyani@sigma- > > > > > systems.com> wrote: > > > > > >>> > > > > > >>>> Hi all, > > > > > >>>> > > > > > >>>> I tried to use a couple of freewares/tools like > > > > > >>>> CyberDuck, DragonDisk, etc > > > > > which interfaces the Cloudstack S3 APIs. > > > > > >>>> I am stuck at one common point in all these tools. > > > > > >>>> All tools say they support the alternative S3 compatible > > > > > >>>> services, which in > > > > > our case is our Cloudstack AWS S3. > > > > > >>>> However, it asks me endpoint of the S3 compatible service. > > > > > >>>> When I mention the endpoint as http://<cloudstack-server- > > > > > IP:port>/awsapi/rest/AmazonS3 it doesnot accept it. > > > > > >>>> It wants the server name as server.com. > > > > > >>>> > > > > > >>> > > > > > >>> Asmita, I woud do two things: > > > > > >>> > > > > > >>> 1-Try to use boto like I explained in the slides, that > > > > > >>> should show you if the S3 > > > > > service is working properly or not. > > > > > >>> 2-If the S3 tools that you are using do not take an IP, > > > > > >>> then you will need to > > > > > register a dns entry. > > > > > >>> > > > > > >>> note that the S3 support is really a tech preview. > > > > > >>> > > > > > >>> -sebastien > > > > > >>> > > > > > >>>> Is there any way I can get the S3 webservice endpoint the > > > > > >>>> way these tools > > > > > want. > > > > > >>>> Is this because Cloudstack is not consistent with the EC2 > > > > > >>>> endpoint. > > > > > >>>> If yes, when are we planning to make it consistent. > > > > > >>>> > > > > > >>>> > > > > > >>>> Thanks and Regards. > > > > > >>>> > > > > > >>>> Asmita > > > > > >>>> > > > > > >>>> -----Original Message----- > > > > > >>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > > > > >>>> Sent: 25 March 2013 PM 09:40 > > > > > >>>> To: users@cloudstack.apache.org; 'cloudstack- > > > > > us...@incubator.apache.org'; 'run...@gmail.com' > > > > > >>>> Subject: RE: CS4 AWS S3 support > > > > > >>>> > > > > > >>>> Am not sure whether s3 browser allows user to configure > > > > > >>>> the settings to > > > > > access the cloudstack s3. > > > > > >>>> If you can configure the settings the way we configure > > > > > >>>> for boto then we > > > > > can use s3 browser. > > > > > >>>> > > > > > >>>> Thanks > > > > > >>>> Rajesh Battala > > > > > >>>> > > > > > >>>>> -----Original Message----- > > > > > >>>>> From: Asmita Vagyani > > > > > >>>>> [mailto:asmita.vagy...@sigma-systems.com] > > > > > >>>>> Sent: Monday, March 25, 2013 7:09 PM > > > > > >>>>> To: 'users@cloudstack.apache.org'; > > > > > >>>>> 'cloudstack-us...@incubator.apache.org'; > > > > > >>>>> 'run...@gmail.com' > > > > > >>>>> Subject: RE: CS4 AWS S3 support > > > > > >>>>> > > > > > >>>>> Are there any other clients, non-python based. > > > > > >>>>> Can I integrate this with S3 browser, such that the S3 > > > > > >>>>> browser shows me buckets created on the NFS drive > > > > > >>>>> mounted using cloud- > > > > > bridge.properties? > > > > > >>>>> User can just upload and download files in S3 browser > > > > > >>>>> and ultimately on the NFS drive buckets. > > > > > >>>>> > > > > > >>>>> > > > > > >>>>> Thanks and Regards. > > > > > >>>>> > > > > > >>>>> Asmita > > > > > >>>>> > > > > > >>>>> -----Original Message----- > > > > > >>>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > > > > >>>>> Sent: 25 March 2013 PM 04:42 > > > > > >>>>> To: users@cloudstack.apache.org; > > > > > >>>>> 'cloudstack-us...@incubator.apache.org'; > > > > > >>>>> 'run...@gmail.com' > > > > > >>>>> Subject: RE: CS4 AWS S3 support > > > > > >>>>> > > > > > >>>>> Asmita, > > > > > >>>>> You can follow this link. > > > > > >>>>> https://cwiki.apache.org/CLOUDSTACK/s3-api-in- > > > > > >>>>> cloudstack.html > > > > > >>>>> > > > > > >>>>> You can use boto client which is python based to send s3 > > > > > >>>>> requests to > > > CS. > > > > > >>>>> > > > > > >>>>> Thanks > > > > > >>>>> Rajesh Battala > > > > > >>>>> > > > > > >>>>>> -----Original Message----- > > > > > >>>>>> From: Asmita Vagyani > > > > > >>>>>> [mailto:asmita.vagy...@sigma-systems.com] > > > > > >>>>>> Sent: Monday, March 25, 2013 4:13 PM > > > > > >>>>>> To: 'users@cloudstack.apache.org'; > > > > > >>>>>> 'cloudstack-us...@incubator.apache.org'; > > > > > >>>>>> 'run...@gmail.com' > > > > > >>>>>> Subject: RE: CS4 AWS S3 support > > > > > >>>>>> > > > > > >>>>>> Hi, > > > > > >>>>>> > > > > > >>>>>> I can see the entry of keys in the usercredential table in CS4 > > > > > >>>>>> DB. > > > > > >>>>>> When I access http://<server-ip>/awsapi link I get the > > > > > >>>>>> servlet > > > > exception : > > > > > >>>>>> > > > > > >>>>>> java.lang.RuntimeException: javax.servlet.ServletException: > > > > > >>>>>> Servlet execution threw an exception > > > > > >>>>>> > > > > > >>>>>> com.cloud.bridge.service.EC2MainServlet.doGetOrPost(EC2 > > > > > >>>>>> Ma > > > > > >>>>>> in > > > > > >>>>>> Se > > > > > >>>>>> rv > > > > > >>>>>> le > > > > > >>>>>> t > > > > > >>>>>> .java:104) > > > > > >>>>>> > > > > > >>>>>> > > > com.cloud.bridge.service.EC2MainServlet.doGet(EC2MainServlet.java: > > > > > >>>>>> 7 > > > > > >>>>>> 3) > > > > > >>>>>> > > > > > >>>>>> How can I run S3 requests? any links to documentation > > > > > >>>>>> would be > > > > helpful. > > > > > >>>>>> > > > > > >>>>>> Thanks and Regards. > > > > > >>>>>> > > > > > >>>>>> Asmita > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>> -----Original Message----- > > > > > >>>>>> From: Rajesh Battala [mailto:rajesh.batt...@citrix.com] > > > > > >>>>>> Sent: 25 March 2013 PM 03:35 > > > > > >>>>>> To: users@cloudstack.apache.org; > > > > > >>>>>> 'cloudstack-us...@incubator.apache.org'; > > > > > >>>>>> 'run...@gmail.com' > > > > > >>>>>> Subject: RE: CS4 AWS S3 support > > > > > >>>>>> > > > > > >>>>>> Hi Asmita, > > > > > >>>>>> > > > > > >>>>>> In the cloudbridge db, usercredentials table has the > > > > > >>>>>> username/password values present? > > > > > >>>>>> Generally, for an existing account, generate the > > > > > >>>>>> apikey/secrect key and register them in awsapi. > > > > > >>>>>> > > > > > >>>>>> Were you able to run s3 requests.? Is awsap is running > > > > > >>>>>> at 7080 > > port? > > > > > >>>>>> > > > > > >>>>>> Thanks > > > > > >>>>>> Rajesh Battala > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>>>> -----Original Message----- > > > > > >>>>>>> From: Asmita Vagyani > > > > > >>>>>>> [mailto:asmita.vagy...@sigma-systems.com] > > > > > >>>>>>> Sent: Monday, March 25, 2013 3:24 PM > > > > > >>>>>>> To: 'cloudstack-us...@incubator.apache.org'; > > 'run...@gmail.com' > > > > > >>>>>>> Subject: CS4 AWS S3 support > > > > > >>>>>>> > > > > > >>>>>>> Hi all, > > > > > >>>>>>> > > > > > >>>>>>> I am trying to do following to have an implementation > > > > > >>>>>>> of > > > > > >>>>>>> S3 in our > > > > > >>>>> application. > > > > > >>>>>>> Followed steps given in: > > > > > >>>>>>> http://www.slideshare.net/sebastiengoasguen/cloudstack > > > > > >>>>>>> -s > > > > > >>>>>>> 3 > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> 1) I have enabled S3 support and EC2 support flag in the > > > > > >>>>>>> CS4 > > > global > > > > > >>>>>> variables. > > > > > >>>>>>> > > > > > >>>>>>> 2) I have mounted the NFS store on the CS4 management > > server > > > (I > > > > > am > > > > > >>>>> using > > > > > >>>>>>> the NFS drive to store the buckets/files uploaded by > > > > > >>>>>>> user on this > > > > > >>>>>>> drive) > > > > > >>>>>>> > > > > > >>>>>>> 3) I have configured the cloud-bridge.properties > appropriately > > > > > >>>>>>> > > > > > >>>>>>> host=http://localhost:7080/awsapi > > > > > >>>>>>> > > > > > >>>>>>> storage.root=/s3storage > > > > > >>>>>>> > > > > > >>>>>>> storage.multipartDir=__multipart__uploads__ > > > > > >>>>>>> > > > > > >>>>>>> bucket.dns=false > > > > > >>>>>>> > > > > > >>>>>>> serviceEndpoint=localhost:7080 > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> > > > > > >>>>>>> 4) I made up a user account into CS4 client console, > generated > > > the > > > > > API key > > > > > >>>>>>> and the Secret key. > > > > > >>>>>>> > > > > > >>>>>>> 5) I have registered the user by using > > > > > >>>>>>> cloudstack-aws-api- > > register > > > > > (refer - > > > > > >>>>>>> http://cloudstack.apache.org/docs/en- > > > US/Apache_CloudStack/4.0.1- > > > > > >>>>>>> incubating/html/Installation_Guide/aws-ec2-user-setup.html ) > > > > > >>>>>>> > > > > > >>>>>>> 6) I have downloaded the S3 browser. > > > > > >>>>>>> > > > > > >>>>>>> 7) I am trying to make an account with the same user API > > > > > >>>>>>> key > > and > > > > > Secret > > > > > >>>>> key > > > > > >>>>>>> which we create on Cloudstack. > > > > > >>>>>>> > > > > > >>>>>>> While creating an account it says - InvalidAccesskeyId > > > > > >>>>>>> > > > > > >>>>>>> Any hints if I am missing some step here? > > > > > >>>>>>> > > > > > >>>>>>> Thanks and Regards. > > > > > >>>>>>> > > > > > >>>>>>> Asmita Patil Vagyani. > > > > > >>>>>>> > > > > > >>>>>>> Tech Lead, Sigma Software Solutions, Pune, > > > > > >>>>>>> > > > > > >>>>>>> T: +91-20-25661992 / 93 / 94. Extn. : 286 > > > > > >>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
