Get S3 working first ... the new AWS command line tools are prety good. http://aws.amazon.com/cli/
But so is the EC2 console nowadays you can quite easily setup simple permissions on buckets. Doing very tight access control is a PITA though because you have to write policies ... but if you can survive with the basics like "owner all, everyone else nothing" its easy. Let me know if you need help. From: [email protected] [mailto:[email protected]] On Behalf Of Harry Bakken Sent: Friday, March 07, 2014 12:54 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] AWS S3 Function Library? As soon as I can get this working right, I'll blog the steps to use this from end-to-end because from where I am sitting, this isn't as straightforward to configure as the documentation implies. I have to think the confusion is more on the bucket policy configuration than on the ML side... On Fri, Mar 7, 2014 at 11:51 AM, David Lee <[email protected]<mailto:[email protected]>> wrote: I actually can't see it documented anywhere so I cant say officially, the closest I can find is here http://www.marklogic.com/what-is-marklogic/marklogic/ where its mentioned you can use s3 as a path for backups. I dont know the official status ... -----Original Message----- From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Michael Blakeley Sent: Friday, March 07, 2014 12:01 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] AWS S3 Function Library? Is that functionality documented? Supported? -- Mike On 6 Mar 2014, at 23:27 , David Lee <[email protected]<mailto:[email protected]>> wrote: > I have a reasonably complete XQuery library for S3 ... > BUT ... if you are using V7 or greater you dont need it unless you need to do > something exotic. > All XQuery functions that expect filesystem paths can also take S3 and > HDFS paths And if your running on EC2 with IAM roles you dont even > need credentials, otherwise it takes your S3 credentials as configured in the > admin gui. > > --- List a directory > xdmp:filesystem-directory( > "s3://my-bucket/" > ) > Get a binary file > let $bin := xdmp:external-binary( > "s3://my-bucket/big-file.bin" > ) > return xdmp:binary-size($bin) > Storing a binary file > let $bin := xdmp:external-binary( > "s3:// big-file.bin " > ) > > return xdmp:save( "s3://your-bucket/bigfile.bin" , $bin ) > > > > > > From: > [email protected]<mailto:[email protected]> > [mailto:[email protected]<mailto:[email protected]>] > On Behalf Of Harry > Bakken > Sent: Thursday, March 06, 2014 11:07 PM > To: MarkLogic Developer Discussion > Subject: Re: [MarkLogic Dev General] AWS S3 Function Library? > > I'd be happy to set up a repo around this if you don't want to mess with it. > I was already thinking in that direction after I sent my message. I have been > looking for a relatively small open source effort to own. I want to be able > to do more than what you've mentioned, but those could serve as a great > start. I definitely need to create the signed requests and would like to have > functions around most of the operations on objects and be able to move/update > binaries in the buckets, etc. > > Thanks, > Harry > > > > On Thu, Mar 6, 2014 at 9:23 PM, Michael Blakeley > <[email protected]<mailto:[email protected]>> wrote: > What sort of things are you interested in doing via S3? > > I have a few simple functions that might be helpful: one is s3-get and the > other signs urls. I wasn't trying for completeness, just getting some work > done. But they're only about 150 lines and might help someone else. They use > various helper functions to sign strings, etc. > > If there's interest I could throw them into a github repo. > > -- Mike > > On 6 Mar 2014, at 18:47 , Harry Bakken > <[email protected]<mailto:[email protected]>> wrote: > > > Has anyone out there built any XQuery functions to assist in leveraging the > > Amazon S3 API in MarkLogic applications? I am specifically looking to > > interact with binaries in the S3 directly, not using the newer capabilities > > of a forest bound to S3. I don't want to reinvent the wheel if something is > > already out there. > > > > Any advice is appreciated! > > Harry > > _______________________________________________ > > General mailing list > > [email protected]<mailto:[email protected]> > > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected]<mailto:[email protected]> > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected]<mailto:[email protected]> > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected]<mailto:[email protected]> http://developer.marklogic.com/mailman/listinfo/general
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
