David, Your reply along with an off-discussion reply from someone else has me on the right path. ML 7 is already equipped to do exactly what we need, as you described. I was mistaken when I thought only a forest could point to the S3, but we can set the large data directory to the S3 or use the directives as you have them here. This is great stuff.
Harry PS- thanks for the link Danny! On Fri, Mar 7, 2014 at 1:27 AM, David Lee <[email protected]> wrote: > I have a reasonably complete XQuery library for S3 ... > > BUT ... if you are using V7 or greater you dont need it unless you need to > do something exotic. > > All XQuery functions that expect filesystem paths can also take S3 and > HDFS paths > > And if your running on EC2 with IAM roles you dont even need credentials, > otherwise it takes your S3 credentials > > as configured in the admin gui. > > > > --- List a directory > > xdmp:filesystem-directory( > > "s3://my-bucket/" > > ) > > Get a binary file > > let $bin := xdmp:external-binary( > > "s3://my-bucket/big-file.bin" > > ) > > return xdmp:binary-size($bin) > > Storing a binary file > > let $bin := xdmp:external-binary( > > "s3:// big-file.bin " > > ) > > > > return xdmp:save( "s3://your-bucket/bigfile.bin" , $bin ) > > > > > > > > > > > > *From:* [email protected] [mailto: > [email protected]] *On Behalf Of *Harry Bakken > *Sent:* Thursday, March 06, 2014 11:07 PM > *To:* MarkLogic Developer Discussion > *Subject:* Re: [MarkLogic Dev General] AWS S3 Function Library? > > > > I'd be happy to set up a repo around this if you don't want to mess with > it. I was already thinking in that direction after I sent my message. I > have been looking for a relatively small open source effort to own. I want > to be able to do more than what you've mentioned, but those could serve as > a great start. I definitely need to create the signed requests and would > like to have functions around most of the operations on objects and be able > to move/update binaries in the buckets, etc. > > > > Thanks, > > Harry > > > > > > On Thu, Mar 6, 2014 at 9:23 PM, Michael Blakeley <[email protected]> > wrote: > > What sort of things are you interested in doing via S3? > > I have a few simple functions that might be helpful: one is s3-get and the > other signs urls. I wasn't trying for completeness, just getting some work > done. But they're only about 150 lines and might help someone else. They > use various helper functions to sign strings, etc. > > If there's interest I could throw them into a github repo. > > -- Mike > > > On 6 Mar 2014, at 18:47 , Harry Bakken <[email protected]> wrote: > > > Has anyone out there built any XQuery functions to assist in leveraging > the Amazon S3 API in MarkLogic applications? I am specifically looking to > interact with binaries in the S3 directly, not using the newer capabilities > of a forest bound to S3. I don't want to reinvent the wheel if something is > already out there. > > > > Any advice is appreciated! > > Harry > > > _______________________________________________ > > General mailing list > > [email protected] > > http://developer.marklogic.com/mailman/listinfo/general > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > > > > _______________________________________________ > General mailing list > [email protected] > http://developer.marklogic.com/mailman/listinfo/general > >
_______________________________________________ General mailing list [email protected] http://developer.marklogic.com/mailman/listinfo/general
