Using resigned URLs is a good way, not only do you divert the load from the server, but you can take advantage of S3's location endpoints and even CloudFront if you like. It isn't currently implemented in XQuery and I do not know of an example but it shouldn't be difficult, providing you have the keys in plain text ... Hence
I asked about IAM roles because those are more tricky. If you want to presign with the credentials of your IAM role you don't have the actual key/secret-key 'in hand', it needs to be fetched at runtime and is temporary itself. ML handles this for S3 but currently does not expose the temporary IAM credentials. You might be able to make use of a JavaScript implementation such as this: http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-examples.html I have not tested this but the JS runtime in ML8 should be able to run any JavaScript that doesn't have platform binary libraries or framework dependencies. Other options are to run a 'mini server' on the host that uses one of the many SDK's (node.js, Java, Python). The startup overhead for each URL is probably too high so you'd want a long live process. The Java AWS SDK is included in /opt/MarkLogic/mlcmd/ext/aws/ (its used by the mlcmd program). http://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURLJavaSDK.html Its a simple few line program to write. This might make a good RFE for a later release to expose as a native ML API. There are various tools available to do this such as boto (python) https://github.com/aws/aws-cli/issues/462 ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected] Phone: +1 812-482-5224 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com/> From: [email protected] [mailto:[email protected]] On Behalf Of Pavadaidurai A Sent: Wednesday, August 19, 2015 8:56 AM To: MarkLogic Developer Discussion <[email protected]> Subject: Re: [MarkLogic Dev General] Marklogic Xquery library for AWS signature version 4 Hi David, I am using the query string authentication method as described In the link below. I am currently using AWS signature version 2 query string authentication method and planning to migrate to AWS signature version 4 query string authentication method. Yes, the method described below uses SHA1 or SHA256 algorithm and the keys to determine the signature. But the keys are encoded in the url itself and hence not visible. For each file, we generate a url and share with the user. We do have IAM role setup in Marklogic and it is running on EC2 instance. We decided to go with pre-signed urls to reduce the load on Marklogic server while downloading the file. Migrating from AWS signature 2: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth to AWS signature 4: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html Thanks, Durai. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of David Lee Sent: Wednesday, August 19, 2015 5:30 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Marklogic Xquery library for AWS signature version 4 Ah thanks ! Pre-signed URLs' That is a further complexity. I believe pre-signed S3 URLS are one of the most useful or required features of AWS as they transition to more IAM authentication for services. Could you outline what your needs are for Presigned URLs ? I won't have a solution for you immediately ( XQuery is no easier or harder to implement this ) but looking for a RFE it would be good to have customer feedback. e.g. Is this only for S3 ? Or other services ? Where do you get the credentials ? (Do you have the access key/secret key in plain text ? ) Do you want/need to make use of the IAM role associated with an EC2 instance ? Is ML on an EC2 instance or remote ? Are you using temporary or federated credentials as your source ? Thanks. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected]<mailto:[email protected]> Phone: +1 812-482-5224 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com/> From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Pavadaidurai A Sent: Wednesday, August 19, 2015 7:46 AM To: MarkLogic Developer Discussion <[email protected]<mailto:[email protected]>> Subject: Re: [MarkLogic Dev General] Marklogic Xquery library for AWS signature version 4 Hi David, Thanks for your response! The application I support has been designed with XQMVC framework. I am aware of the fact that Marklogic APIs can access S3. But whoever designed the application, decided to create the pre-signed S3 urls for the customers to download their files. The idea is that the load is now delegated to S3, since pre-signed urls directly hit S3 instead of hitting Marklogic server. S3 provides libraries for languages like Java, dotnet etc. But for Xquery there is no library available, to create the pre-signed S3 urls. Hence seeking help in this forum. Thanks, Durai. From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of David Lee Sent: Wednesday, August 19, 2015 5:03 PM To: MarkLogic Developer Discussion Subject: Re: [MarkLogic Dev General] Marklogic Xquery library for AWS signature version 4 Signature V4 is not required in most regions yet. What region are you accessing ? The next maintained release of ML 7x and 8x will use Signature V4 for all S3 access. Did you know you can use "s3://" URI's for S3 access anywhere a 'file' is normally used in ML instead of having to do your own HTTP call ? Where possible I recommend converting to using ML's internal S3 support, its more efficient and will be maintained as requirements and API's evolve. We are looking into exposing the signing algorithm as an API, it shouldn't be much different than the current XQuery code in 'ec2-2009-11-30.xqy' , however signing is a bit tedious to get absolutely correct in any language. ----------------------------------------------------------------------------- David Lee Lead Engineer MarkLogic Corporation [email protected]<mailto:[email protected]> Phone: +1 812-482-5224 Cell: +1 812-630-7622 www.marklogic.com<http://www.marklogic.com/> From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Aroul, Pavadai Durai (ELS-CON) Sent: Wednesday, August 19, 2015 6:57 AM To: [email protected]<mailto:[email protected]> Subject: [MarkLogic Dev General] Marklogic Xquery library for AWS signature version 4 Hello everyone, I am currently with Marklogic version 7 hosted in a cloud environment. I have an xquery to create a pre-signed url for “GET method” using AWS signature version 2 which uses SHA1 hashing algorithm. But I am planning to upgrade the process to AWS signature version 4, since newer S3 accounts support only AWS signature version 4. Do any of you have xquery code/library to create the pre-singed urls using AWS signature version 4 as described in the below link. It seems the newer S3 accounts support only AWS signature version 4 and not 2. If any of you can share the code in GIThub, it would be very helpful. http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationQueryStringAuth Thanks, Durai. **************** CAUTION - Disclaimer ***************** This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely for the use of the addressee(s). If you are not the intended recipient, please notify the sender by e-mail and delete the original message. Further, you are not to copy, disclose, or distribute this e-mail or its contents to any other person and any such actions are unlawful. This e-mail may contain viruses. Infosys has taken every reasonable precaution to minimize this risk, but is not liable for any damage you may sustain as a result of any virus in this e-mail. You should carry out your own virus checks before opening the e-mail or attachment. Infosys reserves the right to monitor and review the content of all messages sent to or from this e-mail address. Messages sent to or from this e-mail address may be stored on the Infosys e-mail system. ***INFOSYS******** End of Disclaimer ********INFOSYS***
_______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
