[
https://issues.apache.org/jira/browse/HADOOP-12548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096906#comment-15096906
]
Matthew Paduano commented on HADOOP-12548:
------------------------------------------
I applied the patch 04 to trunk and attempted the end to end steps described in
the PDF.
Placing the credential file in HDFS and using the correct URI *succeeded* with
{{distcp}}.
Great work! thanks! :)
Other notes:
- in {{getAWSAccessKeys()}}, exceptions from accessing the config are simply
eaten. This includes stuff like typo's in filenames and mistyped URI's and so
on. All the user gets is a cryptic message with
{{com.amazonaws.AmazonClientException}}. I recommend letting exceptions
through as the eaten exceptions all had nice messages that would have helped
with debugging.
- in the PDF there is a sample {{distcp}} command line using the -D flag.
The -D flag is not parsed by {{distcp}} in the way that is expected and this
fails. Recommend removing that -D (which is already in brackets) from the
example. Also, this means that to run {{distcp}} one has to edit
{{core-site.xml}} to pass the provider path. For unit tests, one can add the
property to {{contract-test-options.xml}} and {{auth-keys.xml}}. Is there
anything like that for {{hadoop}} command line? e.g. {{hadoop distcp
--property_overrides=props.xml ...}}
- the HDFS creds cannot be tested using the unit tests. I am not sure what
can be done about it, but it is disappointing.
- are there plans to take this functionality to {{S3FileSystem}} or
{{NativeS3FileSystem}}?
> read s3 creds from a Credential Provider
> ----------------------------------------
>
> Key: HADOOP-12548
> URL: https://issues.apache.org/jira/browse/HADOOP-12548
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3
> Reporter: Allen Wittenauer
> Assignee: Larry McCay
> Attachments: CredentialProviderAPIforS3FS-002.pdf,
> HADOOP-12548-01.patch, HADOOP-12548-02.patch, HADOOP-12548-03.patch,
> HADOOP-12548-04.patch
>
>
> It would be good if we could read s3 creds from a source other than via a
> java property/Hadoop configuration option
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
