[
https://issues.apache.org/jira/browse/FLINK-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994667#comment-16994667
]
Arvid Heise commented on FLINK-15215:
-------------------------------------
For 1.10, we want to get rid of shading altogether. We fixed one bug
FLINK-14574 that prevented us from doing so and we are in the process to remove
the shading in [https://github.com/apache/flink/pull/10482] . The release is
planned in the beginning of January.
We are also backporting the fix to 1.9.0, but it's unlikely that we will also
backport the shading as it may break configuration. However, once 1.9.2 is out,
you could probably use the 1.10.0 s3 libraries as plugins.
Nevertheless, 1.9.2 will probably not appear much sooner and since you run on
EMR, it will also take some time until you could use that. Once we merge the
PR, you could of course try out the unshaded snapshot versions of 1.10. If you
are not hitting the above mentioned bug (only affect batch outputs and not
streaming), it could already work for you.
> Not able to provide a custom AWS credentials provider with flink-s3-fs-hadoop
> -----------------------------------------------------------------------------
>
> Key: FLINK-15215
> URL: https://issues.apache.org/jira/browse/FLINK-15215
> Project: Flink
> Issue Type: Bug
> Components: FileSystems
> Affects Versions: 1.9.0
> Reporter: Tank
> Priority: Major
>
> I am using Flink 1.9.0 on EMR (emr 5.28), with StreamingFileSink using an S3
> filesystem.
> I want flink to write to an S3 bucket which is running in another AWS
> account, and I want to do that by assuming a role in the other account. This
> can be easily accomplished by providing a custom credential provider (similar
> to
> [https://aws.amazon.com/blogs/big-data/securely-analyze-data-from-another-aws-account-with-emrfs/])
>
> As described in
> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/#pluggable-file-systems,
> I copied {{flink-s3-fs-hadoop-1.9.0.jar}} to the plugins directory. But the
> configuration parameter 'fs.s3a.aws.credentials.provider' is getting shaded
> [https://github.com/apache/flink/blob/master/flink-filesystems/flink-s3-fs-hadoop/src/main/java/org/apache/flink/fs/s3hadoop/S3FileSystemFactory.java#L47]}},
> and so are all the aws sdk dependencies, so when I provide a custom
> credential provider, it complained that I was not implementing the correct
> interface (AWSCredentialsProvider)
> The fix made in https://issues.apache.org/jira/browse/FLINK-13044 allows
> users to use one of the built-in credential providers like
> `_InstanceProfileCredentialsProvider`,_ but still does not help with
> providing custom credential providers.
>
> Related: https://issues.apache.org/jira/browse/FLINK-13602
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
