[
https://issues.apache.org/jira/browse/FLINK-28216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17557856#comment-17557856
]
Prabhu Joseph commented on FLINK-28216:
---------------------------------------
[~martijnvisser] EMR has their own
[EMRFS|https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.html] to
access Amazon S3. Currently, EMR Hive, Spark and Presto uses EMRFS. But Flink
has hardcoded to Hadoop S3AFileSystem. It does not check the fs.s3a.impl config
from core-site.xml
{code}
<property>
<name>fs.s3.impl</name>
<value>com.amazon.ws.emr.hadoop.fs.EmrFileSystem</value>
</property>
{code}
bq. Is it not already possible, as documented on
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/
?
Yes it is possible to access S3 from Flink using S3AFileSystem as per above
document but not using EMRFS or any other implementation.
> Hadoop S3FileSystemFactory does not honor fs.s3.impl
> ----------------------------------------------------
>
> Key: FLINK-28216
> URL: https://issues.apache.org/jira/browse/FLINK-28216
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Affects Versions: 1.15.0
> Reporter: Prabhu Joseph
> Priority: Minor
>
> Currently Hadoop S3FileSystemFactory has hardcoded the S3 FileSystem
> implementation to S3AFileSystem. It does not allow to configure any other
> implementation specified in fs.s3.impl. Suggest to read the fs.s3.impl from
> Hadoop Config loaded and use the same.
>
> {code:java}
> @Override
> protected org.apache.hadoop.fs.FileSystem createHadoopFileSystem() {
> return new S3AFileSystem();
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
