[
https://issues.apache.org/jira/browse/FLINK-19144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201247#comment-17201247
]
Jingsong Lee commented on FLINK-19144:
--------------------------------------
When table.exec.hive.fallback-mapred-writer is false, writer will fallback to
Flink native writer, and use Flink FileSystem. But committer still use Hadoop
FileSystem, then will lead to this problem.
We can fix this, let committer use Flink FileSystem when use native Flink
writer.
> Error when writing to partitioned table with s3 FileSystem
> ----------------------------------------------------------
>
> Key: FLINK-19144
> URL: https://issues.apache.org/jira/browse/FLINK-19144
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
> Affects Versions: 1.11.1
> Reporter: Pei He
> Priority: Major
> Fix For: 1.11.3
>
>
> It looks like HadoopFileSystemFactory is created in
> [https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTableSink.java#L134]
> However, it cannot recognize s3 files system implementations which are based
> on org.apache.flink.core.fs.FileSystemFactory.
>
> {code:java}
> Caused by: java.io.IOException: No FileSystem for scheme: s3Caused by:
> java.io.IOException: No FileSystem for scheme: s3 at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> ~[hadoop-common-2.8.3.jar:?] at
> org.apache.flink.connectors.hive.HadoopFileSystemFactory.create(HadoopFileSystemFactory.java:46)
> ~[flink-connector-hive_2.11-1.11.1.jar:1.11.1] at
> org.apache.flink.table.filesystem.stream.StreamingFileCommitter.lambda$initializeState$0(Hive.java:125)
> ~[flink-table-blink_2.11-1.11.1.jar:1.11.1]
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
