Artsem Semianenka created FLINK-11302:
-----------------------------------------
Summary: FlinkS3FileSystem uses an incorrect path for temporary
files.
Key: FLINK-11302
URL: https://issues.apache.org/jira/browse/FLINK-11302
Project: Flink
Issue Type: Bug
Components: FileSystem
Affects Versions: 1.7.1, 1.7.0, 1.8.0
Reporter: Artsem Semianenka
Assignee: Artsem Semianenka
I'm running the Flink job which stores the parquet files on S3.
For that purpose, I use StreamingFileSink (in Bulk format) which under the hood
uses FlinkS3FileSystem from flink-s3-fs-hadoop-1.7.1.jar
When I try to submit it in Yarn cluster I got the following exception:
{code:java}
java.nio.file.NoSuchFileException:
/mnt/e1/yarn/nm/usercache/ec2-user/appcache/application_1544473029846_0487,/mnt/e2/yarn/nm/usercache/ec2-user/appcache/application_1544473029846_0487/.tmp_d7a04184-6fed-4c69-9d2f-f950e06bf5f0
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at
java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at
org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:95)
at
org.apache.flink.fs.s3.common.utils.RefCountedTmpFileCreator.apply(RefCountedTmpFileCreator.java:43)
at
org.apache.flink.fs.s3.common.utils.RefCountedBufferingFileStream.openNew(RefCountedBufferingFileStream.java:174)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.boundedBufferingFileStream(S3RecoverableFsDataOutputStream.java:271)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.newStream(S3RecoverableFsDataOutputStream.java:236)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableWriter.open(S3RecoverableWriter.java:85)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.rollPartFile(Bucket.java:221)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.write(Bucket.java:212)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.onElement(Buckets.java:268)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.invoke(StreamingFileSink.java:370)
at
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
....
{code}
As we can see the target path form stacktrace consist of two paths separated by
a comma
It happens because in
[AbstractS3FileSystemFactory|https://github.com/apache/flink/blob/bf5dc6c8bac73e7f8d54e983be2080cf6ce48a94/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/AbstractS3FileSystemFactory.java#L141]
we use the value of
[CoreOptions.TMP_DIRS|https://github.com/apache/flink/blob/43a74977de27869e41e97878e821dcf363d81911/flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java#L216]
"as is" and we expect that the value will contain only one path. But in
reality, it may contain more than one path.
Proposed solution: Split path by comma and "|" and use only the first path.
Bugfix already exist and I will create a pull request in which
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
