Liu created FLINK-28548:
---------------------------
Summary: The commit partition base path is not created when no
data is sent which may cause FileNotFoundException
Key: FLINK-28548
URL: https://issues.apache.org/jira/browse/FLINK-28548
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.15.1, 1.14.5, 1.16.0
Reporter: Liu
The commit partition base path is not created when no data is sent which may
cause FileNotFoundException. The exception is as following:
{code:java}
Caused by: java.io.FileNotFoundException: File
/home/ljg/test_sql.db/flink_batch_test/.staging_1657697612169 does not exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:771)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:120)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:828)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:824)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics$1.doCall(FileSystemLinkResolverWithStatistics.java:37)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at org.apache.hadoop.hdfs.perflog.PerfProxy.call(PerfProxy.java:49)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics.resolve(FileSystemLinkResolverWithStatistics.java:39)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:835)
~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238)
~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238)
~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.fs.viewfs.ChRootedFileSystem.listStatus(ChRootedFileSystem.java:241)
~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:376)
~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
at
org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170)
~[flink-sql-connector-hive-2.3.6_2.12-1.15.0.jar:1.15.0]
at
org.apache.flink.connector.file.table.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:87)
~[flink-connector-files-1.15.0.jar:1.15.0]
at
org.apache.flink.connector.file.table.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:78)
~[flink-connector-files-1.15.0.jar:1.15.0]
at
org.apache.flink.connector.file.table.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89)
~[flink-connector-files-1.15.0.jar:1.15.0]
at
org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:153)
~[flink-dist-1.15.0.jar:1.15.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.jobFinished(DefaultExecutionGraph.java:1190)
~[flink-dist-1.15.0.jar:1.15.0]
... 43 more {code}
We should check whether the base path exists before listStatus for the path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
