[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
todd updated FLINK-23725:
-------------------------
Description:
When the HDFS file is written, if the part file exists, only false will be
returned if the duplicate name fails.Whether to throw an exception that already
exists in the part, or print related logs.
```
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
public void commit() throws IOException {
final Path src = recoverable.tempFile();
final Path dest = recoverable.targetFile();
final long expectedLength = recoverable.offset();
try {
//always return false or ture
fs.rename(src, dest);
} catch (IOException e) {
throw new IOException(
"Committing file by rename failed: " + src + " to " + dest, e);
}
}
was:
When the HDFS file is written, if the part file exists, only false will be
returned if the duplicate name fails.Whether to throw an exception that already
exists in the part, or print related logs.
```
org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
public void commit() throws IOException {
final Path src = recoverable.tempFile();
final Path dest = recoverable.targetFile();
final long expectedLength = recoverable.offset();
final FileStatus srcStatus;
try {
srcStatus = fs.getFileStatus(src);
} catch (IOException e) {
throw new IOException("Cannot clean commit: Staging file does not exist.");
}
if (srcStatus.getLen() != expectedLength) {
// something was done to this file since the committer was created.
// this is not the "clean" case
throw new IOException("Cannot clean commit: File has trailing junk data.");
}
try {
// return false or ture
fs.rename(src, dest);
} catch (IOException e) {
throw new IOException(
"Committing file by rename failed: " + src + " to " + dest, e);
}
}
> HadoopFsCommitter， Prompt for file rename failure
> -------------------------------------------------
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility
> Affects Versions: 1.11.1, 1.12.1
> Reporter: todd
> Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
