Gabor Somogyi created FLINK-36124:
-------------------------------------
Summary: S3RecoverableFsDataOutputStream.sync closes the stream
and further write operation throw exception
Key: FLINK-36124
URL: https://issues.apache.org/jira/browse/FLINK-36124
Project: Flink
Issue Type: Bug
Components: Connectors / AWS
Affects Versions: 2.0.0
Reporter: Gabor Somogyi
This behaviour is introduced in FLINK-28513.
Rationale why I think it's a bug:
* `sync` method is defined in `FSDataOutputStream` with the following
definition:
{code:java}
/**
* Flushes the data all the way to the persistent non-volatile storage (for
example disks). The
* method behaves similar to the <i>fsync</i> function, forcing all data to
be persistent on the
* devices.
*
* @throws IOException Thrown if an I/O error occurs
*/
{code}
* In case `sync` method call user of the writer instance is expected to call
further `write` methods
* What is actually happening it's blowing up the next write with the following
exception:
{code:java}
java.io.IOException: Stream closed.
at
org.apache.flink.core.fs.RefCountedFileWithStream.requireOpened(RefCountedFileWithStream.java:72)
at
org.apache.flink.core.fs.RefCountedFileWithStream.write(RefCountedFileWithStream.java:52)
at
org.apache.flink.core.fs.RefCountedBufferingFileStream.flush(RefCountedBufferingFileStream.java:104)
at
org.apache.flink.core.fs.RefCountedBufferingFileStream.write(RefCountedBufferingFileStream.java:87)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.write(S3RecoverableFsDataOutputStream.java:112)
at java.base/java.io.OutputStream.write(OutputStream.java:122)
{code}
* This can be super easily tested with
`S3RecoverableFsDataOutputStreamTest.testSync`. Please remove the `expected =
Exception.class` from the beginning of the test.
* The following line in the test is testing nothing because never ever called:
https://github.com/apache/flink/blob/56c81995d3b34ed9066b6771755407b93438f5ab/flink-filesystems/flink-s3-fs-base/src/test/java/org/apache/flink/fs/s3/common/writer/S3RecoverableFsDataOutputStreamTest.java#L264
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
