[
https://issues.apache.org/jira/browse/FLINK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-30049:
---------------------------------
Description:
{code:java}
Caused by: org.apache.flink.util.SerializedThrowable: Cannot sync state to
system like S3. Use persist() to create a persistent recoverable intermediate
point.
at
org.apache.flink.core.fs.RefCountedBufferingFileStream.sync(RefCountedBufferingFileStream.java:111)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.sync
at
org.apache.flink.formats.csv.CsvBulkWriter.finish(CsvBulkWriter.java:106)
at
org.apache.flink.connector.file.table.FileSystemTableSink$ProjectionBulkFactory$1.finish(FileSystemTableSink.java:653)
at
org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.closeForCommit(BulkPartWriter.java:64)
{code}
It looks like we should not call `sync` in CsvBulkWriter, we should just use
`flush`.
was:
{code:java}
Caused by: org.apache.flink.util.SerializedThrowable: Cannot sync state to
system like S3. Use persist() to create a persistent recoverable intermediate
point.
at
org.apache.flink.core.fs.RefCountedBufferingFileStream.sync(RefCountedBufferingFileStream.java:111)
at
org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.sync
at
org.apache.flink.formats.csv.CsvBulkWriter.finish(CsvBulkWriter.java:106)
at
org.apache.flink.connector.file.table.FileSystemTableSink$ProjectionBulkFactory$1.finish(FileSystemTableSink.java:653)
at
org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.closeForCommit(BulkPartWriter.java:64)
{code}
> CsvBulkWriter is unsupported for S3 FileSystem in streaming sink
> ----------------------------------------------------------------
>
> Key: FLINK-30049
> URL: https://issues.apache.org/jira/browse/FLINK-30049
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.16.0, 1.15.2
> Reporter: Jingsong Lee
> Priority: Major
>
> {code:java}
> Caused by: org.apache.flink.util.SerializedThrowable: Cannot sync state to
> system like S3. Use persist() to create a persistent recoverable intermediate
> point.
> at
> org.apache.flink.core.fs.RefCountedBufferingFileStream.sync(RefCountedBufferingFileStream.java:111)
>
> at
> org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.sync
> at
> org.apache.flink.formats.csv.CsvBulkWriter.finish(CsvBulkWriter.java:106)
> at
> org.apache.flink.connector.file.table.FileSystemTableSink$ProjectionBulkFactory$1.finish(FileSystemTableSink.java:653)
>
> at
> org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.closeForCommit(BulkPartWriter.java:64)
>
> {code}
> It looks like we should not call `sync` in CsvBulkWriter, we should just use
> `flush`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
