[
https://issues.apache.org/jira/browse/FLINK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314079#comment-17314079
]
Galen Warren edited comment on FLINK-11838 at 4/2/21, 8:54 PM:
---------------------------------------------------------------
[~xintongsong] – actually, is that the right thing to do, start a new PR? I'm
thinking that might be the cleanest way to proceed, since we never really used
the old one that I created prematurely, and the info in the old one is out of
date, but I'll follow your guidance here on what to do.
Old PR is [here|https://github.com/apache/flink/pull/14875]
was (Author: galenwarren):
[~xintongsong] – actually, is that the right thing to do, start a new PR? I'm
thinking that might be the cleanest way to proceed, since we never really used
the old one that I created prematurely, and the info in the old one is out of
date, but I'll follow your guidance here on what to do.
> Create RecoverableWriter for GCS
> --------------------------------
>
> Key: FLINK-11838
> URL: https://issues.apache.org/jira/browse/FLINK-11838
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem
> Affects Versions: 1.8.0
> Reporter: Fokko Driesprong
> Assignee: Galen Warren
> Priority: Major
> Labels: pull-request-available, usability
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> GCS supports the resumable upload which we can use to create a Recoverable
> writer similar to the S3 implementation:
> https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
> After using the Hadoop compatible interface:
> https://github.com/apache/flink/pull/7519
> We've noticed that the current implementation relies heavily on the renaming
> of the files on the commit:
> https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L233-L259
> This is suboptimal on an object store such as GCS. Therefore we would like to
> implement a more GCS native RecoverableWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
