[
https://issues.apache.org/jira/browse/FLINK-11838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284339#comment-17284339
]
Xintong Song commented on FLINK-11838:
--------------------------------------
bq. perhaps we expose this chunk size as a Flink option to give the user some
control over the process
Sounds good to me.
bq. it seems to me that we can't safely delete any of the temporary blobs along
way, because it's possible that we might restore to a checkpoint prior to some
or all of the incremental compose operations having occurred
That's a good point. Seems we have to keep the temporary blobs anyway, and
composing blobs in advance only serves as a performance optimization at the
cost of more temporary storage space and system complexity.
Maybe for the first step, it's good enough to simply do all the composing and
deleting at the end. We can try to optimize it later if we indeed see a
performance problem in composing and deleting the temporary blobs.
> Create RecoverableWriter for GCS
> --------------------------------
>
> Key: FLINK-11838
> URL: https://issues.apache.org/jira/browse/FLINK-11838
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem
> Affects Versions: 1.8.0
> Reporter: Fokko Driesprong
> Assignee: Galen Warren
> Priority: Major
> Labels: pull-request-available, usability
> Fix For: 1.13.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> GCS supports the resumable upload which we can use to create a Recoverable
> writer similar to the S3 implementation:
> https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload
> After using the Hadoop compatible interface:
> https://github.com/apache/flink/pull/7519
> We've noticed that the current implementation relies heavily on the renaming
> of the files on the commit:
> https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L233-L259
> This is suboptimal on an object store such as GCS. Therefore we would like to
> implement a more GCS native RecoverableWriter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
