[
https://issues.apache.org/jira/browse/BEAM-11494?focusedWorklogId=553849&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-553849
]
ASF GitHub Bot logged work on BEAM-11494:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Feb/21 19:53
Start Date: 17/Feb/21 19:53
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #13558:
URL: https://github.com/apache/beam/pull/13558#issuecomment-780811179
Do other file systems have these retention policies? If not, is it safe to
just continue overwriting in the HDFS case?
On Wed, Feb 17, 2021 at 10:32 AM Pablo <[email protected]> wrote:
> No. This is to avoid overwriting files that were successfully written
> previously, but are part of a bundle that is being retried.
>
> We do this because some users are reporting issues where pipelines
> experience errors on retries for GCS buckets with specific retention
> policies (files cannot be deleted, and thus cannot be overwritten)
>
> —
> You are receiving this because your review was requested.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/13558#issuecomment-780759786>, or
> unsubscribe
>
<https://github.com/notifications/unsubscribe-auth/AFAYJVIHTFPPZXPZ4BN6TJ3S7QDSRANCNFSM4U5FVBKA>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 553849)
Time Spent: 3h 10m (was: 3h)
> FileIO.Write overwrites destination files on retries
> ----------------------------------------------------
>
> Key: BEAM-11494
> URL: https://issues.apache.org/jira/browse/BEAM-11494
> Project: Beam
> Issue Type: Improvement
> Components: io-java-files
> Reporter: Pablo Estrada
> Assignee: Pablo Estrada
> Priority: P2
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Users have reported cases of FileIO.Write becoming stuck or failing due to
> overwriting destination files.
> The failure/stuckness occurs because there are some file system buckets with
> strict retention policies that do not allow files to be deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
