[
https://issues.apache.org/jira/browse/BEAM-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767812#comment-15767812
]
Daniel Halperin commented on BEAM-1190:
---------------------------------------
Here are my general concerns:
* How do we distinguish this event from files getting deleted between expand
and read time, which is obviously an error.
* How do we ensure consistency? Aka, suppose I produce 3 shards of a file
returned by a list on an eventually-list-consistent filesystem, and when we try
to open them at runtime the file does not exist. How can we distinguish whether
the file was deleted before expand-glob, after expand-glob but before any
reading, after reading some shards but not others, etc?
* If the filesystem (or the part of the filesystem used by the user) is not
eventually consistent, this is clearly a bad thing to do. So this would somehow
have to be filesystem+glob aware.
So I would prefer a solution like giving [BEAM-60] which will give the user
code they can execute at expand-the-glob time. That's where the user can put in
the "stat the file and make sure it's there, or remove it" code, if they want
such.
For your proposal:
* Will it inspect the properties of the Filesystem to ensure it is eventually
consistent for the expanded glob?
* Do we plan to add a mandatory stat at glob-expand time? That seems like the
only correct place to add this logic -- otherwise, we can't tell from files
that legitimately did exist at expand time, but were deleted in error at
runtime.
* How do we plan to allow the user to opt into this behavior?
I think putting the user explicitly in the loop is the right thing to do,
either through one of the two JIRA issues I linked earlier. I am not yet
persuaded that changing default behavior is the right answer.
> FileBasedSource should ignore files that matched the glob but don't exist
> -------------------------------------------------------------------------
>
> Key: BEAM-1190
> URL: https://issues.apache.org/jira/browse/BEAM-1190
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Eugene Kirpichov
> Assignee: Eugene Kirpichov
>
> See user issue:
> http://stackoverflow.com/questions/41251741/coping-with-eventual-consistency-of-gcs-bucket-listing
> We should, after globbing the files in FileBasedSource, individually stat
> every file and remove those that don't exist, to account for the possibility
> that glob yielded non-existing files due to eventual consistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
