[
https://issues.apache.org/jira/browse/BEAM-60?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17132187#comment-17132187
]
Beam JIRA Bot commented on BEAM-60:
-----------------------------------
This issue is P2 but has been unassigned without any comment for 60 days so it
has been labeled "stale-P2". If this issue is still affecting you, we care!
Please comment and remove the label. Otherwise, in 14 days the issue will be
moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed
explanation of what these priorities mean.
> FileBasedSource/IOChannelFactory: Custom glob expansion
> -------------------------------------------------------
>
> Key: BEAM-60
> URL: https://issues.apache.org/jira/browse/BEAM-60
> Project: Beam
> Issue Type: New Feature
> Components: sdk-java-core
> Reporter: Dan Halperin
> Priority: P2
> Labels: stale-P2
>
> Many cloud and distributed filesystems are eventually consistent, for
> instance Amazon s3 and Google Cloud Storage.
> To work around this, many systems that produce files such as Beam's
> FileBasedSinks, or Google BigQuery will provide methods to determine the
> number and set of files produced. E.g.,
> * Beam FileBasedSink uses -00000-of-NNNNN
> * BigQuery export jobs uses -000000 -000001 -000002 ... until an empty file
> is produced
> * Another system may produce a .filelist suffix that contains a list of all
> files.
> Users should be able to supply a glob to FileBasedSource but additionally
> supply a "glob expander" that can provide a custom implementation for file
> expansion. That way, e.g., Beam pipelines can be run back-to-back-to-back
> where each consumes the output of the previous, on an inconsistent
> filesystem, without data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
