[
https://issues.apache.org/jira/browse/BEAM-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16768367#comment-16768367
]
Robert Bradshaw commented on BEAM-4822:
---------------------------------------
IMHO the MetadataCoder should be written to be easily extensible without
requiring a new version each time, but that seems orthogonal to the question
about reading previous versions of a file (unless this gets merged into some
watch transform that would get all versions then filter on their metadata or
something like that).
It seems that what one wants in this case is
{code:java}
TextIO.read().from("/filepattern/pattern", EXTRA_ARGUMENTS)
{code}
where EXTRA_ARGUMENTS may be filesystem-specific and gets passed to the
FileSystem object operations such as open, list, etc.The quick way to do this,
of course, is to embed it as part of the string which a particular FileSystem
would strip and interpret in its own special way, e.g.
{code:java}
TextIO.read().from("/filepattern/pattern@VERSION")
{code}
> Beam FileIO should support versioned file systems
> -------------------------------------------------
>
> Key: BEAM-4822
> URL: https://issues.apache.org/jira/browse/BEAM-4822
> Project: Beam
> Issue Type: New Feature
> Components: io-java-files
> Reporter: Reuven Lax
> Assignee: Chamikara Jayalath
> Priority: Major
> Labels: triaged
>
> Some file systems (e.g. GCS) are versioned, and support reading previous
> generations of files. Since Beam's file support does not currently support
> this concept, the latest versions of files are always the ones returned.
> Users should be able to specify that they want to read a previous version of
> a file in FileIO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
