[
https://issues.apache.org/jira/browse/BEAM-4822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758551#comment-16758551
]
Chamikara Jayalath commented on BEAM-4822:
------------------------------------------
I think one of the issues is updates to current metadata coder [1] not being
backwards compatible when it comes to features such as dynamic pipeline updates
supported by Dataflow streaming. If we can use a coder that can be extended in
a backwards compatible way such as ProtoCoder or if we use schemas, probably we
can avoid having to introduce a new class for every update. [~reuvenlax] WDYT ?
[1]
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/fs/MetadataCoderV2.java
> Beam FileIO should support versioned file systems
> -------------------------------------------------
>
> Key: BEAM-4822
> URL: https://issues.apache.org/jira/browse/BEAM-4822
> Project: Beam
> Issue Type: New Feature
> Components: io-java-files
> Reporter: Reuven Lax
> Assignee: Chamikara Jayalath
> Priority: Major
>
> Some file systems (e.g. GCS) are versioned, and support reading previous
> generations of files. Since Beam's file support does not currently support
> this concept, the latest versions of files are always the ones returned.
> Users should be able to specify that they want to read a previous version of
> a file in FileIO.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
