[
https://issues.apache.org/jira/browse/BEAM-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784832#comment-16784832
]
Eugene Kirpichov commented on BEAM-2857:
----------------------------------------
The WriteFiles transform is a leftover of pre-FileIO.write() file writing
functionality in Java, and I think there's even a JIRA somewhere out there to
get rid of it or at least rewrite FileIO.write() as an independent transform.
It does not make sense to reimplement WriteFiles in Python - much better to
implement FileIO.write independently. However, WriteFiles is a valuable source
of insight for how to structure the implementation as it addresses many very
subtle points related to data consistency, performance, batch/streaming
unification etc. - one *needs* to completely understand the whole
implementation of WriteFiles before embarking on an implementation of
FileIO.write.
> Create FileIO in Python
> -----------------------
>
> Key: BEAM-2857
> URL: https://issues.apache.org/jira/browse/BEAM-2857
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
> Reporter: Eugene Kirpichov
> Assignee: Pablo Estrada
> Priority: Major
> Labels: gsoc, gsoc2019, mentor, triaged
>
> Beam Java has a FileIO with operations: match()/matchAll(), readMatches(),
> which together cover the majority of needs for general-purpose file
> ingestion. Beam Python should have something similar.
> An early design document for this: https://s.apache.org/fileio-beam-python
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
