GitHub user jkff opened a pull request:
https://github.com/apache/beam/pull/4145
Many simplifications to WriteFiles
Commits explain what's going on. I recommend reviewing each commit
individually. Most importantly, this unifies windowed and unwindowed finalize
(this is the only meaningful change - everything else is just restructuring),
and refactors the transform into sub-transforms for better readability.
Many more simplifications are possible in WriteOperation/FileBasedSink
themselves, but I'll defer that to post-FileIO
https://github.com/apache/beam/pull/3817 (this PR can be reviewed in parallel
with that one)
R: @reuvenlax or feel free to reassign to anybody else.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkff/incubator-beam simplify-write-files
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/4145.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4145
----
commit d04e7e7b77cb407952ee83044ce60f571adaa6a2
Author: Eugene Kirpichov <ekirpic...@gmail.com>
Date: 2017-10-18T00:06:43Z
Merges Writer.openWindowed/Unwindowed and removes result of close()
commit 3f470c2066a4b4bc9ff535d9ebd0de6651027d6c
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-16T02:06:39Z
enforce fixed sharding
commit 9a6a0bef0ae0cba3440c65db91170462684d3a8b
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-16T02:25:14Z
non-null window/pane in FileResult
commit 1390eb6a5dcf35c2ba33d1a3308039fa7de3e126
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-16T02:58:28Z
remove ShardAssignment
commit 617bdf72b2374c13ab000368c3b9ea6e65c06b97
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-16T03:51:10Z
consolidates windowed/unwindowed finalize fns somewhat
commit a71233fd01012e6f40a7f4465bb1e6d123417ff5
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-16T04:19:09Z
Unifies windowed and unwindowed finalize.
commit 1aea7729b178903435ee03119599d80b99a11eef
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-17T20:25:45Z
Refactors WriteFiles into sub-transforms
commit 1d78970904fcc08a1c5e2ab9700dd958ce8f5c9a
Author: Eugene Kirpichov <kirpic...@google.com>
Date: 2017-11-17T20:38:15Z
Converts WriteFiles to AutoValue
----
---
