[
https://issues.apache.org/jira/browse/BEAM-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376035#comment-15376035
]
ASF GitHub Bot commented on BEAM-434:
-------------------------------------
GitHub user tgroh opened a pull request:
https://github.com/apache/incubator-beam/pull/651
[BEAM-434] Control the number of output shards in the Direct Runner
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the PR title is formatted like:
`[BEAM-<Jira issue #>] Description of pull request`
- [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
- [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
number, if there is one.
- [ ] If this contribution is large, please file an Apache
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.txt).
---
Add a Write Override Factory that limits the number of shards is
unspecified. This ensures that we will not write an output file per-key
due to bundling.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tgroh/incubator-beam write_sharding
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/651.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #651
----
commit dfecd170d67c90ae20a4baed3986bd1c7116e79e
Author: Thomas Groh <[email protected]>
Date: 2016-07-13T21:26:10Z
Control the number of output shards in the Direct Runner
Add a Write Override Factory that limits the number of shards is
unspecified. This ensures that we will not write an output file per-key
due to bundling.
----
> Limit the number of output files a beam-examples execution writes
> -----------------------------------------------------------------
>
> Key: BEAM-434
> URL: https://issues.apache.org/jira/browse/BEAM-434
> Project: Beam
> Issue Type: Bug
> Components: examples-java
> Reporter: Amit Sela
> Assignee: Amit Sela
> Priority: Minor
>
> When using `TextIO.Write.to("/path/to/output")` without any restrictions on
> the number of shards, it might generate many output files (depending on your
> input), for WordCount for example, you'll get as many output files as unique
> words in your input.
> Since I think examples are expected to execute in a friendly manner to "see"
> what it does and not optimize for performance in some way, I suggest to use
> `withoutSharding()` when writing the example output to an output file.
> Examples I could find that behave this way:
> org.apache.beam.examples.WordCount
> org.apache.beam.examples.complete.TfIdf
> org.apache.beam.examples.cookbook.DeDupExample
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
