GitHub user peihe opened a pull request:

    [BEAM-165] Initial implementation of the MapReduce runner.

    Follow this checklist to help us incorporate your contribution quickly and 
     - [ ] Make sure there is a [JIRA 
issue]( filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
     - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
     - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
     - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
     - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
     - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](

You can merge this pull request into a Git repository by running:

    $ git pull mr-runner

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3705
commit 9fffd554f1e5fd6465989bb3568dfb6f2d854eeb
Author: Pei He <>
Date:   2017-07-06T02:22:27Z

    Initial commit for MapReduceRunner.

commit 3bacc3e6099718bbcb672ab738ad607204fa8487
Author: Pei He <>
Date:   2017-07-11T02:45:11Z

    MapReduceRunner: add Graph and its visitors.

commit b62238545c1ba95e9857710d91609431cd0a2f93
Author: Pei He <>
Date:   2017-07-13T06:09:10Z

    MapReduceRunner: add unit tests for GraphConverter and GraphPlanner.

commit 64548dc949d0251949efdd02df68eed6032a64f4
Author: Pei He <>
Date:   2017-07-21T05:46:36Z

    mr-runner: support BoundedSource with BeamInputFormat.

commit 3070fded4bc0dde8f08b63e53f94342d21d4bc53
Author: Pei He <>
Date:   2017-07-24T12:15:37Z

    mr-runner: add JobPrototype and translate it to a MR job.

commit 0e16c52463278c6c4f9db61253c6b8287c4718ff
Author: Pei He <>
Date:   2017-07-25T13:44:34Z

    mr-runner: add ParDoOperation and support ParDos chaining.

commit 72a50aa508726e34110475448e9bb52381711faf
Author: Pei He <>
Date:   2017-07-26T13:19:30Z

    mr-runner: add BeamReducer and support GroupByKey.

commit 1b449b0981ae2bb2e1b397113b48eec1df53a4b1
Author: Pei He <>
Date:   2017-07-27T07:01:22Z

    core-java: InMemoryTimerInternals expose getTimers() for timer firings in 

commit 6d152a623550446b06bde91ad0c54df1f7e5c60b
Author: Pei He <>
Date:   2017-07-27T02:52:32Z

    mr-runner: support reduce side ParDos and WordCount.

commit 1ef0dec520ee301328007f99419c25b7a7b5b46f
Author: Pei He <>
Date:   2017-07-27T07:05:06Z

    mr-runner: add JarClassInstanceFactory to run ValidatesRunner tests.

commit 02c77375cc114a210f99079cf3efec3d2426941e
Author: Pei He <>
Date:   2017-07-28T08:31:41Z

    mr-runner: refactors and creates Graph data structures to handle general 
Beam pipelines.

commit bb3349e10c0cfacd81b610880ddfec030fedf34d
Author: Pei He <>
Date:   2017-08-02T11:19:14Z

    mr-runner: support graph visualization with dotfiles.

commit 0fd2f15847e1f9bdd42f4388f6de6e566f9b64ef
Author: Pei He <>
Date:   2017-08-02T13:59:21Z

    mr-runner: hack to get around that ViewAsXXX.expand() return wrong output 

commit 5079322c2e2a092a85b9740d04a7ca9bd887460e
Author: Pei He <>
Date:   2017-08-08T03:30:29Z

    mr-runner: support PCollections materialization with multiple MR jobs.

commit ad4cd2d5ea2af795bba86319d6447e7f8c415bf2
Author: Pei He <>
Date:   2017-08-08T07:49:04Z

    mr-runner: support multiple SourceOperations by composing and partitioning.

commit de2859e1092bfc3fdd036c3becf9e79fbb8fc8fa
Author: Pei He <>
Date:   2017-08-08T09:38:58Z

    mr-runner: support side inputs by reading in all views contents.

commit 69ee0f92bf170f0628d788d5dabeb339e7f1ad0c
Author: Pei He <>
Date:   2017-08-08T14:07:12Z

    mr-runner: setup file paths for read and write sides of materialization.


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

Reply via email to