[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768076#comment-15768076
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1663


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765799#comment-15765799
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang reopened a pull request:

https://github.com/apache/incubator-beam/pull/1663

[BEAM-79] merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1663


commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
Author: Jakob Homan 
Date:   2016-12-07T00:59:50Z

[BEAM-1099] Minor typos in KafkaIO

commit afedd68e806830549724dfc0f2565d756cc6b46d
Author: Davor Bonaci 
Date:   2016-12-07T01:03:54Z

This closes #1524

commit e8c9686a2e898d38afd692328eb171c542084747
Author: Pei He 
Date:   2016-11-23T23:59:56Z

[BEAM-1047] Add DataflowClient wrapper on top of JSON library.

commit ded58832ceaef487f4590d9396f09744288c955d
Author: Pei He 
Date:   2016-11-24T00:14:27Z

[Code Health] Remove redundant projectId from DataflowPipelineJob.

commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
Author: Pei He 
Date:   2016-11-28T19:47:42Z

[BEAM-1047] Update dataflow runner code to use DataflowClient wrapper.

commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
Author: bchambers 
Date:   2016-12-07T01:08:13Z

Closes #1434

commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

BigQueryIO.Read: support runtime options

commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
Author: Aljoscha Krettek 
Date:   2016-12-07T05:28:13Z

[FLINK-1102] Fix Aggregator Registration in Flink Batch Runner

commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
Author: Aljoscha Krettek 
Date:   2016-12-07T07:26:02Z

This closes #1530

commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T07:37:33Z

[BEAM-1094] Set test scope for Kafka IO and junit

commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
Author: Sela 
Date:   2016-12-07T08:31:38Z

This closes #1531

commit dce3a196a3a26fdd42225520faf3d9084ee48123
Author: Sela 
Date:   2016-12-07T09:20:07Z

[BEAM-329] Update Spark runner README.

commit 02bb8c375c48847b1686d70184fb194500a62e8c
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T11:51:09Z

[BEAM-329] This closes #1532

commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
Author: Kenneth Knowles 
Date:   2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn

commit 1526184ae8be1f8ae6863f830653204157a584cd
Author: Thomas Groh 
Date:   2016-12-07T16:51:02Z

This closes #1527

commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
Author: Kenneth Knowles 
Date:   2016-12-07T05:01:37Z

Port ParDoTest from OldDoFn to new DoFn

commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
Author: Kenneth Knowles 
Date:   2016-12-07T17:00:18Z

This closes #1529

commit 55d333bff68809ff1a9154491ace02d2d16e3b85
Author: Thomas Groh 
Date:   2016-12-05T22:29:05Z

Only provide expanded Inputs and Outputs

This removes PInput and POutput from the immediate API Surface of
TransformHierarchy.Node, and forces Pipeline Visitors to access only
the expanded version of the output.

This is part of the move towards the runner-agnostic representation of a
graph.

commit 5b31a369962907e257de8019fbf6cde4c615b1c0
Author: Thomas Groh 
Date:   2016-12-07T17:14:38Z

This closes #1511

commit 43fef2775145f67def3ab8a246ecca192a7d650b
Author: Dan Halperin 
Date:   2016-12-07T12:06:57Z

[BEAM-905] Add shading config to examples archetype and enable it for 

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764476#comment-15764476
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/1663


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764357#comment-15764357
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1663

[BEAM-79] merge master into gearpump-runner branch

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1663.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1663


commit 0af145bba721f7cdf3fbcd15330df433d5a8431d
Author: Davor Bonaci 
Date:   2016-12-07T01:00:31Z

This closes #1504

commit e9f254ef2769a082c7fbb500c1c28c6224ac5a7f
Author: Jakob Homan 
Date:   2016-12-07T00:59:50Z

[BEAM-1099] Minor typos in KafkaIO

commit afedd68e806830549724dfc0f2565d756cc6b46d
Author: Davor Bonaci 
Date:   2016-12-07T01:03:54Z

This closes #1524

commit e8c9686a2e898d38afd692328eb171c542084747
Author: Pei He 
Date:   2016-11-23T23:59:56Z

[BEAM-1047] Add DataflowClient wrapper on top of JSON library.

commit ded58832ceaef487f4590d9396f09744288c955d
Author: Pei He 
Date:   2016-11-24T00:14:27Z

[Code Health] Remove redundant projectId from DataflowPipelineJob.

commit ce03f30c1ee0b84ad2e7f10a6272ffb25548244a
Author: Pei He 
Date:   2016-11-28T19:47:42Z

[BEAM-1047] Update dataflow runner code to use DataflowClient wrapper.

commit b2b570f27808b1671bf6cd0fc60f874da671d4ca
Author: bchambers 
Date:   2016-12-07T01:08:13Z

Closes #1434

commit 0a2ed832ce5af7556db605e99b985ed4ffc1b152
Author: Sam McVeety 
Date:   2016-10-30T18:58:44Z

BigQueryIO.Read: support runtime options

commit 869b2710efdb90bc3ce5b6e9d4f3b49a3a804a63
Author: Aljoscha Krettek 
Date:   2016-12-07T05:28:13Z

[FLINK-1102] Fix Aggregator Registration in Flink Batch Runner

commit b41a46e86fd38c4a887f31bdf6cb75969f4750d3
Author: Aljoscha Krettek 
Date:   2016-12-07T07:26:02Z

This closes #1530

commit baf5e6bd9b1011f4c5c3974aa46393471b340c15
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T07:37:33Z

[BEAM-1094] Set test scope for Kafka IO and junit

commit 9ccf6dbea0d3807fef6a7c0432906fffd2b8ec3f
Author: Sela 
Date:   2016-12-07T08:31:38Z

This closes #1531

commit dce3a196a3a26fdd42225520faf3d9084ee48123
Author: Sela 
Date:   2016-12-07T09:20:07Z

[BEAM-329] Update Spark runner README.

commit 02bb8c375c48847b1686d70184fb194500a62e8c
Author: Jean-Baptiste Onofré 
Date:   2016-12-07T11:51:09Z

[BEAM-329] This closes #1532

commit b2d72237b592e1dcb5cca30f5cbc9a11d2890c0f
Author: Kenneth Knowles 
Date:   2016-12-06T23:20:28Z

Port most of DoFnRunner Javadoc to new DoFn

commit 1526184ae8be1f8ae6863f830653204157a584cd
Author: Thomas Groh 
Date:   2016-12-07T16:51:02Z

This closes #1527

commit 8e1e46e73edf9cce376ed7bd194db00edc3e60b4
Author: Kenneth Knowles 
Date:   2016-12-07T05:01:37Z

Port ParDoTest from OldDoFn to new DoFn

commit ae52ec1bc3f3120e9f8e150e50c18564055a467c
Author: Kenneth Knowles 
Date:   2016-12-07T17:00:18Z

This closes #1529

commit 55d333bff68809ff1a9154491ace02d2d16e3b85
Author: Thomas Groh 
Date:   2016-12-05T22:29:05Z

Only provide expanded Inputs and Outputs

This removes PInput and POutput from the immediate API Surface of
TransformHierarchy.Node, and forces Pipeline Visitors to access only
the expanded version of the output.

This is part of the move towards the runner-agnostic representation of a
graph.

commit 5b31a369962907e257de8019fbf6cde4c615b1c0
Author: Thomas Groh 
Date:   2016-12-07T17:14:38Z

This closes #1511

commit 43fef2775145f67def3ab8a246ecca192a7d650b
Author: 

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646372#comment-15646372
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1306


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634763#comment-15634763
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/1234


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621964#comment-15621964
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1234

[BEAM-79] port OldDoFn to new DoFn

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-v2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1234


commit ccee98f39236062dd501b63d9a89e88dd1fd47bd
Author: manuzhang 
Date:   2016-10-31T03:52:22Z

[BEAM-79] port OldDoFn to new DoFn




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608972#comment-15608972
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1193


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491929#comment-15491929
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/959


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491675#comment-15491675
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/959

[BEAM-79] upgrade surefire version to exclude tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #959


commit c879a7fa02259458029d8dca82ecaf80d36019b7
Author: manuzhang 
Date:   2016-09-14T22:52:09Z

[BEAM-79] upgrade surefire version to exclude tests




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489387#comment-15489387
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/956


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15489256#comment-15489256
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/956

[BEAM-79] fix integration-test failure

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump-runner-sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/956.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #956


commit 272fe9f6eb6c721a1d4a39cc816e3de9dc705373
Author: manuzhang 
Date:   2016-09-14T03:20:27Z

[BEAM-79] fix integration-test failure




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438364#comment-15438364
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/750


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-gearpump
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397251#comment-15397251
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/750

[BEAM-79] Merge branch 'master' into gearpump_runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner_sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/750.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #750


commit 34467f92d5a31b47f95b734c737fde0a8277311b
Author: Thomas Groh 
Date:   2016-07-15T17:51:24Z

Make TransformEvaluatorFactory reuse Explicit

Transform Evaluator Factories must be reused for the entire execution of
a Pipeline and must not be reused across pipelines.

Remove EvaluatorKey, and key explicitly by the transform application.

commit ad6ee01825740ee47f25ca036fa5f352375bbe6e
Author: Dan Halperin 
Date:   2016-07-19T06:40:13Z

Checkstyle: enforce package-info.java presence

Ignores tests and examples

commit c4ad11832235eef0b73d299cd02d1a224c130ece
Author: Dan Halperin 
Date:   2016-07-20T16:16:28Z

Closes #692

commit 6d7efe3df8cde590722d94441eca3922a3a67734
Author: Kenneth Knowles 
Date:   2016-07-20T17:30:55Z

This closes #666

commit 5f19e4caf207ff02d226fec2933be2f37ca66b4d
Author: Thomas Groh 
Date:   2016-06-30T17:06:52Z

Add withAllowedLateness with Closing Behavior to Window

This makes the static constructors for withAllowedLateness symmetric to
the PTransform builder methods. It also allows references to
Window#withAllowedLateness(Duration, ClosingBehavior).

commit f547f70e1c9535ff663d124f67c72c7ec2c55e9e
Author: Kenneth Knowles 
Date:   2016-07-20T17:40:13Z

This closes #567

commit 00195d2543eb347cc3669a4ac89e98da0bc4dca4
Author: Thomas Groh 
Date:   2016-06-28T22:44:49Z

Use the ParDo Application to Cache DoFns

A DoFn application is the scope of reuse.

Factor CloningThreadLocal as the top-level class instead of
SerializableCloningThreadLocalCacheLoader, and extract the Fn from the
AppliedPTransform when loading an absent element.

commit 436e4a34ebb222545cb03cb6d39ea4ca2d905254
Author: Kenneth Knowles 
Date:   2016-07-20T17:55:53Z

This closes #554

commit b240525affb205a83054577233f3a4a508fe1c72
Author: Dan Halperin 
Date:   2016-07-18T19:05:02Z

BigQueryIO: move to google-cloud-platform module

* Move package from io to io.gcp.bigquery
* Move from SDK core into GCP-IO module
* Fixup references and import orders
* Separate AvroUtils into generic AvroUtils and BigQueryAvroUtils
* Rewrite a unit test in sdk core to not depend on BigQueryIO
* Fixup Javadoc in SDK core that need not depend on BigQueryIO
* Make utility classes package-private

commit 7ec8781a2e18548a23c882329f0b50f7254202ec
Author: Dan Halperin 
Date:   2016-07-20T20:02:43Z

Closes #681

commit bdb65278873a5010a625dc6a569ba25b17374c06
Author: Kenneth Knowles 
Date:   2016-07-20T04:36:10Z

Add os-maven-plugin to Spark runner

commit 84332ee9716233af928e85c14c534714ab828531
Author: Chandni Singh 
Date:   2016-07-20T00:30:16Z

BEAM-372 verfify if a nested coder consumes bytes equal to encoded bytes

commit 6d5e8186a2da532eb1c29097bc1259a19d9f72c9
Author: Luke Cwik 
Date:   2016-07-21T13:34:52Z

[BEAM-372] added a test that verifies if a coder consumes bytes equal to 
encoded bytes

This closes #695

commit ae2144196c351cc2ee544e030d793929d8607696
Author: Dan Halperin 
Date:   2016-07-20T17:26:02Z

BigtableIO: upgrade to 0.9.1

* Use the uber jar
* Remove OS classifier mumbo jumbo
* Move common dependency versioning to 

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390464#comment-15390464
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/714


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390102#comment-15390102
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/714

[BEAM-79] Add Gearpump runner to runners parent pom.xml

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

I think this line got lost in the shuffle. It is why we aren't seeing the 
module in the Jenkins web UI, though the [tests are running and 
passing](https://builds.apache.org/view/Beam/job/beam_PostCommit_RunnableOnService_GearpumpLocal/).

R: @dhalperi 
CC: @manuzhang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam gearpump-module

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #714


commit f5e1abb61ea36866cd2d15be6e82c2b0785a4d9c
Author: Kenneth Knowles 
Date:   2016-07-22T19:50:43Z

Add Gearpump runner to runners parent pom.xml




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386813#comment-15386813
 ] 

ASF GitHub Bot commented on BEAM-79:


Github user manuzhang closed the pull request at:

https://github.com/apache/incubator-beam/pull/323


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296122#comment-15296122
 ] 

Jean-Baptiste Onofré commented on BEAM-79:
--

Starting to review and test. Thanks !

> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279595#comment-15279595
 ] 

ASF GitHub Bot commented on BEAM-79:


GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/323

[BEAM-79] add Gearpump runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR adds Gearpump runner to Beam meeting the goals of phase 1 in the 
[design 
document](https://docs.google.com/document/d/1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g/edit).

The Gearpump runner supports the following functionalities,

* Transformations: ParDo, GroupByKey, Flatten
* Windows: using Beam's window logic and code
* side outputs
* serialization/deserialization: using Gearpump's Kryo serializer
* sources: Beam's UnboundedSource
* message delivery guarantee: at-most-once
* tests: integration test for various translators

Here's a snapshot of running the following Beam example on Gearpump cluster

```java
PCollection> wordCounts =
p.apply(Read.from(new UnboundedTextSource()).named("WordStream"))
.apply(ParDo.of(new ExtractWordsFn()))

.apply(Window.into(FixedWindows.of(Duration.standardSeconds(10
.apply(Count.perElement());

wordCounts.apply(ParDo.of(new FormatAsStringFn()));
```


![snip20160511_4](https://cloud.githubusercontent.com/assets/1191767/15171197/fd6ffba0-177e-11e6-99a1-30c7c2597244.png)

Note that the Gearpump runner is still in early stage and lacking 
capabilities like trigger, side inputs, aggregator. However, I'd like to have 
the community to get a feel of what Gearpump is like, whether Beam and Gearpump 
go well, and gather ideas for improvements. 




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam gearpump_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/323.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #323


commit 73e5978f599bcf32ed8c2f1d54b6bd3bd8350092
Author: manuzhang 
Date:   2016-03-15T08:15:16Z

[BEAM-79] add Gearpump runner




> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229717#comment-15229717
 ] 

Jean-Baptiste Onofré commented on BEAM-79:
--

It sounds good to me.  Thanks Manu !

> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[Manu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15229667#comment-15229667
 ] 

Manu Zhang commented on BEAM-79:


Here is a [design document | 
https://drive.google.com/open?id=1nw64QUWVfT8L7FUprPGLEeNjSBpDMkn1otfLt2rHM5g]. 
Please kindly comment it. 

> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-79) Gearpump runner

[Frances Perry (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173229#comment-15173229
 ] 

Frances Perry commented on BEAM-79:
---

Happy to assign it to you, as you will clearly be the expert on Gearpump ;-)

But please note that Beam is still very much under construction and there are a 
number of breaking API changes likely in the near future. So please reach out 
before getting beyond the early design phase / determining how well the models 
align. If you haven't yet, I'd start with these resources: 
http://beam.incubator.apache.org/getting_started/


> Gearpump runner
> ---
>
> Key: BEAM-79
> URL: https://issues.apache.org/jira/browse/BEAM-79
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-ideas
>Reporter: Tyler Akidau
>Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF 
> (https://wiki.apache.org/incubator/GearpumpProposal). Appears to be a mix of 
> low-level primitives a la MillWheel, with some higher level primitives like 
> non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)