[jira] [Created] (BEAM-1946) Add IO module for restful api

[JiJun Tang (JIRA)]

JiJun Tang created BEAM-1946:


 Summary: Add IO module for restful api
 Key: BEAM-1946
 URL: https://issues.apache.org/jira/browse/BEAM-1946
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-extensions
Reporter: JiJun Tang
Assignee: Davor Bonaci


Create a RestIO for read or write data by resful api.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #208: Add JSON as a planned IO to built-in IOs page.

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam-site/pull/208

Add JSON as a planned IO to built-in IOs page.

R: @iemejia  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam-site add-json-to-io-page

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/208.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #208


commit d4e83a258f5d1ae85e2b1f8c6c98c1ddca70
Author: Zur, Aviem 
Date:   2017-04-12T05:22:09Z

Add JSON as a planned IO to built-in IOs page.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2813

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1944) Add Source Watermark Metrics in Spark runner

[Jingsong Lee (JIRA)]

Jingsong Lee created BEAM-1944:
--

 Summary: Add Source Watermark Metrics in Spark runner
 Key: BEAM-1944
 URL: https://issues.apache.org/jira/browse/BEAM-1944
 Project: Beam
  Issue Type: Sub-task
  Components: runner-spark
Reporter: Jingsong Lee






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1945) Add Source Watermark Metrics in Apex runner

[Jingsong Lee (JIRA)]

Jingsong Lee created BEAM-1945:
--

 Summary: Add Source Watermark Metrics in Apex runner
 Key: BEAM-1945
 URL: https://issues.apache.org/jira/browse/BEAM-1945
 Project: Beam
  Issue Type: Sub-task
  Components: runner-apex
Reporter: Jingsong Lee






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1943) Add Source Watermark Metrics in Dataflow runner

[Jingsong Lee (JIRA)]

Jingsong Lee created BEAM-1943:
--

 Summary: Add Source Watermark Metrics in Dataflow runner
 Key: BEAM-1943
 URL: https://issues.apache.org/jira/browse/BEAM-1943
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Jingsong Lee






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1942) Add Source Watermark Metrics in Flink Runner

[Jingsong Lee (JIRA)]

Jingsong Lee created BEAM-1942:
--

 Summary: Add Source Watermark Metrics in Flink Runner
 Key: BEAM-1942
 URL: https://issues.apache.org/jira/browse/BEAM-1942
 Project: Beam
  Issue Type: Sub-task
  Components: runner-flink
Reporter: Jingsong Lee






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1941) Add Source Watermark Metrics in Runners

[Jingsong Lee (JIRA)]

Jingsong Lee created BEAM-1941:
--

 Summary: Add Source Watermark Metrics in Runners
 Key: BEAM-1941
 URL: https://issues.apache.org/jira/browse/BEAM-1941
 Project: Beam
  Issue Type: New Feature
  Components: runner-ideas
Reporter: Jingsong Lee


The source watermark metrics show the consumer latency of Source. 
It allows the user to know the health of the job, or it can be used to monitor 
and alarm.
Since each runner is likely already tracking a watermark, another option here 
is to just have the runner report it appropriately, rather than having the 
source report it using metrics. This also addresses the fact that even if the 
source has advanced to 8:00, the runner may still know about buffered elements 
at 7:00, and so not advance the watermark all the way to 8:00. [~bchambers]
Includes:
1.Source watermark (`min` amongst all splits):
   type = Gauge, namespace = io, name = source_watermark
2.Source watermark per split:
   type = Gauge, namespace = io.splits, name = .source_watermark



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2504

[dhalperi]

This closes #2504


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/82f2f2cf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/82f2f2cf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/82f2f2cf

Branch: refs/heads/master
Commit: 82f2f2cff0f026800df465f1bff5451a7891ee13
Parents: ccf247c e7f4337
Author: Dan Halperin 
Authored: Tue Apr 11 18:34:14 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 11 18:34:14 2017 -0700

--
 .../java/org/apache/beam/sdk/transforms/windowing/Triggers.java| 2 ++
 1 file changed, 2 insertions(+)
--

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2811

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1799) IO ITs: simplify data loading design pattern

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965239#comment-15965239
 ] 

ASF GitHub Bot commented on BEAM-1799:
--

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2507

[BEAM-1799] JdbcIOIT now uses writeThenRead style

Removes JdbcTestDataSet's main, since it is no longer necessary for
loading data.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
I've now switched JdbcIOIT to do a write then a read. This code:
* creates a table during test class setup (this runs on the client)
* during the write test, generates data and populates data into that table
* reads the data from the table and verifies it via HashingFn + count

I really like the data generation - feels very clean, and I love the fact 
that using CountingSource means this part can be parallelized for larger data 
sets. I suspect that we may choose to refactor the "generate data" portion out 
into a helper method alongside HashingFn in io-common.

R @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam jdbc-it-writeThenRead

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2507


commit 26fb99021b1edd48f6660b95a106ecd9116c6f16
Author: Stephen Sisk 
Date:   2017-04-04T22:46:19Z

JdbcIOIT now uses writeThenRead style

Removes JdbcTestDataSet's main, since it is no longer necessary for
loading data.




> IO ITs: simplify data loading design pattern
> 
>
> Key: BEAM-1799
> URL: https://issues.apache.org/jira/browse/BEAM-1799
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Problems with the current solution
> =
> * The IO IT data loading guidelines [1] are complicated & aren't "native 
> junit" - you end up working around junit rather than working with it (I was a 
> part of defining them[0], so I critique the rules with (heart) )
> * Doing data loading using external tools means we have additional 
> dependencies outside of the tests themselves. If we *must* use them, it's 
> worth the time, but I think we have another option. I find it especially 
> amusing since the data loading tools are things like ycsb which themselves 
> are benchmarking tools ("I heard you like performance benchmarking, so here's 
> a performance benchmarking tool to use before you use your performance 
> benchmarking tool"), and really are just solving the problem of "I want to 
> write data in parallel to this data store" - that sounds familiar :) 
> The current guidelines also don't scale well to performance tests:
> * We want to write medium sized data for perf tests - doing data loading 
> using external tools means a minimum of 2 reads & writes. For the small scale 
> ITs, that's not a big deal, but for the large scale tests, if we assume we're 
> working with a fixed budget, more data transferred/stored ~= fewer tests.
> * If you want to verify that large data sets are correct (or create them), 
> you need to actually read and write those large data sets - currently, the 
> plan is that data loading/testing infrastructure only runs on one machine, so 
> those operations are going to be slow. We aren't working with actual large 
> data sets, so it won't take too long, but it's always nice to have faster 
> tests.
> New Proposed Solution
> ===
> Instead of trying to test read and write separately, the test should be a 
> "write, then read back what you just wrote", all using the IO under test. To 
> support scenarios like "I want to run my read test repeatedly without 
> re-writing the data", tests would add flags for "skipCleanUp" and 
> "useExistingData".
> Check out the example I wrote up [2]
> I didn't want to invest much time on this before I opened a Jira/talked to 
> others, so I plan on expanding on this a bit 

[GitHub] beam pull request #2507: [BEAM-1799] JdbcIOIT now uses writeThenRead style

[ssisk]

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2507

[BEAM-1799] JdbcIOIT now uses writeThenRead style

Removes JdbcTestDataSet's main, since it is no longer necessary for
loading data.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
I've now switched JdbcIOIT to do a write then a read. This code:
* creates a table during test class setup (this runs on the client)
* during the write test, generates data and populates data into that table
* reads the data from the table and verifies it via HashingFn + count

I really like the data generation - feels very clean, and I love the fact 
that using CountingSource means this part can be parallelized for larger data 
sets. I suspect that we may choose to refactor the "generate data" portion out 
into a helper method alongside HashingFn in io-common.

R @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam jdbc-it-writeThenRead

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2507


commit 26fb99021b1edd48f6660b95a106ecd9116c6f16
Author: Stephen Sisk 
Date:   2017-04-04T22:46:19Z

JdbcIOIT now uses writeThenRead style

Removes JdbcTestDataSet's main, since it is no longer necessary for
loading data.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1937) PipelineSurgery renumbers already-unique transforms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965231#comment-15965231
 ] 

ASF GitHub Bot commented on BEAM-1937:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2506

[BEAM-1937] Free PTransform Names if they are being Replaced

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Naming is based on what's in the graph, not what once was there.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam replacement_names_freed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2506


commit 9fbe5e384ee0b64ceb87a6851ae985802f8e5eb4
Author: Thomas Groh 
Date:   2017-04-12T01:17:02Z

Free PTransform Names if they are being Replaced

Naming is based on what's in the graph, not what once was there.




> PipelineSurgery renumbers already-unique transforms
> ---
>
> Key: BEAM-1937
> URL: https://issues.apache.org/jira/browse/BEAM-1937
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Thomas Groh
> Attachments: wordcount renumbered.png
>
>
> In the attached WordCount graph, it appears that some transforms have a 2 at 
> the end after submission. However, I'm pretty confident that there only 1 
> finalize and only 1 WriteBundles in this graph.
> [~tgroh] believes this is a bug in pipeline surgery.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1912) Move HashingFn into io/common so it can be used by other tests

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk resolved BEAM-1912.

   Resolution: Fixed
Fix Version/s: First stable release

> Move HashingFn into io/common so it can be used by other tests
> --
>
> Key: BEAM-1912
> URL: https://issues.apache.org/jira/browse/BEAM-1912
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>Priority: Minor
> Fix For: First stable release
>
>
> HadoopInputFormatIO has a hashing function that can be used to easily verify 
> reads - we should move it to a common place so other IOs can use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965150#comment-15965150
 ] 

ASF GitHub Bot commented on BEAM-115:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2505

[BEAM-115] Represent a Pipeline via a list of Top-level Transforms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
The root node is a synthetic transform which does not appear within the
graph, as it never has any components of note. Instead of referring to a
single "root node" in the Pipeline message, refer to the top-level nodes
which do not have an enclosing PTransform.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam top_level_transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2505


commit 31d5c272c4a6822ef35836be207224b557e122b4
Author: Thomas Groh 
Date:   2017-04-11T23:42:28Z

Represent a Pipeline via a list of Top-level Transforms

The root node is a synthetic transform which does not appear within the
graph, as it never has any components of note. Instead of referring to a
single "root node" in the Pipeline message, refer to the top-level nodes
which do not have an enclosing PTransform.




> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2505: [BEAM-115] Represent a Pipeline via a list of Top-l...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2505

[BEAM-115] Represent a Pipeline via a list of Top-level Transforms

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
The root node is a synthetic transform which does not appear within the
graph, as it never has any components of note. Instead of referring to a
single "root node" in the Pipeline message, refer to the top-level nodes
which do not have an enclosing PTransform.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam top_level_transforms

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2505


commit 31d5c272c4a6822ef35836be207224b557e122b4
Author: Thomas Groh 
Date:   2017-04-11T23:42:28Z

Represent a Pipeline via a list of Top-level Transforms

The root node is a synthetic transform which does not appear within the
graph, as it never has any components of note. Instead of referring to a
single "root node" in the Pipeline message, refer to the top-level nodes
which do not have an enclosing PTransform.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2504: Triggers: handle missing case

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2504

Triggers: handle missing case

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam reshuffle-trigger

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2504.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2504


commit e7f433720e229dcc296d279c459ed65c08fbb531
Author: Dan Halperin 
Date:   2017-04-11T23:26:41Z

Triggers: handle missing case




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1722) Get PubsubIO and the remaining parts of the GCP IOs out of the core SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965122#comment-15965122
 ] 

ASF GitHub Bot commented on BEAM-1722:
--

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/2503

[BEAM-1722] Move PubsubIO into the google-cloud-platform module

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1722-pubsub-out-of-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2503


commit 20466aa3de4dd50d47d83e34f7ea65e647bc61e6
Author: Ismaël Mejía 
Date:   2017-03-29T22:21:54Z

[BEAM-1722] Move PubsubIO into the google-cloud-platform module




> Get PubsubIO and the remaining parts of the GCP IOs out of the core SDK
> ---
>
> Key: BEAM-1722
> URL: https://issues.apache.org/jira/browse/BEAM-1722
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
> Fix For: First stable release
>
>
> As discussed with Davor, we still have to wait for some ongoing changes for 
> this, but I am going to create the JIRA to track this as an issue for the 
> first stable release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2503: [BEAM-1722] Move PubsubIO into the google-cloud-pla...

[iemejia]

GitHub user iemejia opened a pull request:

https://github.com/apache/beam/pull/2503

[BEAM-1722] Move PubsubIO into the google-cloud-platform module

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/iemejia/beam BEAM-1722-pubsub-out-of-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2503


commit 20466aa3de4dd50d47d83e34f7ea65e647bc61e6
Author: Ismaël Mejía 
Date:   2017-03-29T22:21:54Z

[BEAM-1722] Move PubsubIO into the google-cloud-platform module




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965116#comment-15965116
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2502

[BEAM-1928] Permit the Transform of an AppliedPTransform to be null

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This is the case if and only if the AppliedPTransform is the root node.
The root node will have an empty full name, an empty input, an empty
output, a null transform, and a Pipeline.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
applied_ptransform_nullable_transofrm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2502.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2502


commit 95318f807c43439a682558b229abf332a2102ab8
Author: Thomas Groh 
Date:   2017-04-11T23:22:45Z

Permit the Transform of an AppliedPTransform to be null

This is the case if and only if the AppliedPTransform is the root node.
The root node will have an empty full name, an empty input, an empty
output, a null transform, and a Pipeline.




> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2502: [BEAM-1928] Permit the Transform of an AppliedPTran...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2502

[BEAM-1928] Permit the Transform of an AppliedPTransform to be null

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This is the case if and only if the AppliedPTransform is the root node.
The root node will have an empty full name, an empty input, an empty
output, a null transform, and a Pipeline.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
applied_ptransform_nullable_transofrm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2502.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2502


commit 95318f807c43439a682558b229abf332a2102ab8
Author: Thomas Groh 
Date:   2017-04-11T23:22:45Z

Permit the Transform of an AppliedPTransform to be null

This is the case if and only if the AppliedPTransform is the root node.
The root node will have an empty full name, an empty input, an empty
output, a null transform, and a Pipeline.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2499: HIFIO Cassandra tests were failing if run twice in ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2499


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Clean HFIOWithEmbeddedCassandraTest before Execution

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a2d328a2d -> ccf247c1d


Clean HFIOWithEmbeddedCassandraTest before Execution

HIFIO Cassandra tests were failing if run twice in a row without a clean


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/041b636b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/041b636b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/041b636b

Branch: refs/heads/master
Commit: 041b636b0d912242ae1292c6fb2e5d0fa12fdaab
Parents: a2d328a
Author: Stephen Sisk 
Authored: Tue Apr 11 14:36:05 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 16:17:23 2017 -0700

--
 .../sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java   | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/041b636b/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
--
diff --git 
a/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
 
b/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
index 402c19c..50b2c4f 100644
--- 
a/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
+++ 
b/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
@@ -147,6 +147,7 @@ public class HIFIOWithEmbeddedCassandraTest implements 
Serializable {
   }
 
   public static void createCassandraData() throws Exception {
+session.execute("DROP KEYSPACE IF EXISTS " + CASSANDRA_KEYSPACE);
 session.execute("CREATE KEYSPACE " + CASSANDRA_KEYSPACE
 + " WITH REPLICATION = {'class':'SimpleStrategy', 
'replication_factor':1};");
 session.execute("USE " + CASSANDRA_KEYSPACE);

[2/2] beam git commit: This closes #2499

[tgroh]

This closes #2499


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ccf247c1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ccf247c1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ccf247c1

Branch: refs/heads/master
Commit: ccf247c1daa868cdaf441f918775c94ae6e3aa0b
Parents: a2d328a 041b636
Author: Thomas Groh 
Authored: Tue Apr 11 16:17:39 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 16:17:39 2017 -0700

--
 .../sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java   | 1 +
 1 file changed, 1 insertion(+)
--

[jira] [Assigned] (BEAM-1722) Get PubsubIO and the remaining parts of the GCP IOs out of the core SDK

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía reassigned BEAM-1722:
--

Assignee: Ismaël Mejía

> Get PubsubIO and the remaining parts of the GCP IOs out of the core SDK
> ---
>
> Key: BEAM-1722
> URL: https://issues.apache.org/jira/browse/BEAM-1722
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
> Fix For: First stable release
>
>
> As discussed with Davor, we still have to wait for some ongoing changes for 
> this, but I am going to create the JIRA to track this as an issue for the 
> first stable release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1938) Side Inputs should be part of the expanded inputs

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1938:
-

 Summary: Side Inputs should be part of the expanded inputs
 Key: BEAM-1938
 URL: https://issues.apache.org/jira/browse/BEAM-1938
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh


Required for the Java SDK to construct the runner API graphs without inspecting 
arbitrary transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1938) Side Inputs should be part of the expanded inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965072#comment-15965072
 ] 

ASF GitHub Bot commented on BEAM-1938:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2501

[BEAM-1938] Update Signature of PTransformOverrideFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This enables replacements to be reobtained with the entire transform
that is being replaced.

This is required when Side Inputs are part of the input of the
PTransform Application, as PTransforms are not applied to their side
inputs.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam override_factory_entire_context

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2501


commit caf32a9742fabf8e95631e018afa6f981d6f5249
Author: Thomas Groh 
Date:   2017-03-29T22:23:21Z

Update Signature of PTransformOverrideFactory

This enables replacements to be reobtained with the entire transform
that is being replaced.

This is required when Side Inputs are part of the input of the
PTransform Application, as PTransforms are not applied to their side
inputs.




> Side Inputs should be part of the expanded inputs
> -
>
> Key: BEAM-1938
> URL: https://issues.apache.org/jira/browse/BEAM-1938
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Required for the Java SDK to construct the runner API graphs without 
> inspecting arbitrary transforms.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2501: [BEAM-1938] Update Signature of PTransformOverrideF...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2501

[BEAM-1938] Update Signature of PTransformOverrideFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
This enables replacements to be reobtained with the entire transform
that is being replaced.

This is required when Side Inputs are part of the input of the
PTransform Application, as PTransforms are not applied to their side
inputs.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam override_factory_entire_context

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2501


commit caf32a9742fabf8e95631e018afa6f981d6f5249
Author: Thomas Groh 
Date:   2017-03-29T22:23:21Z

Update Signature of PTransformOverrideFactory

This enables replacements to be reobtained with the entire transform
that is being replaced.

This is required when Side Inputs are part of the input of the
PTransform Application, as PTransforms are not applied to their side
inputs.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2498

[dhalperi]

This closes #2498


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a2d328a2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a2d328a2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a2d328a2

Branch: refs/heads/master
Commit: a2d328a2d5d22def7901eb1d814118a36d58b5b7
Parents: 132d4c5 f473672
Author: Dan Halperin 
Authored: Tue Apr 11 15:39:45 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 11 15:39:45 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 16 
 1 file changed, 16 deletions(-)
--

[GitHub] beam pull request #2498: DataflowRunner: remove dead code

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2498


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: DataflowRunner: remove dead code

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 132d4c55d -> a2d328a2d


DataflowRunner: remove dead code

It is not possible to try to use an unbounded source in batch mode any more, as 
the
runner will automatically enable streaming mode.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f4736729
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f4736729
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f4736729

Branch: refs/heads/master
Commit: f4736729a5a04dc9d1cb30a44b2dd4692be7e935
Parents: 132d4c5
Author: Dan Halperin 
Authored: Tue Apr 11 14:12:43 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 11 15:39:40 2017 -0700

--
 .../beam/runners/dataflow/DataflowRunner.java   | 16 
 1 file changed, 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f4736729/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 1708d8a..8726635 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -39,7 +39,6 @@ import com.google.common.base.Strings;
 import com.google.common.base.Utf8;
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Iterables;
 import java.io.File;
 import java.io.IOException;
@@ -65,7 +64,6 @@ import 
org.apache.beam.runners.core.construction.PTransformMatchers;
 import org.apache.beam.runners.core.construction.ReplacementOutputs;
 import 
org.apache.beam.runners.core.construction.SingleInputOutputOverrideFactory;
 import 
org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource;
-import org.apache.beam.runners.core.construction.UnsupportedOverrideFactory;
 import 
org.apache.beam.runners.dataflow.BatchViewOverrides.BatchCombineGloballyAsSingletonViewFactory;
 import 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.JobSpecification;
 import 
org.apache.beam.runners.dataflow.StreamingViewOverrides.StreamingCreatePCollectionViewFactory;
@@ -330,14 +328,6 @@ public class DataflowRunner extends 
PipelineRunner {
   
PTransformMatchers.classEqualTo(View.CreatePCollectionView.class),
   new StreamingCreatePCollectionViewFactory()));
 } else {
-  // In batch mode must use the custom Pubsub bounded source/sink.
-  for (Class unsupported :
-  ImmutableSet.of(PubsubUnboundedSink.class, 
PubsubUnboundedSource.class)) {
-overridesBuilder.add(
-PTransformOverride.of(
-PTransformMatchers.classEqualTo(unsupported),
-
UnsupportedOverrideFactory.withMessage(getUnsupportedMessage(unsupported, 
false;
-  }
   overridesBuilder
   // State and timer pardos are implemented by expansion to 
GBK-then-ParDo
   .add(
@@ -399,12 +389,6 @@ public class DataflowRunner extends 
PipelineRunner {
 return overridesBuilder.build();
   }
 
-  private String getUnsupportedMessage(Class unsupported, boolean 
streaming) {
-return String.format(
-"%s is not supported in %s",
-NameUtils.approximateSimpleName(unsupported), streaming ? "streaming" 
: "batch");
-  }
-
   private static class ReflectiveOneToOneOverrideFactory<
   InputT extends PValue,
   OutputT extends PValue,

[GitHub] beam pull request #2496: Throw specialized exception in value providers

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2496


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2496

[altay]

This closes #2496


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/132d4c55
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/132d4c55
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/132d4c55

Branch: refs/heads/master
Commit: 132d4c55dac66f63adab0d32ad7dc53ffda82df9
Parents: 17a41ab ee92b96
Author: Ahmet Altay 
Authored: Tue Apr 11 15:32:21 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Apr 11 15:32:21 2017 -0700

--
 .../apache_beam/runners/dataflow/dataflow_runner.py   |  6 ++
 sdks/python/apache_beam/utils/value_provider.py   | 14 --
 2 files changed, 18 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-1196) Serialize/deserialize Pipeline/TransformHierarchy to Runner API Protos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965032#comment-15965032
 ] 

ASF GitHub Bot commented on BEAM-1196:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2495


> Serialize/deserialize Pipeline/TransformHierarchy to Runner API Protos
> --
>
> Key: BEAM-1196
> URL: https://issues.apache.org/jira/browse/BEAM-1196
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>
> The beam_runner_api.proto definition is a language-independent representation 
> of the Pipeline that SDKs are expected to be able to serialize a Pipeline in.
> These should be pushed all the way through and added to the Java SDK.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2495: [BEAM-1196] DataflowRunner: send windowing strategy...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2495


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2495

[dhalperi]

This closes #2495


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/17a41ab1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/17a41ab1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/17a41ab1

Branch: refs/heads/master
Commit: 17a41ab10cd841e495b8f9f634119f5e10be4ca9
Parents: bef2d37 69f412d
Author: Dan Halperin 
Authored: Tue Apr 11 15:06:29 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 11 15:06:29 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml| 12 +++-
 .../runners/dataflow/DataflowPipelineTranslator.java  | 14 --
 2 files changed, 23 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: DataflowRunner: send windowing strategy using Runner API proto

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master bef2d3738 -> 17a41ab10


DataflowRunner: send windowing strategy using Runner API proto


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69f412dc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69f412dc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69f412dc

Branch: refs/heads/master
Commit: 69f412dc34f36df40b034c2160b8b0cdad815011
Parents: bef2d37
Author: Dan Halperin <dhalp...@google.com>
Authored: Tue Apr 11 13:32:11 2017 -0700
Committer: Dan Halperin <dhalp...@google.com>
Committed: Tue Apr 11 15:04:38 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml| 12 +++-
 .../runners/dataflow/DataflowPipelineTranslator.java  | 14 --
 2 files changed, 23 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/69f412dc/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index d0d86e6..a57744c 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170410
+    
beam-master-20170411
 
1
 
6
   
@@ -114,6 +114,7 @@
 
   
com.google.cloud.bigtable:bigtable-client-core
   com.google.guava:guava
+  
org.apache.beam:beam-runners-core-construction-java
 
   
   
@@ -153,6 +154,10 @@
 
com.google.cloud.bigtable.grpc.BigtableTableName
   
 
+
+  org.apache.beam.runners.core
+  
org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core
+
   
   
 
@@ -178,6 +183,11 @@
 
 
   org.apache.beam
+  beam-sdks-common-runner-api
+
+
+
+  org.apache.beam
   beam-runners-core-construction-java
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/69f412dc/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
index 34da996..abeca4d 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java
@@ -55,6 +55,7 @@ import java.util.List;
 import java.util.Map;
 import java.util.concurrent.atomic.AtomicLong;
 import javax.annotation.Nullable;
+import org.apache.beam.runners.core.construction.WindowingStrategies;
 import 
org.apache.beam.runners.dataflow.BatchViewOverrides.GroupByKeyAndSortValuesOnly;
 import org.apache.beam.runners.dataflow.DataflowRunner.CombineGroupedValues;
 import 
org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory.ParDoSingle;
@@ -111,6 +112,15 @@ public class DataflowPipelineTranslator {
   private static final Logger LOG = 
LoggerFactory.getLogger(DataflowPipelineTranslator.class);
   private static final ObjectMapper MAPPER = new ObjectMapper();
 
+  private static byte[] serializeWindowingStrategy(WindowingStrategy 
windowingStrategy) {
+try {
+  return WindowingStrategies.toProto(windowingStrategy).toByteArray();
+} catch (Exception e) {
+  throw new RuntimeException(
+  String.format("Unable to format windowing strategy %s as bytes", 
windowingStrategy), e);
+}
+  }
+
   /**
* A map from {@link PTransform} subclass to the corresponding
* {@link TransformTranslator} to use to translate that transform.
@@ -813,7 +823,7 @@ public class DataflowPipelineTranslator {
 stepContext.addInput(PropertyNames.DISALLOW_COMBINER_LIFTING, 
disallowCombinerLifting);
 stepContext.addInput(
 PropertyNames.SERIALIZED_FN,
-
byteArrayToJsonString(serializeToByteArray(windowingStrategy)));
+
byteArrayToJsonString(serializeWindowingStrategy(windowingStrategy)));
 stepContext.addInput(
 PropertyNames.IS_MERGING_WINDOW_FN,
 !windowingStrategy.getWindowFn().isNonMerging());
@@ -891,7 +901,7 @@ public class DataflowPipelineTranslator {
 stepContext.addOutput(co

[GitHub] beam pull request #2497: DataflowRunner misc cleanups

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2497


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1937) PipelineSurgery renumbers already-unique transforms

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-1937:
-

 Summary: PipelineSurgery renumbers already-unique transforms
 Key: BEAM-1937
 URL: https://issues.apache.org/jira/browse/BEAM-1937
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Daniel Halperin
Assignee: Thomas Groh


In the attached WordCount graph, it appears that some transforms have a 2 at 
the end after submission. However, I'm pretty confident that there only 1 
finalize and only 1 WriteBundles in this graph.

[~tgroh] believes this is a bug in pipeline surgery.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2500: [BEAM-1915] Removes use of OldDoFn from Apex

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/2500

[BEAM-1915] Removes use of OldDoFn from Apex

This is the last occurrence of OldDoFn in the Beam repository outside of 
OldDoFn itself.

It's also used in the Dataflow worker, but it can be deleted entirely once 
we (Dataflow team) take care of that.

This is effectively inlining the (trivial) code of 
GroupAlsoByWindowViaWindowSetDoFn into ApexGroupByKeyOperator.

R: @tweise 
CC: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam apex-gbk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2500


commit 1830d4e8a2d96969fafd46241cb51fd893073a02
Author: Eugene Kirpichov 
Date:   2017-04-11T05:51:16Z

[BEAM-1915] Removes use of OldDoFn from Apex

This is the last occurrence of OldDoFn in the Beam repository
outside of OldDoFn itself.

It's also used in the Dataflow worker, but it can be
deleted entirely once we (Dataflow team) take care of that.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1915) Remove OldDoFn dependency in ApexGroupByKeyOperator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965009#comment-15965009
 ] 

ASF GitHub Bot commented on BEAM-1915:
--

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/2500

[BEAM-1915] Removes use of OldDoFn from Apex

This is the last occurrence of OldDoFn in the Beam repository outside of 
OldDoFn itself.

It's also used in the Dataflow worker, but it can be deleted entirely once 
we (Dataflow team) take care of that.

This is effectively inlining the (trivial) code of 
GroupAlsoByWindowViaWindowSetDoFn into ApexGroupByKeyOperator.

R: @tweise 
CC: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam apex-gbk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2500


commit 1830d4e8a2d96969fafd46241cb51fd893073a02
Author: Eugene Kirpichov 
Date:   2017-04-11T05:51:16Z

[BEAM-1915] Removes use of OldDoFn from Apex

This is the last occurrence of OldDoFn in the Beam repository
outside of OldDoFn itself.

It's also used in the Dataflow worker, but it can be
deleted entirely once we (Dataflow team) take care of that.




> Remove OldDoFn dependency in ApexGroupByKeyOperator
> ---
>
> Key: BEAM-1915
> URL: https://issues.apache.org/jira/browse/BEAM-1915
> Project: Beam
>  Issue Type: Task
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Eugene Kirpichov
>Priority: Minor
> Fix For: First stable release
>
>
> It is the last remaining occurrence in the ApexRunner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2499: HIFIO Cassandra tests were failing if run twice in ...

[ssisk]

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2499

HIFIO Cassandra tests were failing if run twice in a row without a clean

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
The embedded cassandra tests work correctly the first time they are run 
after a mvn clean - on any test phase execution of the jdk1.8-tests directory 
after that (ie, any time you run this unit tests again), they will fail. Thus, 
this passes our "mvn clean verify" runs and is not breaking CI, but will likely 
break any devs running tests in this directory locally without clean-ing. (aka, 
me)

R @dhalperi 
R @tgroh 
...whoever gets to this first

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam fix-cassandra-uts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2499.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2499


commit e72b13577d9d836aa7f0bc494725b1607aa8eb83
Author: Stephen Sisk 
Date:   2017-04-11T21:36:05Z

HIFIO Cassandra tests were failing if run twice in a row without a clean




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2809

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1936) Allow user provided function to extract custom timestamp from payload in pubsubIO

[Keith Berkoben (JIRA)]

Keith Berkoben created BEAM-1936:


 Summary: Allow user provided function to extract custom timestamp 
from payload in pubsubIO
 Key: BEAM-1936
 URL: https://issues.apache.org/jira/browse/BEAM-1936
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-gcp
Reporter: Keith Berkoben
Assignee: Daniel Halperin


Currently the PubsubIO runner only allows the caller to set a custom timestamp 
if the timestamp is defined in the attributes of the message.  This can be 
problematic when the user does not control the publisher.  In such a case, 
proper windowing of data requires the timestamp to be pulled out of the message 
payload.  

Since a payload could have arbitrary data, the user would have to provide a 
Function() that would extract the timestamp from the payload:

PubsubIo.Read.timestampLabel(Function extractor);



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2498: DataflowRunner: remove dead code

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2498

DataflowRunner: remove dead code

It is not possible to try to use an unbounded source in batch mode any 
more, as the
runner will automatically enable streaming mode.

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam delete-old-streaming

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2498.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2498


commit e6da75de9b3976b3fee34a176fb033a54db6c069
Author: Dan Halperin 
Date:   2017-04-11T21:12:43Z

DataflowRunner: remove dead code

It is not possible to try to use an unbounded source in batch mode any 
more, as the
runner will automatically enable streaming mode.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-184) Using Merging Windows and/or Triggers without a downstream aggregation should fail

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-184:


Assignee: (was: Thomas Groh)

> Using Merging Windows and/or Triggers without a downstream aggregation should 
> fail
> --
>
> Key: BEAM-184
> URL: https://issues.apache.org/jira/browse/BEAM-184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ben Chambers
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Both merging windows (such as sessions) and triggering only actually happen 
> at an aggregation (GroupByKey). We should produce errors in any of these 
> cases:
> 1. Merging window used without a downstream GroupByKey
> 2. Triggers used without a downstream GroupByKey
> 3. Window inspected after inserting a merging window and before the GroupByKey



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1508) PInput, POutput#expand should not be ordered

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1508.
---
   Resolution: Fixed
Fix Version/s: First stable release

> PInput, POutput#expand should not be ordered
> 
>
> Key: BEAM-1508
> URL: https://issues.apache.org/jira/browse/BEAM-1508
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> Ordering should be a local concern to how the PTransform is defined; the 
> outputs should have meaning independent of the order in which they are 
> enumerated.
> This also simplifies some implementation on the Runner-API side.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2497: DataflowRunner misc cleanups

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2497

DataflowRunner misc cleanups

Bad use of format string, very old debugging assert

R: @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam misc-cleanup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2497.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2497


commit ce43b7f4b6e6472a01f815a3dea31c0f9cdf7730
Author: Dan Halperin 
Date:   2017-04-11T20:47:34Z

DataflowRunner misc cleanups

Bad use of format string, very old debugging assert




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1935) DirectRunner Cancel should never throw a RejectedExecutionException

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1935.
---
   Resolution: Fixed
Fix Version/s: First stable release

> DirectRunner Cancel should never throw a RejectedExecutionException
> ---
>
> Key: BEAM-1935
> URL: https://issues.apache.org/jira/browse/BEAM-1935
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> cancelShouldStopPipeline(org.apache.beam.runners.direct.DirectRunnerTest)  
> Time elapsed: 0.025 sec  <<< ERROR!
> java.util.concurrent.RejectedExecutionException: Task 
> java.util.concurrent.FutureTask@391aff79 rejected from 
> java.util.concurrent.ThreadPoolExecutor@3d257c82[Shutting down, pool size = 
> 1, active threads = 1, queue
> d tasks = 0, completed tasks = 0]
> at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at 
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at 
> org.apache.beam.runners.direct.TransformExecutorServices$ParallelEvaluationState.schedule(TransformExecutorServices.java:69)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.evaluateBundle(ExecutorServiceParallelExecutor.java:245)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.scheduleConsumption(ExecutorServiceParallelExecutor.java:208)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1935) DirectRunner Cancel should never throw a RejectedExecutionException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964938#comment-15964938
 ] 

ASF GitHub Bot commented on BEAM-1935:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2493


> DirectRunner Cancel should never throw a RejectedExecutionException
> ---
>
> Key: BEAM-1935
> URL: https://issues.apache.org/jira/browse/BEAM-1935
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> cancelShouldStopPipeline(org.apache.beam.runners.direct.DirectRunnerTest)  
> Time elapsed: 0.025 sec  <<< ERROR!
> java.util.concurrent.RejectedExecutionException: Task 
> java.util.concurrent.FutureTask@391aff79 rejected from 
> java.util.concurrent.ThreadPoolExecutor@3d257c82[Shutting down, pool size = 
> 1, active threads = 1, queue
> d tasks = 0, completed tasks = 0]
> at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at 
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at 
> org.apache.beam.runners.direct.TransformExecutorServices$ParallelEvaluationState.schedule(TransformExecutorServices.java:69)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.evaluateBundle(ExecutorServiceParallelExecutor.java:245)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.scheduleConsumption(ExecutorServiceParallelExecutor.java:208)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2493: [BEAM-1935] Improve Work Rejection handling

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2493


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2493

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 7c169a614 -> 7587d29f5


This closes #2493


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7587d29f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7587d29f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7587d29f

Branch: refs/heads/master
Commit: 7587d29f50654397f00a754596bfc5a225d4a64f
Parents: 7c169a6 db81205
Author: Thomas Groh 
Authored: Tue Apr 11 13:44:34 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 13:44:34 2017 -0700

--
 .../direct/TransformExecutorServices.java   | 37 +++
 .../direct/TransformExecutorServicesTest.java   | 48 
 2 files changed, 77 insertions(+), 8 deletions(-)
--

[2/2] beam git commit: Improve Work Rejection handling

[tgroh]

Improve Work Rejection handling

The timing between checking for a shutdown state and submitting work is
racy. Re-check to see if a work rejection is acceptable because the
underlying executor is shut down before throwing a
RejectedExecutionException.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/db81205a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/db81205a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/db81205a

Branch: refs/heads/master
Commit: db81205af9b38245ff85ce2801af433cc31bce42
Parents: 7c169a6
Author: Thomas Groh 
Authored: Tue Apr 11 11:33:53 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 13:44:34 2017 -0700

--
 .../direct/TransformExecutorServices.java   | 37 +++
 .../direct/TransformExecutorServicesTest.java   | 48 
 2 files changed, 77 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/db81205a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
index 6733758..53087bf 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TransformExecutorServices.java
@@ -21,8 +21,11 @@ import com.google.common.base.MoreObjects;
 import java.util.Queue;
 import java.util.concurrent.ConcurrentLinkedQueue;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.RejectedExecutionException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Static factory methods for constructing instances of {@link 
TransformExecutorService}.
@@ -37,7 +40,7 @@ final class TransformExecutorServices {
* parallel.
*/
   public static TransformExecutorService parallel(ExecutorService executor) {
-return new ParallelEvaluationState(executor);
+return new ParallelTransformExecutor(executor);
   }
 
   /**
@@ -45,7 +48,7 @@ final class TransformExecutorServices {
* serial.
*/
   public static TransformExecutorService serial(ExecutorService executor) {
-return new SerialEvaluationState(executor);
+return new SerialTransformExecutor(executor);
   }
 
   /**
@@ -55,18 +58,36 @@ final class TransformExecutorServices {
* A principal use of this is for the evaluation of an unkeyed Step. 
Unkeyed computations are
* processed in parallel.
*/
-  private static class ParallelEvaluationState implements 
TransformExecutorService {
+  private static class ParallelTransformExecutor implements 
TransformExecutorService {
+private static final Logger LOG = 
LoggerFactory.getLogger(ParallelTransformExecutor.class);
+
 private final ExecutorService executor;
 private final AtomicBoolean active = new AtomicBoolean(true);
 
-private ParallelEvaluationState(ExecutorService executor) {
+private ParallelTransformExecutor(ExecutorService executor) {
   this.executor = executor;
 }
 
 @Override
 public void schedule(TransformExecutor work) {
   if (active.get()) {
-executor.submit(work);
+try {
+  executor.submit(work);
+} catch (RejectedExecutionException rejected) {
+  boolean stillActive = active.get();
+  if (stillActive) {
+throw new IllegalStateException(
+String.format(
+"Execution of Work %s was rejected, but the %s is still 
active",
+work, ParallelTransformExecutor.class.getSimpleName()));
+  } else {
+LOG.debug(
+"Rejected execution of Work {} on executor {}. "
++ "Suppressed exception because evaluator is not active",
+work,
+this);
+  }
+}
   }
 }
 
@@ -88,14 +109,14 @@ final class TransformExecutorServices {
* A principal use of this is for the serial evaluation of a (Step, Key) 
pair.
* Keyed computations are processed serially per step.
*/
-  private static class SerialEvaluationState implements 
TransformExecutorService {
+  private static class SerialTransformExecutor implements 
TransformExecutorService {
 private final ExecutorService executor;
 
 private AtomicReference currentlyEvaluating;
 private final Queue workQueue;
 private boolean active = 

[GitHub] beam pull request #2496: Throw specialized exception in value providers

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2496

Throw specialized exception in value providers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @mariapython @chamikaramj PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam 
BEAM-change-runtime-valueprovider-exception

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2496.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2496


commit 609db12850732e1689084fbca41e4ab6b47f16de
Author: Sourabh Bajaj 
Date:   2017-04-11T20:38:26Z

Throw specialized exception in value providers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2495: DataflowRunner: send windowing strategy using Runne...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2495

DataflowRunner: send windowing strategy using Runner API proto



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam proto-runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2495.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2495


commit 0f49e4b5fd36087fe6f6d4056e229e7072b1a3ab
Author: Dan Halperin 
Date:   2017-04-11T20:32:11Z

DataflowRunner: send windowing strategy using Runner API proto




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2808

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1508) PInput, POutput#expand should not be ordered

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964874#comment-15964874
 ] 

ASF GitHub Bot commented on BEAM-1508:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2430


> PInput, POutput#expand should not be ordered
> 
>
> Key: BEAM-1508
> URL: https://issues.apache.org/jira/browse/BEAM-1508
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Ordering should be a local concern to how the PTransform is defined; the 
> outputs should have meaning independent of the order in which they are 
> enumerated.
> This also simplifies some implementation on the Runner-API side.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[3/4] beam git commit: Remove Orderdness of Input, Output expansions

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/0e5737fd/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
index 8b4573f..b44c890 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/DirectGraphVisitorTest.java
@@ -45,7 +45,6 @@ import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.PInput;
 import org.apache.beam.sdk.values.POutput;
-import org.apache.beam.sdk.values.TaggedPValue;
 import org.hamcrest.Matchers;
 import org.junit.Rule;
 import org.junit.Test;
@@ -101,9 +100,9 @@ public class DirectGraphVisitorTest implements Serializable 
{
 graph.getProducer(created), graph.getProducer(counted), 
graph.getProducer(unCounted)));
 for (AppliedPTransform root : graph.getRootTransforms())  {
   // Root transforms will have no inputs
-  assertThat(root.getInputs(), emptyIterable());
+  assertThat(root.getInputs().entrySet(), emptyIterable());
   assertThat(
-  Iterables.getOnlyElement(root.getOutputs()).getValue(),
+  Iterables.getOnlyElement(root.getOutputs().values()),
   Matchers.isOneOf(created, counted, unCounted));
 }
   }
@@ -121,7 +120,7 @@ public class DirectGraphVisitorTest implements Serializable 
{
 Matchers.containsInAnyOrder(graph.getProducer(empty)));
 AppliedPTransform onlyRoot = 
Iterables.getOnlyElement(graph.getRootTransforms());
 assertThat(onlyRoot.getTransform(), Matchers.equalTo(flatten));
-assertThat(onlyRoot.getInputs(), Matchers.emptyIterable());
+assertThat(onlyRoot.getInputs().entrySet(), emptyIterable());
 assertThat(onlyRoot.getOutputs(), equalTo(empty.expand()));
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/0e5737fd/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
index c85b85e..2a94d48 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/ParDoEvaluatorTest.java
@@ -162,7 +162,7 @@ public class ParDoEvaluatorTest {
 evaluationContext,
 stepContext,
 transform,
-((PCollection) 
Iterables.getOnlyElement(transform.getInputs()).getValue())
+((PCollection) 
Iterables.getOnlyElement(transform.getInputs().values()))
 .getWindowingStrategy(),
 fn,
 null /* key */,

http://git-wip-us.apache.org/repos/asf/beam/blob/0e5737fd/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
index 946cd69..ecb8130 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/StatefulParDoEvaluatorFactoryTest.java
@@ -308,7 +308,7 @@ public class StatefulParDoEvaluatorFactoryTest implements 
Serializable {
 BUNDLE_FACTORY
 .createBundle(
 (PCollection>>)
-
Iterables.getOnlyElement(producingTransform.getInputs()).getValue())
+
Iterables.getOnlyElement(producingTransform.getInputs().values()))
 .add(gbkOutputElement)
 .commit(Instant.now());
 TransformEvaluator>> evaluator =

http://git-wip-us.apache.org/repos/asf/beam/blob/0e5737fd/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactoryTest.java
index fc689fe..0d909c2 100644
--- 

[GitHub] beam pull request #2430: [BEAM-1508] Remove Orderdness of Input, Output expa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2430


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/4] beam git commit: This closes #2430

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a5a5bf946 -> 7c169a614


This closes #2430


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c169a61
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c169a61
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c169a61

Branch: refs/heads/master
Commit: 7c169a614f339ed829c6d8b6bbc5c4868ff33ef5
Parents: a5a5bf9 0e5737f
Author: Thomas Groh 
Authored: Tue Apr 11 12:40:46 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 12:40:46 2017 -0700

--
 .../FlattenPCollectionTranslator.java   |  13 +-
 .../apex/translation/ParDoTranslator.java   |  13 +-
 .../apex/translation/TranslationContext.java|  10 +-
 .../DeduplicatedFlattenFactory.java |  18 ++-
 .../EmptyFlattenAsCreateFactory.java|   7 +-
 .../core/construction/PTransformMatchers.java   |   5 +-
 .../core/construction/PrimitiveCreate.java  |   7 +-
 .../core/construction/ReplacementOutputs.java   |  63 -
 .../SingleInputOutputOverrideFactory.java   |  11 +-
 .../UnsupportedOverrideFactory.java |   8 +-
 .../DeduplicatedFlattenFactoryTest.java |   6 +-
 .../EmptyFlattenAsCreateFactoryTest.java|   8 +-
 .../construction/PTransformMatchersTest.java| 132 +--
 .../construction/ReplacementOutputsTest.java| 109 ++-
 .../SingleInputOutputOverrideFactoryTest.java   |   6 +-
 .../UnsupportedOverrideFactoryTest.java |   7 +-
 .../direct/BoundedReadEvaluatorFactory.java |   2 +-
 .../beam/runners/direct/DirectGraphVisitor.java |   5 +-
 .../direct/ExecutorServiceParallelExecutor.java |   4 +-
 .../runners/direct/FlattenEvaluatorFactory.java |   2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 +-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   4 +-
 .../direct/KeyedPValueTrackingVisitor.java  |  14 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  10 +-
 .../direct/ParDoMultiOverrideFactory.java   |   9 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   8 +-
 .../direct/TestStreamEvaluatorFactory.java  |   8 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |   4 +-
 .../runners/direct/ViewEvaluatorFactory.java|   4 +-
 .../runners/direct/ViewOverrideFactory.java |   9 +-
 .../beam/runners/direct/WatermarkManager.java   |  19 +--
 .../runners/direct/WindowEvaluatorFactory.java  |   2 +-
 .../direct/WriteWithShardingFactory.java|  10 +-
 .../runners/direct/DirectGraphVisitorTest.java  |   7 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |   2 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |   2 +-
 .../flink/FlinkBatchTransformTranslators.java   |  36 ++---
 .../flink/FlinkBatchTranslationContext.java |  11 +-
 .../flink/FlinkStreamingPipelineTranslator.java |   9 +-
 .../FlinkStreamingTransformTranslators.java |  32 ++---
 .../flink/FlinkStreamingTranslationContext.java |  12 +-
 .../dataflow/BatchStatefulParDoOverrides.java   |  15 +--
 .../dataflow/DataflowPipelineTranslator.java|  20 +--
 .../beam/runners/dataflow/DataflowRunner.java   |  27 ++--
 .../runners/dataflow/TransformTranslator.java   |   6 +-
 .../dataflow/DataflowPipelineJobTest.java   |   7 +-
 .../apache/beam/runners/spark/SparkRunner.java  |  20 +--
 .../beam/runners/spark/TestSparkRunner.java |   7 +-
 .../spark/translation/EvaluationContext.java|  11 +-
 .../spark/translation/TransformTranslator.java  |  25 ++--
 .../streaming/StreamingTransformTranslator.java |  20 +--
 .../sdk/runners/PTransformOverrideFactory.java  |  10 +-
 .../beam/sdk/runners/TransformHierarchy.java|  80 +--
 .../beam/sdk/transforms/AppliedPTransform.java  |  17 ++-
 .../transforms/join/KeyedPCollectionTuple.java  |  12 +-
 .../java/org/apache/beam/sdk/values/PBegin.java |   6 +-
 .../apache/beam/sdk/values/PCollectionList.java |  27 ++--
 .../beam/sdk/values/PCollectionTuple.java   |  10 +-
 .../java/org/apache/beam/sdk/values/PDone.java  |   6 +-
 .../java/org/apache/beam/sdk/values/PInput.java |   4 +-
 .../org/apache/beam/sdk/values/POutput.java |   4 +-
 .../java/org/apache/beam/sdk/values/PValue.java |   4 +-
 .../org/apache/beam/sdk/values/PValueBase.java  |   6 +-
 .../apache/beam/sdk/values/TaggedPValue.java|   5 +
 .../java/org/apache/beam/sdk/PipelineTest.java  |  20 ++-
 .../sdk/runners/TransformHierarchyTest.java |  72 +-
 .../beam/sdk/values/PCollectionListTest.java|  70 --
 .../beam/sdk/values/PCollectionTupleTest.java   |   5 +-
 .../beam/sdk/io/gcp/bigquery/WriteResult.java   |  11 +-
 71 files changed, 526 insertions(+), 658 deletions(-)

[4/4] beam git commit: Remove Orderdness of Input, Output expansions

[tgroh]

Remove Orderdness of Input, Output expansions

This brings the PInput and POutput expansion signatures back in line
with the Runner API.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0e5737fd
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0e5737fd
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0e5737fd

Branch: refs/heads/master
Commit: 0e5737fdbee5478ee7f39c4b1a1ac95353ec7b08
Parents: a5a5bf9
Author: Thomas Groh 
Authored: Tue Apr 4 16:51:55 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 12:40:46 2017 -0700

--
 .../FlattenPCollectionTranslator.java   |  13 +-
 .../apex/translation/ParDoTranslator.java   |  13 +-
 .../apex/translation/TranslationContext.java|  10 +-
 .../DeduplicatedFlattenFactory.java |  18 ++-
 .../EmptyFlattenAsCreateFactory.java|   7 +-
 .../core/construction/PTransformMatchers.java   |   5 +-
 .../core/construction/PrimitiveCreate.java  |   7 +-
 .../core/construction/ReplacementOutputs.java   |  63 -
 .../SingleInputOutputOverrideFactory.java   |  11 +-
 .../UnsupportedOverrideFactory.java |   8 +-
 .../DeduplicatedFlattenFactoryTest.java |   6 +-
 .../EmptyFlattenAsCreateFactoryTest.java|   8 +-
 .../construction/PTransformMatchersTest.java| 132 +--
 .../construction/ReplacementOutputsTest.java| 109 ++-
 .../SingleInputOutputOverrideFactoryTest.java   |   6 +-
 .../UnsupportedOverrideFactoryTest.java |   7 +-
 .../direct/BoundedReadEvaluatorFactory.java |   2 +-
 .../beam/runners/direct/DirectGraphVisitor.java |   5 +-
 .../direct/ExecutorServiceParallelExecutor.java |   4 +-
 .../runners/direct/FlattenEvaluatorFactory.java |   2 +-
 .../GroupAlsoByWindowEvaluatorFactory.java  |   2 +-
 .../direct/GroupByKeyOnlyEvaluatorFactory.java  |   4 +-
 .../direct/KeyedPValueTrackingVisitor.java  |  14 +-
 .../runners/direct/ParDoEvaluatorFactory.java   |  10 +-
 .../direct/ParDoMultiOverrideFactory.java   |   9 +-
 .../direct/StatefulParDoEvaluatorFactory.java   |   8 +-
 .../direct/TestStreamEvaluatorFactory.java  |   8 +-
 .../direct/UnboundedReadEvaluatorFactory.java   |   4 +-
 .../runners/direct/ViewEvaluatorFactory.java|   4 +-
 .../runners/direct/ViewOverrideFactory.java |   9 +-
 .../beam/runners/direct/WatermarkManager.java   |  19 +--
 .../runners/direct/WindowEvaluatorFactory.java  |   2 +-
 .../direct/WriteWithShardingFactory.java|  10 +-
 .../runners/direct/DirectGraphVisitorTest.java  |   7 +-
 .../beam/runners/direct/ParDoEvaluatorTest.java |   2 +-
 .../StatefulParDoEvaluatorFactoryTest.java  |   2 +-
 .../direct/TestStreamEvaluatorFactoryTest.java  |   5 +-
 .../runners/direct/ViewOverrideFactoryTest.java |   2 +-
 .../flink/FlinkBatchTransformTranslators.java   |  36 ++---
 .../flink/FlinkBatchTranslationContext.java |  11 +-
 .../flink/FlinkStreamingPipelineTranslator.java |   9 +-
 .../FlinkStreamingTransformTranslators.java |  32 ++---
 .../flink/FlinkStreamingTranslationContext.java |  12 +-
 .../dataflow/BatchStatefulParDoOverrides.java   |  15 +--
 .../dataflow/DataflowPipelineTranslator.java|  20 +--
 .../beam/runners/dataflow/DataflowRunner.java   |  27 ++--
 .../runners/dataflow/TransformTranslator.java   |   6 +-
 .../dataflow/DataflowPipelineJobTest.java   |   7 +-
 .../apache/beam/runners/spark/SparkRunner.java  |  20 +--
 .../beam/runners/spark/TestSparkRunner.java |   7 +-
 .../spark/translation/EvaluationContext.java|  11 +-
 .../spark/translation/TransformTranslator.java  |  25 ++--
 .../streaming/StreamingTransformTranslator.java |  20 +--
 .../sdk/runners/PTransformOverrideFactory.java  |  10 +-
 .../beam/sdk/runners/TransformHierarchy.java|  80 +--
 .../beam/sdk/transforms/AppliedPTransform.java  |  17 ++-
 .../transforms/join/KeyedPCollectionTuple.java  |  12 +-
 .../java/org/apache/beam/sdk/values/PBegin.java |   6 +-
 .../apache/beam/sdk/values/PCollectionList.java |  27 ++--
 .../beam/sdk/values/PCollectionTuple.java   |  10 +-
 .../java/org/apache/beam/sdk/values/PDone.java  |   6 +-
 .../java/org/apache/beam/sdk/values/PInput.java |   4 +-
 .../org/apache/beam/sdk/values/POutput.java |   4 +-
 .../java/org/apache/beam/sdk/values/PValue.java |   4 +-
 .../org/apache/beam/sdk/values/PValueBase.java  |   6 +-
 .../apache/beam/sdk/values/TaggedPValue.java|   5 +
 .../java/org/apache/beam/sdk/PipelineTest.java  |  20 ++-
 .../sdk/runners/TransformHierarchyTest.java |  72 +-
 .../beam/sdk/values/PCollectionListTest.java|  70 --
 .../beam/sdk/values/PCollectionTupleTest.java   |   5 +-
 .../beam/sdk/io/gcp/bigquery/WriteResult.java   |  11 +-
 71 files changed, 526 insertions(+), 658 deletions(-)

[2/4] beam git commit: Remove Orderdness of Input, Output expansions

[tgroh]

http://git-wip-us.apache.org/repos/asf/beam/blob/0e5737fd/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
index f62b320..3638fc8 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/runners/TransformHierarchyTest.java
@@ -17,21 +17,18 @@
  */
 package org.apache.beam.sdk.runners;
 
-import static org.hamcrest.Matchers.contains;
 import static org.hamcrest.Matchers.containsInAnyOrder;
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
 import static org.junit.Assert.assertThat;
 
-import com.google.common.base.Function;
 import com.google.common.collect.Iterables;
-import com.google.common.collect.Lists;
 import java.io.Serializable;
 import java.util.Collections;
 import java.util.HashSet;
-import java.util.List;
 import java.util.Map;
+import java.util.Map.Entry;
 import java.util.Set;
 import org.apache.beam.sdk.Pipeline.PipelineVisitor;
 import org.apache.beam.sdk.Pipeline.PipelineVisitor.Defaults;
@@ -224,11 +221,13 @@ public class TransformHierarchyTest implements 
Serializable {
 assertThat(hierarchy.getCurrent(), equalTo(replacement));
 hierarchy.setOutput(replacementOutput);
 
-TaggedPValue taggedOriginal = 
Iterables.getOnlyElement(originalOutput.expand());
-TaggedPValue taggedReplacement = 
Iterables.getOnlyElement(replacementOutput.expand());
+TaggedPValue taggedReplacement = 
TaggedPValue.ofExpandedValue(replacementOutput);
 Map replacementOutputs =
 Collections.singletonMap(
-replacementOutput, ReplacementOutput.of(taggedOriginal, 
taggedReplacement));
+replacementOutput,
+ReplacementOutput.of(
+TaggedPValue.ofExpandedValue(originalOutput),
+taggedReplacement));
 hierarchy.replaceOutputs(replacementOutputs);
 
 assertThat(replacement.getInputs(), equalTo(original.getInputs()));
@@ -238,8 +237,9 @@ public class TransformHierarchyTest implements Serializable 
{
 replacement.getTransform(), Matchers.equalTo(replacementTransform));
 // THe tags of the replacement transform are matched to the appropriate 
PValues of the original
 assertThat(
-replacement.getOutputs(),
-contains(TaggedPValue.of(taggedReplacement.getTag(), 
taggedOriginal.getValue(;
+replacement.getOutputs().keySet(),
+Matchers.contains(taggedReplacement.getTag()));
+assertThat(replacement.getOutputs().values(), 
Matchers.contains(originalOutput));
 hierarchy.popNode();
   }
 
@@ -294,21 +294,23 @@ public class TransformHierarchyTest implements 
Serializable {
 hierarchy.popNode();
 hierarchy.setOutput(replacementOutput.get(longs));
 
-TaggedPValue originalLongs = Iterables.getOnlyElement(output.expand());
-TaggedPValue replacementLongs = 
Iterables.getOnlyElement(replacementOutput.expand());
+Entry
+replacementLongs = 
Iterables.getOnlyElement(replacementOutput.expand().entrySet());
 hierarchy.replaceOutputs(
 Collections.singletonMap(
-replacementOutput.get(longs), ReplacementOutput.of(originalLongs, 
replacementLongs)));
+replacementOutput.get(longs),
+ReplacementOutput.of(
+TaggedPValue.ofExpandedValue(output),
+TaggedPValue.of(replacementLongs.getKey(), 
replacementLongs.getValue();
 
 assertThat(
-replacementParNode.getOutputs(),
-contains(TaggedPValue.of(replacementLongs.getTag(), 
originalLongs.getValue(;
+replacementParNode.getOutputs().keySet(),
+Matchers.contains(replacementLongs.getKey()));
+assertThat(replacementParNode.getOutputs().values(), 
Matchers.contains(output));
 assertThat(
-compositeNode.getOutputs(),
-contains(
-TaggedPValue.of(
-
Iterables.getOnlyElement(replacementOutput.get(longs).expand()).getTag(),
-originalLongs.getValue(;
+compositeNode.getOutputs().keySet(),
+equalTo(replacementOutput.get(longs).expand().keySet()));
+assertThat(compositeNode.getOutputs().values(), 
Matchers.contains(output));
 hierarchy.popNode();
   }
 
@@ -340,10 +342,10 @@ public class TransformHierarchyTest implements 
Serializable {
 TransformHierarchy.Node compositeNode = hierarchy.pushNode("Create", 
begin, create);
 hierarchy.finishSpecifyingInput();
 assertThat(hierarchy.getCurrent(), equalTo(compositeNode));

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3258

[Apache Jenkins Server]

See 

[1/2] beam git commit: Ignore more python build artifacts.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master 77712c936 -> a5a5bf946


Ignore more python build artifacts.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a1683578
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a1683578
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a1683578

Branch: refs/heads/master
Commit: a168357885b17bde688a548ed1fa12231a49658d
Parents: 77712c9
Author: Robert Bradshaw 
Authored: Tue Apr 11 11:49:16 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Apr 11 11:49:16 2017 -0700

--
 .gitignore | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a1683578/.gitignore
--
diff --git a/.gitignore b/.gitignore
index bc9f675..69946a9 100644
--- a/.gitignore
+++ b/.gitignore
@@ -19,6 +19,9 @@ build/
 dist/
 distribute-*
 env/
+sdks/python/**/*.c
+sdks/python/**/*.so
+sdks/python/**/*.egg
 
 # Ignore IntelliJ files.
 .idea/

[2/2] beam git commit: Closes #2494

[robertwb]

Closes #2494


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a5a5bf94
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a5a5bf94
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a5a5bf94

Branch: refs/heads/master
Commit: a5a5bf946f69d75897279f66aeeb0dcea3babfb2
Parents: 77712c9 a168357
Author: Robert Bradshaw 
Authored: Tue Apr 11 12:12:52 2017 -0700
Committer: Robert Bradshaw 
Committed: Tue Apr 11 12:12:52 2017 -0700

--
 .gitignore | 3 +++
 1 file changed, 3 insertions(+)
--

[GitHub] beam pull request #2494: Ignore more python build artifacts.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2494

Ignore more python build artifacts.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam git-ignore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2494.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2494


commit a168357885b17bde688a548ed1fa12231a49658d
Author: Robert Bradshaw 
Date:   2017-04-11T18:49:16Z

Ignore more python build artifacts.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1832) Potentially unclosed OutputStream in ApexYarnLauncher

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964799#comment-15964799
 ] 

Davor Bonaci commented on BEAM-1832:


Done. Welcome [~rekhajoshm]!

> Potentially unclosed OutputStream in ApexYarnLauncher
> -
>
> Key: BEAM-1832
> URL: https://issues.apache.org/jira/browse/BEAM-1832
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: First stable release
>
>
> Here is an example from createJar():
> {code}
>   final OutputStream out = 
> Files.newOutputStream(zipfs.getPath(JarFile.MANIFEST_NAME));
>   if (!manifestFile.exists()) {
> new Manifest().write(out);
>   } else {
> FileUtils.copyFile(manifestFile, out);
>   }
>   out.close();
> {code}
> If FileUtils.copyFile throws IOException, out would be left unclosed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1832) Potentially unclosed OutputStream in ApexYarnLauncher

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-1832:
--

Assignee: Rekha Joshi

> Potentially unclosed OutputStream in ApexYarnLauncher
> -
>
> Key: BEAM-1832
> URL: https://issues.apache.org/jira/browse/BEAM-1832
> Project: Beam
>  Issue Type: Bug
>  Components: runner-apex
>Reporter: Ted Yu
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: First stable release
>
>
> Here is an example from createJar():
> {code}
>   final OutputStream out = 
> Files.newOutputStream(zipfs.getPath(JarFile.MANIFEST_NAME));
>   if (!manifestFile.exists()) {
> new Manifest().write(out);
>   } else {
> FileUtils.copyFile(manifestFile, out);
>   }
>   out.close();
> {code}
> If FileUtils.copyFile throws IOException, out would be left unclosed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1935) DirectRunner Cancel should never throw a RejectedExecutionException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964790#comment-15964790
 ] 

ASF GitHub Bot commented on BEAM-1935:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2493

[BEAM-1935] Improve Work Rejection handling

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
The timing between checking for a shutdown state and submitting work is
racy. Re-check to see if a work rejection is acceptable because the
underlying executor is shut down before throwing a
RejectedExecutionException.

Renames to internal implementation classes of TransformExecutorServices.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam cancel_rejected_execution

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2493


commit 528830185a9735bb49a64074480760a661e39408
Author: Thomas Groh 
Date:   2017-04-11T18:33:53Z

Improve Work Rejection handling

The timing between checking for a shutdown state and submitting work is
racy. Re-check to see if a work rejection is acceptable because the
underlying executor is shut down before throwing a
RejectedExecutionException.




> DirectRunner Cancel should never throw a RejectedExecutionException
> ---
>
> Key: BEAM-1935
> URL: https://issues.apache.org/jira/browse/BEAM-1935
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> cancelShouldStopPipeline(org.apache.beam.runners.direct.DirectRunnerTest)  
> Time elapsed: 0.025 sec  <<< ERROR!
> java.util.concurrent.RejectedExecutionException: Task 
> java.util.concurrent.FutureTask@391aff79 rejected from 
> java.util.concurrent.ThreadPoolExecutor@3d257c82[Shutting down, pool size = 
> 1, active threads = 1, queue
> d tasks = 0, completed tasks = 0]
> at 
> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
> at 
> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
> at 
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
> at 
> java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
> at 
> org.apache.beam.runners.direct.TransformExecutorServices$ParallelEvaluationState.schedule(TransformExecutorServices.java:69)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.evaluateBundle(ExecutorServiceParallelExecutor.java:245)
> at 
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.scheduleConsumption(ExecutorServiceParallelExecutor.java:208)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2493: [BEAM-1935] Improve Work Rejection handling

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2493

[BEAM-1935] Improve Work Rejection handling

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
The timing between checking for a shutdown state and submitting work is
racy. Re-check to see if a work rejection is acceptable because the
underlying executor is shut down before throwing a
RejectedExecutionException.

Renames to internal implementation classes of TransformExecutorServices.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam cancel_rejected_execution

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2493.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2493


commit 528830185a9735bb49a64074480760a661e39408
Author: Thomas Groh 
Date:   2017-04-11T18:33:53Z

Improve Work Rejection handling

The timing between checking for a shutdown state and submitting work is
racy. Re-check to see if a work rejection is acceptable because the
underlying executor is shut down before throwing a
RejectedExecutionException.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2492: Fix build breaks caused by overlaps between b615013...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2492


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #2492

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 38cf23ebe -> 77712c936


This closes #2492


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/77712c93
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/77712c93
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/77712c93

Branch: refs/heads/master
Commit: 77712c9363937892208e791d6f6856639cbf0c67
Parents: 38cf23e 732610d
Author: Thomas Groh 
Authored: Tue Apr 11 11:12:13 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 11:12:13 2017 -0700

--
 .../io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java  | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--

[2/2] beam git commit: Fix build breaks caused by overlaps between b615013 and c08b7b1

[tgroh]

Fix build breaks caused by overlaps between b615013 and c08b7b1


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/732610d8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/732610d8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/732610d8

Branch: refs/heads/master
Commit: 732610d8eb5c682934d6a1bec0c5c8d8f25c635d
Parents: 38cf23e
Author: Stephen Sisk 
Authored: Tue Apr 11 11:07:48 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 11:12:13 2017 -0700

--
 .../io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java  | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/732610d8/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
--
diff --git 
a/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
 
b/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
index 97addcf..402c19c 100644
--- 
a/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
+++ 
b/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java
@@ -26,7 +26,7 @@ import com.datastax.driver.mapping.annotations.Table;
 
 import java.io.Serializable;
 
-import org.apache.beam.sdk.io.hadoop.inputformat.hashing.HashingFn;
+import org.apache.beam.sdk.io.common.HashingFn;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Combine;
@@ -78,7 +78,7 @@ public class HIFIOWithEmbeddedCassandraTest implements 
Serializable {
   @Test
   public void testHIFReadForCassandra() throws Exception {
 // Expected hashcode is evaluated during insertion time one time and 
hardcoded here.
-String expectedHashCode = "4651110ba1ef2cd3a7315091ca27877b18fceb0e";
+String expectedHashCode = "1b9780833cce000138b9afa25ba63486";
 Configuration conf = getConfiguration();
 PCollection> cassandraData =
 p.apply(HadoopInputFormatIO.read().withConfiguration(conf)
@@ -109,7 +109,7 @@ public class HIFIOWithEmbeddedCassandraTest implements 
Serializable {
   @Test
   public void testHIFReadForCassandraQuery() throws Exception {
 Long expectedCount = 1L;
-String expectedChecksum = "6a62f24ccce0713004889aec1cf226949482d188";
+String expectedChecksum = "f11caabc7a9fc170e22b41218749166c";
 Configuration conf = getConfiguration();
 conf.set("cassandra.input.cql", "select * from " + CASSANDRA_KEYSPACE + 
"." + CASSANDRA_TABLE
 + " where token(id) > ? and token(id) <= ? and scientist='Faraday1' 
allow filtering");

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3257

[Apache Jenkins Server]

See 


--
[...truncated 1.26 MB...]
2017-04-11T18:19:01.482 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
 (181 KB at 4412.3 KB/sec)
2017-04-11T18:19:01.483 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
2017-04-11T18:19:01.516 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
 (68 KB at 2037.9 KB/sec)
2017-04-11T18:19:01.517 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
2017-04-11T18:19:01.543 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
 (9 KB at 333.1 KB/sec)
2017-04-11T18:19:01.545 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
2017-04-11T18:19:01.572 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
 (18 KB at 663.2 KB/sec)
2017-04-11T18:19:01.573 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
2017-04-11T18:19:01.602 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
 (27 KB at 912.0 KB/sec)
2017-04-11T18:19:01.604 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar
2017-04-11T18:19:01.756 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar
 (1654 KB at 10875.7 KB/sec)
2017-04-11T18:19:01.760 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar
2017-04-11T18:19:01.877 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar
 (1202 KB at 10268.1 KB/sec)
2017-04-11T18:19:01.880 [WARNING] Failure to find 
cascading:cascading-hadoop:jar:2.6.3 in https://repo.maven.apache.org/maven2 
was cached in the local repository, resolution will not be reattempted until 
the update interval of central has elapsed or updates are forced

Try downloading the file manually from the project website.

Then, install it using the command: 
mvn install:install-file -DgroupId=cascading -DartifactId=cascading-hadoop 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there: 
mvn deploy:deploy-file -DgroupId=cascading -DartifactId=cascading-hadoop 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] 
-DrepositoryId=[id]

Path to dependency: 
1) 
org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:0.7.0-SNAPSHOT
2) org.elasticsearch:elasticsearch-hadoop:jar:5.0.0
3) cascading:cascading-hadoop:jar:2.6.3


  cascading:cascading-hadoop:jar:2.6.3

from the specified remote repositories:
  Nexus (http://repository.apache.org/snapshots, releases=false, 
snapshots=true),
  central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)

2017-04-11T18:19:01.882 [WARNING] Failure to find 
cascading:cascading-local:jar:2.6.3 in https://repo.maven.apache.org/maven2 was 
cached in the local repository, resolution will not be reattempted until the 
update interval of central has elapsed or updates are forced

Try downloading the file manually from the project website.

Then, install it using the command: 
mvn install:install-file -DgroupId=cascading -DartifactId=cascading-local 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there: 
mvn deploy:deploy-file -DgroupId=cascading -DartifactId=cascading-local 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] 
-DrepositoryId=[id]

Path to dependency: 
1) 
org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:0.7.0-SNAPSHOT
2) org.elasticsearch:elasticsearch-hadoop:jar:5.0.0
3) cascading:cascading-local:jar:2.6.3


  cascading:cascading-local:jar:2.6.3

from the specified remote repositories:
  Nexus (http://repository.apache.org/snapshots, releases=false, 
snapshots=true),
  central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)

2017-04-11T18:19:05.847 [INFO] 
2017-04-11T18:19:05.847 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-04-11T18:19:10.664 [INFO] 
2017-04-11T18:19:10.664 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-04-11T18:19:10.665 [INFO] Using 'UTF-8' 

[jira] [Created] (BEAM-1935) DirectRunner Cancel should never throw a RejectedExecutionException

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1935:
-

 Summary: DirectRunner Cancel should never throw a 
RejectedExecutionException
 Key: BEAM-1935
 URL: https://issues.apache.org/jira/browse/BEAM-1935
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Thomas Groh
Assignee: Thomas Groh


cancelShouldStopPipeline(org.apache.beam.runners.direct.DirectRunnerTest)  Time 
elapsed: 0.025 sec  <<< ERROR!
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.FutureTask@391aff79 rejected from 
java.util.concurrent.ThreadPoolExecutor@3d257c82[Shutting down, pool size = 1, 
active threads = 1, queue
d tasks = 0, completed tasks = 0]
at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at 
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at 
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at 
org.apache.beam.runners.direct.TransformExecutorServices$ParallelEvaluationState.schedule(TransformExecutorServices.java:69)
at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.evaluateBundle(ExecutorServiceParallelExecutor.java:245)
at 
org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.scheduleConsumption(ExecutorServiceParallelExecutor.java:208)




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2807

[Apache Jenkins Server]

See 

[GitHub] beam pull request #2492: Fix build breaks caused by overlaps between b615013...

[ssisk]

GitHub user ssisk opened a pull request:

https://github.com/apache/beam/pull/2492

Fix build breaks caused by overlaps between b615013 and c08b7b1

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


There were conflicting changes in the cassandra unit tests & hashingfn 
changes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ssisk/beam fix-cassandra-hash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2492.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2492


commit ccae1267442a9d8b176a531aa886437d5ad5c9ca
Author: Stephen Sisk 
Date:   2017-04-11T18:07:48Z

Fix build breaks caused by overlaps between b615013 and c08b7b1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3254

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3256

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove Jdk1.8-tests/.toDelete

--
[...truncated 1.23 MB...]
2017-04-11T17:59:35.877 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
 (181 KB at 3413.3 KB/sec)
2017-04-11T17:59:35.879 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
2017-04-11T17:59:35.911 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
 (68 KB at 2101.6 KB/sec)
2017-04-11T17:59:35.912 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
2017-04-11T17:59:35.939 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
 (9 KB at 320.8 KB/sec)
2017-04-11T17:59:35.941 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
2017-04-11T17:59:35.968 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
 (18 KB at 663.2 KB/sec)
2017-04-11T17:59:35.969 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
2017-04-11T17:59:35.998 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
 (27 KB at 912.0 KB/sec)
2017-04-11T17:59:36.001 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar
2017-04-11T17:59:36.148 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar
 (1654 KB at 11245.6 KB/sec)
2017-04-11T17:59:36.153 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar
2017-04-11T17:59:36.264 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty/3.8.0.Final/netty-3.8.0.Final.jar
 (1202 KB at 10823.1 KB/sec)
2017-04-11T17:59:36.267 [WARNING] Failure to find 
cascading:cascading-hadoop:jar:2.6.3 in https://repo.maven.apache.org/maven2 
was cached in the local repository, resolution will not be reattempted until 
the update interval of central has elapsed or updates are forced

Try downloading the file manually from the project website.

Then, install it using the command: 
mvn install:install-file -DgroupId=cascading -DartifactId=cascading-hadoop 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there: 
mvn deploy:deploy-file -DgroupId=cascading -DartifactId=cascading-hadoop 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] 
-DrepositoryId=[id]

Path to dependency: 
1) 
org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:0.7.0-SNAPSHOT
2) org.elasticsearch:elasticsearch-hadoop:jar:5.0.0
3) cascading:cascading-hadoop:jar:2.6.3


  cascading:cascading-hadoop:jar:2.6.3

from the specified remote repositories:
  Nexus (http://repository.apache.org/snapshots, releases=false, 
snapshots=true),
  central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)

2017-04-11T17:59:36.268 [WARNING] Failure to find 
cascading:cascading-local:jar:2.6.3 in https://repo.maven.apache.org/maven2 was 
cached in the local repository, resolution will not be reattempted until the 
update interval of central has elapsed or updates are forced

Try downloading the file manually from the project website.

Then, install it using the command: 
mvn install:install-file -DgroupId=cascading -DartifactId=cascading-local 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file

Alternatively, if you host your own repository you can deploy the file there: 
mvn deploy:deploy-file -DgroupId=cascading -DartifactId=cascading-local 
-Dversion=2.6.3 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] 
-DrepositoryId=[id]

Path to dependency: 
1) 
org.apache.beam:beam-sdks-java-io-hadoop-jdk1.8-tests:jar:0.7.0-SNAPSHOT
2) org.elasticsearch:elasticsearch-hadoop:jar:5.0.0
3) cascading:cascading-local:jar:2.6.3


  cascading:cascading-local:jar:2.6.3

from the specified remote repositories:
  Nexus (http://repository.apache.org/snapshots, releases=false, 
snapshots=true),
  central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)

2017-04-11T17:59:40.598 [INFO] 
2017-04-11T17:59:40.598 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-io-hadoop-jdk1.8-tests ---
2017-04-11T17:59:44.823 [INFO] 
2017-04-11T17:59:44.823 [INFO] --- maven-resources-plugin:2.7:resources 
(default-resources) @ 

Build failed in Jenkins: beam_PerformanceTests_Dataflow #296

[Apache Jenkins Server]

See 


Changes:

[dhalperi] PubsubIO: remove support for BoundedReader

[dhalperi] Move HashingFn to io/common, switch to better hash

[iemejia] Flip dependency edge between Dataflow runner and IO-GCP

[tgroh] Add Coder utilities for Proto conversions

[dhalperi] Improve HadoopInputFormatIO DisplayData and Cassandra tests

[tgroh] Remove Jdk1.8-tests/.toDelete

--
[...truncated 249.74 KB...]
 ! 7e0dd10...2cc86f5 refs/pull/2348/merge -> origin/pr/2348/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! bace20f...08ad8b2 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2363/merge: No such 
file or directory
 ! 49f1638...e32d8df refs/pull/2363/merge -> origin/pr/2363/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2378/merge: No such 
file or directory
 ! c6d11ae...01ec401 refs/pull/2378/merge -> origin/pr/2378/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2401/merge: No such 
file or directory
 ! 1cc81ea...a1da37d refs/pull/2401/merge -> origin/pr/2401/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2403/merge: No such 
file or directory
 ! cd78b7b...18c9ac1 refs/pull/2403/merge -> origin/pr/2403/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2406/merge: No such 
file or directory
 ! bbb3c84...ca34b8a refs/pull/2406/merge -> origin/pr/2406/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2415/merge: No such 
file or directory
 ! e09eec6...180830b refs/pull/2415/merge -> origin/pr/2415/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2417/merge: No such 
file or directory
 ! cef4000...09d8aa8 refs/pull/2417/merge -> origin/pr/2417/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2421/merge: No such 
file or directory
 ! d6345c3...8fd4d87 refs/pull/2421/merge -> origin/pr/2421/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2423/head: No such 
file or directory
 ! dcd42cd..5bab984  refs/pull/2423/head -> origin/pr/2423/head  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2423/merge: No such 
file or directory
 ! 9ea0cc0...fa6a060 refs/pull/2423/merge -> origin/pr/2423/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2426/merge: No such 
file or directory
 ! ab88227...2b1cbb1 refs/pull/2426/merge -> origin/pr/2426/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2430/merge: No such 
file or directory
 ! 7916f75...5f85d8a refs/pull/2430/merge -> origin/pr/2430/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2431/merge: No such 
file or directory
 ! 53495b0...3419835 refs/pull/2431/merge -> origin/pr/2431/merge  (unable to 
update local ref)
 * [new ref] refs/pull/2436/head -> origin/pr/2436/head
 * [new ref] refs/pull/2436/merge -> origin/pr/2436/merge
 * [new ref] refs/pull/2437/head -> origin/pr/2437/head
 * [new ref] refs/pull/2437/merge -> origin/pr/2437/merge
 * [new ref] refs/pull/2438/head -> origin/pr/2438/head
 * [new ref] refs/pull/2439/head -> origin/pr/2439/head
 * [new ref] refs/pull/2439/merge -> origin/pr/2439/merge
 * [new ref] refs/pull/2440/head -> origin/pr/2440/head
 * [new ref] refs/pull/2440/merge -> origin/pr/2440/merge
 * [new ref] refs/pull/2441/head -> origin/pr/2441/head
 * [new ref] refs/pull/2441/merge -> origin/pr/2441/merge
 * [new ref] refs/pull/2442/head -> origin/pr/2442/head
 * [new ref] refs/pull/2442/merge -> origin/pr/2442/merge
 * [new ref] refs/pull/2443/head -> origin/pr/2443/head
 * [new ref] refs/pull/2443/merge -> origin/pr/2443/merge
 * [new ref] refs/pull/2444/head -> origin/pr/2444/head
 * [new ref] refs/pull/2444/merge -> origin/pr/2444/merge
 * [new ref] refs/pull/2445/head -> origin/pr/2445/head
 * [new ref] refs/pull/2445/merge -> origin/pr/2445/merge
 * [new ref] refs/pull/2446/head -> origin/pr/2446/head
 * [new ref] refs/pull/2446/merge -> origin/pr/2446/merge
 * [new ref] refs/pull/2447/head -> origin/pr/2447/head
 * [new ref] refs/pull/2447/merge -> origin/pr/2447/merge
 * [new ref] refs/pull/2448/head -> origin/pr/2448/head
 * [new ref] refs/pull/2449/head -> origin/pr/2449/head
 * [new ref] 

[jira] [Created] (BEAM-1933) Revisit Runner API PCollection unique_name

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1933:
-

 Summary: Revisit Runner API PCollection unique_name
 Key: BEAM-1933
 URL: https://issues.apache.org/jira/browse/BEAM-1933
 Project: Beam
  Issue Type: Bug
  Components: beam-model-runner-api, sdk-java-core
Reporter: Thomas Groh
Assignee: Kenneth Knowles
Priority: Minor


The Java SDK performs no work to ensure that a PCollection has a unique name. 
Most names will end up being unique, because they are autogenerated based on 
the full name of the producing transform, but names can be set by users 
arbitrarily.

We can generate a unique name, potentially one based on the PCollection name, 
but this may not map to something that is "stable over multiple evolutions of a 
pipeline"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-260) WindowMappingFn: Know the getSideInputWindow upper bound to release side input resources

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-260.
--
   Resolution: Fixed
Fix Version/s: First stable release

> WindowMappingFn: Know the getSideInputWindow upper bound to release side 
> input resources
> 
>
> Key: BEAM-260
> URL: https://issues.apache.org/jira/browse/BEAM-260
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api, beam-model-runner-api
>Reporter: Mark Shields
>Assignee: Thomas Groh
> Fix For: First stable release
>
>
> We currently have no static knowledge about the getSideInputWindow function, 
> and runners are thus forced to hold on to all side input state / elements in 
> case a future element reaches back into an earlier side input element.
> Maybe we need an upper bound on lag from current to result of 
> getSideInputWindow so we can have a progressing gc horizon as we do for  GKB 
> window state. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-260) WindowMappingFn: Know the getSideInputWindow upper bound to release side input resources

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964694#comment-15964694
 ] 

Thomas Groh commented on BEAM-260:
--

This is supported in the Java SDK. I've filed 
https://issues.apache.org/jira/browse/BEAM-1931 and 
https://issues.apache.org/jira/browse/BEAM-1932 to track the use of this in the 
Java SDK, and I'm going to close this as preempted by those two issues.

> WindowMappingFn: Know the getSideInputWindow upper bound to release side 
> input resources
> 
>
> Key: BEAM-260
> URL: https://issues.apache.org/jira/browse/BEAM-260
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model-fn-api, beam-model-runner-api
>Reporter: Mark Shields
>Assignee: Thomas Groh
>
> We currently have no static knowledge about the getSideInputWindow function, 
> and runners are thus forced to hold on to all side input state / elements in 
> case a future element reaches back into an earlier side input element.
> Maybe we need an upper bound on lag from current to result of 
> getSideInputWindow so we can have a progressing gc horizon as we do for  GKB 
> window state. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2491: Remove Jdk1.8-tests/.toDelete

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2491


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2491: Remove Jdk1.8-tests/.toDelete

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2491

Remove Jdk1.8-tests/.toDelete

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Breaks RAT. File not useful.

TBR: @ssisk @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam fix_rat_todelete

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2491






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3255

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Improve HadoopInputFormatIO DisplayData and Cassandra tests

--
[...truncated 508.32 KB...]
2017-04-11T17:30:22.243 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.6/doxia-site-renderer-1.6.pom
2017-04-11T17:30:22.270 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.6/doxia-site-renderer-1.6.pom
 (6 KB at 221.0 KB/sec)
2017-04-11T17:30:22.274 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.6/doxia-core-1.6.pom
2017-04-11T17:30:22.302 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-core/1.6/doxia-core-1.6.pom
 (5 KB at 146.6 KB/sec)
2017-04-11T17:30:22.305 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.6/doxia-module-xhtml-1.6.pom
2017-04-11T17:30:22.330 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.6/doxia-module-xhtml-1.6.pom
 (2 KB at 63.7 KB/sec)
2017-04-11T17:30:22.332 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-modules/1.6/doxia-modules-1.6.pom
2017-04-11T17:30:22.357 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-modules/1.6/doxia-modules-1.6.pom
 (3 KB at 102.4 KB/sec)
2017-04-11T17:30:22.360 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.6/doxia-module-fml-1.6.pom
2017-04-11T17:30:22.387 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.6/doxia-module-fml-1.6.pom
 (5 KB at 173.8 KB/sec)
2017-04-11T17:30:22.390 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.21/plexus-utils-3.0.21.pom
2017-04-11T17:30:22.417 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.21/plexus-utils-3.0.21.pom
 (4 KB at 137.8 KB/sec)
2017-04-11T17:30:22.424 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.2/doxia-logging-api-1.2.jar
2017-04-11T17:30:22.425 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/rat/apache-rat-core/0.12/apache-rat-core-0.12.jar
2017-04-11T17:30:22.425 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/rat/apache-rat-api/0.12/apache-rat-api-0.12.jar
2017-04-11T17:30:22.426 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.jar
2017-04-11T17:30:22.426 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-io/commons-io/2.2/commons-io-2.2.jar
2017-04-11T17:30:22.452 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-logging-api/1.2/doxia-logging-api-1.2.jar
 (12 KB at 395.8 KB/sec)
2017-04-11T17:30:22.452 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.10/commons-compress-1.10.jar
2017-04-11T17:30:22.463 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/1.2-alpha-9/plexus-classworlds-1.2-alpha-9.jar
 (42 KB at 1064.9 KB/sec)
2017-04-11T17:30:22.463 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.6/doxia-decoration-model-1.6.jar
2017-04-11T17:30:22.469 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/rat/apache-rat-core/0.12/apache-rat-core-0.12.jar
 (120 KB at 2715.8 KB/sec)
2017-04-11T17:30:22.469 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-sink-api/1.6/doxia-sink-api-1.6.jar
2017-04-11T17:30:22.470 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/rat/apache-rat-api/0.12/apache-rat-api-0.12.jar
 (11 KB at 230.2 KB/sec)
2017-04-11T17:30:22.470 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-site-renderer/1.6/doxia-site-renderer-1.6.jar
2017-04-11T17:30:22.471 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-io/commons-io/2.2/commons-io-2.2.jar
 (170 KB at 3685.2 KB/sec)
2017-04-11T17:30:22.471 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-xhtml/1.6/doxia-module-xhtml-1.6.jar
2017-04-11T17:30:22.498 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-decoration-model/1.6/doxia-decoration-model-1.6.jar
 (56 KB at 771.0 KB/sec)
2017-04-11T17:30:22.499 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/doxia/doxia-module-fml/1.6/doxia-module-fml-1.6.jar
2017-04-11T17:30:22.502 [INFO] Downloaded: 

[GitHub] beam-site pull request #207: remove extra streaming refs in capability matri...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/beam-site/pull/207

remove extra streaming refs in capability matrix: Flink state/timers

these streaming references seem to be out of place after PR 198
R: @aljoscha 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/beam-site capmatrix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #207






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1931) The DirectRunner should garbage collect PCollectionView contents

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1931:
-

 Summary: The DirectRunner should garbage collect PCollectionView 
contents
 Key: BEAM-1931
 URL: https://issues.apache.org/jira/browse/BEAM-1931
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


It should use the Maximum Lookback of the associated {{WindowMappingFn}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-819) The DirectRunner can complete before reporting the results of all ModelEnforcements

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-819:


Assignee: (was: Thomas Groh)

> The DirectRunner can complete before reporting the results of all 
> ModelEnforcements
> ---
>
> Key: BEAM-819
> URL: https://issues.apache.org/jira/browse/BEAM-819
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Thomas Groh
>
> The termination condition in the 
> ExecutorServiceParallelExecutor.MonitorRunnable checks to see if the 
> evaluation context is done, among other things. Due to the signatures of 
> ModelEnforcement, output is committed before ModelEnforcement runs to ensure 
> that the enforcement has access to the final elements. If the transform that 
> performs ModelEnforcement completes, and all PTransforms are done, the 
> pipeline can shutdown before model enforcement fails.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-436) DirectRunner RunnableOnService tempLocation configuration insufficient

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh reassigned BEAM-436:


Assignee: (was: Thomas Groh)

> DirectRunner RunnableOnService tempLocation configuration insufficient
> --
>
> Key: BEAM-436
> URL: https://issues.apache.org/jira/browse/BEAM-436
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Scott Wegner
>
> The way we specify temp storage locations for RunnableOnService tests is not 
> sufficient, specifically for DirectRunner execution. Right now, 
> RunnableOnService tests are run for DirectRunner and DataflowRunner, who set 
> their temp locations differently:
> * DirectRunner doesn't specify a temp location directly, but test classes 
> will use a JUnit @Rule TemporaryDirectory. Individual tests set it as 
> necessary for tempLocation, and set a fake gs:// path for individual GCP IO 
> tests.
> * DataflowRunner tests pass an actual GCS path as tempRoot, and 
> TestDataflowRunner will initialize stagingLocation to this path.
> This setup makes it difficult to write RunnableOnService tests which pass for 
> both runners. We should separate temp location setup out of individual test 
> classes so that RunnableOnService tests "just work" on any runner.
> One solution would be to add logic inside TestPipeline#testingPipelineOptions:
> # If --tempRoot is specified, use it to set tempLocation and stagingLocation. 
> Otherwise, use a JUnit TemporaryDirectory to set it
> # If tempLocation is a GCS path, use it to set stagingLocation. Otherwise, 
> use a fake gcs path (i.e. gs://foo)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2460: Hifio Cassandra and Elasticsearch Part2 fixes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2460


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964671#comment-15964671
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2490

[BEAM-1928] Use SdkComponents in WindowingStrategy.toProto

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam windowing_strategies_components

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2490


commit ca61a6dfe1ac94be823d6e7cb2beddca54c6eaf5
Author: Thomas Groh 
Date:   2017-04-07T20:41:29Z

Use SdkComponents in WindowingStrategy.toProto




> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2490: [BEAM-1928] Use SdkComponents in WindowingStrategy....

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/2490

[BEAM-1928] Use SdkComponents in WindowingStrategy.toProto

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam windowing_strategies_components

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2490.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2490


commit ca61a6dfe1ac94be823d6e7cb2beddca54c6eaf5
Author: Thomas Groh 
Date:   2017-04-07T20:41:29Z

Use SdkComponents in WindowingStrategy.toProto




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2460

[dhalperi]

This closes #2460


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d255fa25
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d255fa25
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d255fa25

Branch: refs/heads/master
Commit: d255fa25f1a1a0bf61ea159e2922f014d36eda43
Parents: 8beea73 c08b7b1
Author: Dan Halperin 
Authored: Tue Apr 11 10:20:35 2017 -0700
Committer: Dan Halperin 
Committed: Tue Apr 11 10:20:35 2017 -0700

--
 .../hadoop/inputformat/HadoopInputFormatIO.java |  33 +--
 .../inputformat/HadoopInputFormatIOTest.java|  99 ++---
 sdks/java/io/hadoop/jdk1.8-tests/.toDelete  |   0
 .../HIFIOWithEmbeddedCassandraTest.java | 214 +++
 4 files changed, 306 insertions(+), 40 deletions(-)
--

[1/2] beam git commit: Improve HadoopInputFormatIO DisplayData and Cassandra tests

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 8beea73c1 -> d255fa25f


Improve HadoopInputFormatIO DisplayData and Cassandra tests


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c08b7b17
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c08b7b17
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c08b7b17

Branch: refs/heads/master
Commit: c08b7b1771481b77c94ec78a96db5b34fec29841
Parents: 8beea73
Author: Dipti Kulkarni 
Authored: Mon Apr 10 15:43:12 2017 +0530
Committer: Dan Halperin 
Committed: Tue Apr 11 10:20:15 2017 -0700

--
 .../hadoop/inputformat/HadoopInputFormatIO.java |  33 +--
 .../inputformat/HadoopInputFormatIOTest.java|  99 ++---
 sdks/java/io/hadoop/jdk1.8-tests/.toDelete  |   0
 .../HIFIOWithEmbeddedCassandraTest.java | 214 +++
 4 files changed, 306 insertions(+), 40 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c08b7b17/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java
--
diff --git 
a/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java
 
b/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java
index 675f4bf..61dc1bf 100644
--- 
a/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java
+++ 
b/sdks/java/io/hadoop/input-format/src/main/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java
@@ -36,9 +36,7 @@ import java.math.BigInteger;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.HashSet;
-import java.util.Iterator;
 import java.util.List;
-import java.util.Map.Entry;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
 
@@ -360,20 +358,6 @@ public class HadoopInputFormatIO {
 + e.getMessage(), e);
   }
 }
-
-@Override
-public void populateDisplayData(DisplayData.Builder builder) {
-  super.populateDisplayData(builder);
-  if (getConfiguration().getHadoopConfiguration() != null) {
-Iterator> configProperties = getConfiguration()
-.getHadoopConfiguration().iterator();
-while (configProperties.hasNext()) {
-  Entry property = configProperties.next();
-  builder.addIfNotNull(DisplayData.item(property.getKey(), 
property.getValue())
-  .withLabel(property.getKey()));
-}
-  }
-}
   }
 
   /**
@@ -447,6 +431,23 @@ public class HadoopInputFormatIO {
 }
 
 @Override
+public void populateDisplayData(DisplayData.Builder builder) {
+  super.populateDisplayData(builder);
+  Configuration hadoopConfig = getConfiguration().getHadoopConfiguration();
+  if (hadoopConfig != null) {
+
builder.addIfNotNull(DisplayData.item("mapreduce.job.inputformat.class",
+hadoopConfig.get("mapreduce.job.inputformat.class"))
+.withLabel("InputFormat Class"));
+builder.addIfNotNull(DisplayData.item("key.class",
+hadoopConfig.get("key.class"))
+.withLabel("Key Class"));
+builder.addIfNotNull(DisplayData.item("value.class",
+hadoopConfig.get("value.class"))
+.withLabel("Value Class"));
+  }
+}
+
+@Override
 public List>> splitIntoBundles(long 
desiredBundleSizeBytes,
 PipelineOptions options) throws Exception {
   // desiredBundleSizeBytes is not being considered as splitting based on 
this

http://git-wip-us.apache.org/repos/asf/beam/blob/c08b7b17/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
--
diff --git 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
index 2f2857b..3a4a99d 100644
--- 
a/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
+++ 
b/sdks/java/io/hadoop/input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
@@ -21,9 +21,7 @@ import static org.junit.Assert.assertThat;
 
 import java.io.IOException;
 import java.util.ArrayList;
-import java.util.Iterator;
 import java.util.List;
-import java.util.Map.Entry;
 
 import org.apache.beam.sdk.coders.AvroCoder;
 import 

[jira] [Commented] (BEAM-1930) Abstract Coder translation from Runner API Protos

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964647#comment-15964647
 ] 

Luke Cwik commented on BEAM-1930:
-

I think with Jackson/ObjectMapper you could do the same thing and have it 
convert from proto -> json -> Java coder and then we might be able to drop 
CloudObject usage within Coder.

> Abstract Coder translation from Runner API Protos
> -
>
> Key: BEAM-1930
> URL: https://issues.apache.org/jira/browse/BEAM-1930
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Priority: Minor
>
> Instead of having a large switch statement that constructs known coders, have 
> a map from URN to an interface that takes {{(RunnerApi.Coder, 
> RunnerApi.Components)}} and produces a java Coder, and delegate all calls in 
> {{fromProto}} to that map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1930) Abstract Coder translation from Runner API Protos

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1930:
-

 Summary: Abstract Coder translation from Runner API Protos
 Key: BEAM-1930
 URL: https://issues.apache.org/jira/browse/BEAM-1930
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Priority: Minor


Instead of having a large switch statement that constructs known coders, have a 
map from URN to an interface that takes {{(RunnerApi.Coder, 
RunnerApi.Components)}} and produces a java Coder, and delegate all calls in 
{{fromProto}} to that map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1928) Populate Runner API Components from the Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964638#comment-15964638
 ] 

ASF GitHub Bot commented on BEAM-1928:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2487


> Populate Runner API Components from the Java SDK
> 
>
> Key: BEAM-1928
> URL: https://issues.apache.org/jira/browse/BEAM-1928
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api, runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Permit conversion of Java SDK components to runner API protocol buffers, and 
> the extraction of those SDK components from the protocol buffers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2487: [BEAM-1928] Add Coder utilities for Proto conversio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2487

[tgroh]

This closes #2487


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8beea73c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8beea73c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8beea73c

Branch: refs/heads/master
Commit: 8beea73c1b57414dfb2b0a3f44f5bee29f088b4e
Parents: 986fcef 01e5a8d
Author: Thomas Groh 
Authored: Tue Apr 11 09:58:44 2017 -0700
Committer: Thomas Groh 
Committed: Tue Apr 11 09:58:44 2017 -0700

--
 .../beam/runners/core/construction/Coders.java  | 162 +++
 .../core/construction/SdkComponents.java|   5 +-
 .../runners/core/construction/CodersTest.java   |  99 
 .../core/construction/SdkComponentsTest.java|  25 ++-
 4 files changed, 288 insertions(+), 3 deletions(-)
--