[jira] [Commented] (BEAM-740) Improve document on running Beam program on Flink Cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564264#comment-15564264
 ] 

ASF GitHub Bot commented on BEAM-740:
-

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1082

[BEAM-740] improve runners-flink README.md

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam flink_doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1082.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1082


commit aedce69bca5a81cd584069fe73984bb4ada41af3
Author: manuzhang 
Date:   2016-10-10T11:38:35Z

[BEAM-740] improve runners-flink README.md




> Improve document on running Beam program on Flink Cluster
> -
>
> Key: BEAM-740
> URL: https://issues.apache.org/jira/browse/BEAM-740
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Affects Versions: 0.2.0-incubating
>Reporter: Manu Zhang
>Priority: Trivial
>
> The uber jar to run on Flink cluster contains a {{LICENSE}} file and a 
> {{license}} directory. The two will conflict on the case-insensitive file 
> system of OS X when the jar file is being extracted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1082: [BEAM-740] improve runners-flink README.m...

[manuzhang]

GitHub user manuzhang opened a pull request:

https://github.com/apache/incubator-beam/pull/1082

[BEAM-740] improve runners-flink README.md

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/manuzhang/incubator-beam flink_doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1082.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1082


commit aedce69bca5a81cd584069fe73984bb4ada41af3
Author: manuzhang 
Date:   2016-10-10T11:38:35Z

[BEAM-740] improve runners-flink README.md




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-740) Improve document on running Beam program on Flink Cluster

[Manu Zhang (JIRA)]

Manu Zhang created BEAM-740:
---

 Summary: Improve document on running Beam program on Flink Cluster
 Key: BEAM-740
 URL: https://issues.apache.org/jira/browse/BEAM-740
 Project: Beam
  Issue Type: Improvement
  Components: runner-flink
Affects Versions: 0.2.0-incubating
Reporter: Manu Zhang
Priority: Trivial


The uber jar to run on Flink cluster contains a {{LICENSE}} file and a 
{{license}} directory. The two will conflict on the case-insensitive file 
system of OS X when the jar file is being extracted. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1305

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1078: [BEAM-736] Fix BigQueryTornadoesIT, broke...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1078


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-736) BigQueryTornadoesIT broken, blocking nightly release.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564019#comment-15564019
 ] 

ASF GitHub Bot commented on BEAM-736:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1078


> BigQueryTornadoesIT broken, blocking nightly release.
> -
>
> Key: BEAM-736
> URL: https://issues.apache.org/jira/browse/BEAM-736
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Pei He
>
> Build break begins here: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1471/
> listing 3 potential culprit commits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c7c13157f -> daf69f87e


[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1d3f31da
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1d3f31da
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1d3f31da

Branch: refs/heads/master
Commit: 1d3f31da7ee9769200e8a93ec3626cd4cd636f67
Parents: c7c1315
Author: Pei He 
Authored: Mon Oct 10 14:30:30 2016 -0700
Committer: Luke Cwik 
Committed: Mon Oct 10 17:28:19 2016 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1d3f31da/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 3d1aba6..5aa952c 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -542,11 +542,11 @@ public class BigQueryIO {
 // For these cases the withoutValidation method can be used to disable 
the check.
 if (validate && table != null) {
   // Check for source table presence for early failure notification.
-  DatasetService datasetService = 
bigQueryServices.getDatasetService(bqOptions);
+  DatasetService datasetService = 
getBigQueryServices().getDatasetService(bqOptions);
   verifyDatasetPresence(datasetService, table);
   verifyTablePresence(datasetService, table);
 } else if (validate && query != null) {
-  JobService jobService = bigQueryServices.getJobService(bqOptions);
+  JobService jobService = 
getBigQueryServices().getJobService(bqOptions);
   try {
 jobService.dryRunQuery(
 bqOptions.getProject(),

[2/2] incubator-beam git commit: [BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039

[lcwik]

[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039

This closes #1078


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/daf69f87
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/daf69f87
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/daf69f87

Branch: refs/heads/master
Commit: daf69f87e7271b2620e9bc023a8a360a485f856a
Parents: c7c1315 1d3f31d
Author: Luke Cwik 
Authored: Mon Oct 10 17:28:52 2016 -0700
Committer: Luke Cwik 
Committed: Mon Oct 10 17:28:52 2016 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: Add license to init files.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk fd27f6848 -> ba624e98f


Add license to init files.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ccc01c98
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ccc01c98
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ccc01c98

Branch: refs/heads/python-sdk
Commit: ccc01c9869b0212a38b4b35a577c42533b5ca3e4
Parents: fd27f68
Author: Ahmet Altay 
Authored: Mon Oct 10 15:50:13 2016 -0700
Committer: Ahmet Altay 
Committed: Mon Oct 10 15:50:13 2016 -0700

--
 .../python/apache_beam/runners/dataflow/__init__.py | 16 
 .../runners/dataflow/native_io/__init__.py  | 16 
 2 files changed, 32 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ccc01c98/sdks/python/apache_beam/runners/dataflow/__init__.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/__init__.py 
b/sdks/python/apache_beam/runners/dataflow/__init__.py
index e69de29..cce3aca 100644
--- a/sdks/python/apache_beam/runners/dataflow/__init__.py
+++ b/sdks/python/apache_beam/runners/dataflow/__init__.py
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ccc01c98/sdks/python/apache_beam/runners/dataflow/native_io/__init__.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/__init__.py 
b/sdks/python/apache_beam/runners/dataflow/native_io/__init__.py
index e69de29..cce3aca 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/__init__.py
+++ b/sdks/python/apache_beam/runners/dataflow/native_io/__init__.py
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

[GitHub] incubator-beam pull request #1081: Add license to init files.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1081


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1081

[robertwb]

Closes #1081


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ba624e98
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ba624e98
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ba624e98

Branch: refs/heads/python-sdk
Commit: ba624e98fe1276a0d5794e7d08aa15cff13c4307
Parents: fd27f68 ccc01c9
Author: Robert Bradshaw 
Authored: Mon Oct 10 17:12:38 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 17:12:38 2016 -0700

--
 .../python/apache_beam/runners/dataflow/__init__.py | 16 
 .../runners/dataflow/native_io/__init__.py  | 16 
 2 files changed, 32 insertions(+)
--

[GitHub] incubator-beam pull request #1080: [BEAM-498] Inline DoFnRunnerBase and fix ...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1080

[BEAM-498] Inline DoFnRunnerBase and fix up

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This class had just one subclass and cleaning up these little bits makes it 
clearer as we move to the new `DoFn`. Nothing radical, but I need to make 
`DoFnRunners` work with old and new `DoFn` so each dangling class just adds 
complexity to the field of operation.

R: @peihe

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam OldDoFnRunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1080


commit 2848f8b8f4203fb6dde319779852d57aa98dac4d
Author: Kenneth Knowles 
Date:   2016-10-10T21:59:54Z

Remove unused ListOutputManager

commit 249d2d970039f5b096da7f9306acdfc41a817ee1
Author: Kenneth Knowles 
Date:   2016-10-10T22:01:41Z

Inline DoFnRunnerBase to its sole subclass

commit b5b28517e8f9cf1394bfc09ec7dfecf8c3d3c048
Author: Kenneth Knowles 
Date:   2016-10-10T22:04:42Z

Fix access levels on SimpleDoFnRunner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563686#comment-15563686
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1080

[BEAM-498] Inline DoFnRunnerBase and fix up

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This class had just one subclass and cleaning up these little bits makes it 
clearer as we move to the new `DoFn`. Nothing radical, but I need to make 
`DoFnRunners` work with old and new `DoFn` so each dangling class just adds 
complexity to the field of operation.

R: @peihe

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam OldDoFnRunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1080.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1080


commit 2848f8b8f4203fb6dde319779852d57aa98dac4d
Author: Kenneth Knowles 
Date:   2016-10-10T21:59:54Z

Remove unused ListOutputManager

commit 249d2d970039f5b096da7f9306acdfc41a817ee1
Author: Kenneth Knowles 
Date:   2016-10-10T22:01:41Z

Inline DoFnRunnerBase to its sole subclass

commit b5b28517e8f9cf1394bfc09ec7dfecf8c3d3c048
Author: Kenneth Knowles 
Date:   2016-10-10T22:04:42Z

Fix access levels on SimpleDoFnRunner




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-739) Log full exception stack trace in WordCountIT and BigQueryTornadoesIT

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563621#comment-15563621
 ] 

ASF GitHub Bot commented on BEAM-739:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1079

[BEAM-739] Log exceptions full stack track in IT tests

R: @lukecwik @jasonkuster

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam log-stack-trace

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1079


commit 3f70aeb2ba6a66b1b445253f5aba34e4074be6e8
Author: Pei He 
Date:   2016-10-10T21:39:55Z

[BEAM-739] Log exceptions full stack track in IT tests




> Log full exception stack trace in WordCountIT and BigQueryTornadoesIT
> -
>
> Key: BEAM-739
> URL: https://issues.apache.org/jira/browse/BEAM-739
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> When IT tests are broken, they don't provide the full stack trace, such as in:
> https://issues.apache.org/jira/browse/BEAM-736
> It makes investigating root causes slower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1079: [BEAM-739] Log exceptions full stack trac...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1079

[BEAM-739] Log exceptions full stack track in IT tests

R: @lukecwik @jasonkuster

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam log-stack-trace

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1079.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1079


commit 3f70aeb2ba6a66b1b445253f5aba34e4074be6e8
Author: Pei He 
Date:   2016-10-10T21:39:55Z

[BEAM-739] Log exceptions full stack track in IT tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-736) BigQueryTornadoesIT broken, blocking nightly release.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563608#comment-15563608
 ] 

ASF GitHub Bot commented on BEAM-736:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1078

[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1078


commit 72efd95df6bc89648f8c758092cf6e9229fda186
Author: Pei He 
Date:   2016-10-10T21:30:30Z

[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039




> BigQueryTornadoesIT broken, blocking nightly release.
> -
>
> Key: BEAM-736
> URL: https://issues.apache.org/jira/browse/BEAM-736
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Pei He
>
> Build break begins here: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1471/
> listing 3 potential culprit commits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1078: [BEAM-736] Fix BigQueryTornadoesIT, broke...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1078

[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam fix-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1078.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1078


commit 72efd95df6bc89648f8c758092cf6e9229fda186
Author: Pei He 
Date:   2016-10-10T21:30:30Z

[BEAM-736] Fix BigQueryTornadoesIT, broken by PR-1039




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-739) Log full exception stack trace in WordCountIT and BigQueryTornadoesIT

[Pei He (JIRA)]

Pei He created BEAM-739:
---

 Summary: Log full exception stack trace in WordCountIT and 
BigQueryTornadoesIT
 Key: BEAM-739
 URL: https://issues.apache.org/jira/browse/BEAM-739
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Pei He
Assignee: Pei He
Priority: Minor


When IT tests are broken, they don't provide the full stack trace, such as in:
https://issues.apache.org/jira/browse/BEAM-736

It makes investigating root causes slower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563569#comment-15563569
 ] 

Pei He edited comment on BEAM-696 at 10/10/16 9:26 PM:
---

re: to "Sessions is not allowed in side inputs"
There are two WindowFns involved, one from main input PCollection and the other 
from side input view.
For example, you can have Sessions in the main input window, and looks up 
values in a fix windowed side input.
Sessions.java#L84 is saying side inputs cannot be windowed by Sessions.

This jira [BEAM-696] is about when the main input is windowed by Sessions, how 
Combine with side input should be executed.


was (Author: pei...@gmail.com):
re: to "Sessions is not allowed in side inputs"
There are two WindowFns involved, one from main input PCollection and the other 
from side input view.
Now, you can have Sessions in the main input window, and looks up values in a 
fix windowed side input.
Sessions.java#L84 is saying side inputs cannot be windowed by Sessions.

This jira [BEAM-696] is about when the main input is windowed by Sessions, how 
Combine with side input should be executed.

> Side-Inputs non-deterministic with merging main-input windows
> -
>
> Key: BEAM-696
> URL: https://issues.apache.org/jira/browse/BEAM-696
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Ben Chambers
>Assignee: Pei He
>
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable 
> because triggers are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to 
> lookup the side-input. This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may 
> cause problems with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute 
> within a Merging WindowFn.
> Possible solution would be to defer running anything that looks up the 
> side-input until we need to extract an output, and using the main-window at 
> that point. Specifically, if the main-window is a MergingWindowFn, don't 
> execute any kind of pre-combine, instead buffer all the inputs and combine 
> later.
> This could still run into some non-determinism if there are triggers 
> controlling when we extract output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563569#comment-15563569
 ] 

Pei He commented on BEAM-696:
-

re: to "Sessions is not allowed in side inputs"
There are two WindowFns involved, one from main input PCollection and the other 
from side input view.
Now, you can have Sessions in the main input window, and looks up values in a 
fix windowed side input.
Sessions.java#L84 is saying side inputs cannot be windowed by Sessions.

This jira [BEAM-696] is about when the main input is windowed by Sessions, how 
Combine with side input should be executed.

> Side-Inputs non-deterministic with merging main-input windows
> -
>
> Key: BEAM-696
> URL: https://issues.apache.org/jira/browse/BEAM-696
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Ben Chambers
>Assignee: Pei He
>
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable 
> because triggers are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to 
> lookup the side-input. This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may 
> cause problems with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute 
> within a Merging WindowFn.
> Possible solution would be to defer running anything that looks up the 
> side-input until we need to extract an output, and using the main-window at 
> that point. Specifically, if the main-window is a MergingWindowFn, don't 
> execute any kind of pre-combine, instead buffer all the inputs and combine 
> later.
> This could still run into some non-determinism if there are triggers 
> controlling when we extract output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-738) Update sourcetestutils to fail when BoundedSources maintain state

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-738:
---

 Summary: Update sourcetestutils to fail when BoundedSources 
maintain state
 Key: BEAM-738
 URL: https://issues.apache.org/jira/browse/BEAM-738
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Maintaining state in BoundedSource implementations is problematic and can lead 
to hard to debug errors. For example (1) pickling errors (2) errors due to a 
runner reusing a BoundedSource object with state.

We can try to prevent users from adding state to BoundedSource implementations 
in following two ways.

(1) Clearly mention in BoundedSource API that objects should not maintain 
transient state.
(2) Update sourcetesutils to catch source objects that maintain local state.

(2) can be done by adding a check that verifies that a source produces expected 
output in the presence of a re-entrant read.

i = s.read_records()
i.next()
i.next()
read the whole thing from s.read_records()
i.next() some more

Verify that 'i' produced correct output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563458#comment-15563458
 ] 

ASF GitHub Bot commented on BEAM-115:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1076

[BEAM-115] Remove PCollectionView#fromIterableInternal

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
fromIterableInternal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1076.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1076


commit c9bf9ec9b3c92a81be76ef1a2eb49e52f35c7e21
Author: Kenneth Knowles 
Date:   2016-10-10T20:22:02Z

Remove last use of PCollectionView.fromIterableInternal

commit a852354c84ac93bb9e6ac867cb76f05b1fc5e96f
Author: Kenneth Knowles 
Date:   2016-10-10T20:45:58Z

Remove the deprecated PCollectionView.fromIterableInternal




> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1491

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1491

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-736) BigQueryTornadoesIT broken, blocking nightly release.

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563328#comment-15563328
 ] 

Pei He commented on BEAM-736:
-

NPE comes from here:
https://github.com/apache/incubator-beam/commit/cca861ba82a2e6ba6c6af122be0b8a9932d53cc5#diff-5b2e45465c4a6abd7d59967281b3a8b4R550

bigQueryServices is lazily initialized. It should be accessed through 
getBigQueryServices().
It is easy to fix.
This bug is introduced during the backport, not from Dataflow.

Two things could be improved:
1. better coding pattern to prevent bigQueryServices being accessed before it 
is initialized.
2. better logging in BigQueryTornadoesIT to provide full stack trace.

> BigQueryTornadoesIT broken, blocking nightly release.
> -
>
> Key: BEAM-736
> URL: https://issues.apache.org/jira/browse/BEAM-736
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Pei He
>
> Build break begins here: 
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1471/
> listing 3 potential culprit commits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15563315#comment-15563315
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1064


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1064: [BEAM-25] Add StateSpec as a curried way ...

[kennknowles]

Github user kennknowles closed the pull request at:

https://github.com/apache/incubator-beam/pull/1064


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1035: [BEAM] Making sure that GcsBufferedReader...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1035


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Move GcsPathValidatorTest to package matching class under test

[lcwik]

Move GcsPathValidatorTest to package matching class under test

This closes #1074


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c7c13157
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c7c13157
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c7c13157

Branch: refs/heads/master
Commit: c7c13157fae00a0af6bcc3aad2369c34956507ed
Parents: 570de74 a37ec0a
Author: Luke Cwik 
Authored: Mon Oct 10 11:49:28 2016 -0700
Committer: Luke Cwik 
Committed: Mon Oct 10 11:49:28 2016 -0700

--
 .../dataflow/util/GcsPathValidatorTest.java | 103 ---
 .../beam/sdk/util/GcsPathValidatorTest.java | 100 ++
 2 files changed, 100 insertions(+), 103 deletions(-)
--

[GitHub] incubator-beam pull request #1074: Move GcsPathValidatorTest to package matc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1074


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Move GcsPathValidatorTest to package matching class under test

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 570de74da -> c7c13157f


Move GcsPathValidatorTest to package matching class under test


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a37ec0ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a37ec0ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a37ec0ae

Branch: refs/heads/master
Commit: a37ec0ae9d07915fe33e932578628b02b3161d02
Parents: 570de74
Author: Kenneth Knowles 
Authored: Mon Oct 10 11:28:11 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 10 11:28:11 2016 -0700

--
 .../dataflow/util/GcsPathValidatorTest.java | 103 ---
 .../beam/sdk/util/GcsPathValidatorTest.java | 100 ++
 2 files changed, 100 insertions(+), 103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a37ec0ae/sdks/java/core/src/test/java/org/apache/beam/runners/dataflow/util/GcsPathValidatorTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/runners/dataflow/util/GcsPathValidatorTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/runners/dataflow/util/GcsPathValidatorTest.java
deleted file mode 100644
index adf4fc2..000
--- 
a/sdks/java/core/src/test/java/org/apache/beam/runners/dataflow/util/GcsPathValidatorTest.java
+++ /dev/null
@@ -1,103 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.dataflow.util;
-
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Mockito.when;
-
-import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.PipelineResult;
-import org.apache.beam.sdk.options.GcsOptions;
-import org.apache.beam.sdk.options.PipelineOptionsFactory;
-import org.apache.beam.sdk.runners.PipelineRunner;
-import org.apache.beam.sdk.util.GcsPathValidator;
-import org.apache.beam.sdk.util.GcsUtil;
-import org.apache.beam.sdk.util.TestCredential;
-import org.apache.beam.sdk.util.gcsfs.GcsPath;
-import org.junit.Before;
-import org.junit.Rule;
-import org.junit.Test;
-import org.junit.rules.ExpectedException;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-import org.mockito.Mock;
-import org.mockito.MockitoAnnotations;
-
-/** Tests for {@link GcsPathValidator}. */
-@RunWith(JUnit4.class)
-public class GcsPathValidatorTest {
-  @Rule public ExpectedException expectedException = ExpectedException.none();
-
-  @Mock private GcsUtil mockGcsUtil;
-  private GcsPathValidator validator;
-
-  private class FakeRunner extends PipelineRunner {
-@Override
-public PipelineResult run(Pipeline pipeline) {
-  throw new UnsupportedOperationException();
-}
-  }
-
-  @Before
-  public void setUp() throws Exception {
-MockitoAnnotations.initMocks(this);
-when(mockGcsUtil.bucketAccessible(any(GcsPath.class))).thenReturn(true);
-when(mockGcsUtil.isGcsPatternSupported(anyString())).thenCallRealMethod();
-GcsOptions options = PipelineOptionsFactory.as(GcsOptions.class);
-options.setRunner(FakeRunner.class);
-options.setGcpCredential(new TestCredential());
-options.setGcsUtil(mockGcsUtil);
-validator = GcsPathValidator.fromOptions(options);
-  }
-
-  @Test
-  public void testValidFilePattern() {
-validator.validateInputFilePatternSupported("gs://bucket/path");
-  }
-
-  @Test
-  public void testInvalidFilePattern() {
-expectedException.expect(IllegalArgumentException.class);
-expectedException.expectMessage(
-"FakeRunner expected a valid 'gs://' path but was given 
'/local/path'");
-validator.validateInputFilePatternSupported("/local/path");
-  }
-
-  @Test
-  public void testWhenBucketDoesNotExist() throws Exception {
-when(mockGcsUtil.bucketAccessible(any(GcsPath.class))).thenReturn(false);
-expectedException.expect(IllegalArgumentException.class);
-

[jira] [Created] (BEAM-737) Update TextSource to not maintain local state

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-737:
---

 Summary: Update TextSource to not maintain local state
 Key: BEAM-737
 URL: https://issues.apache.org/jira/browse/BEAM-737
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Currently, TextSource [1] maintains a temporary buffer as local state. This 
could be problematic if a runner implementation tries to pickle the source. We 
could update TextSource to not maintain any temporary state.

[1] 
https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/textio.py#L32



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1075: Define DataSourceConfiguration as public ...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1075

Define DataSourceConfiguration as public in JdbcIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam JDBCIO_DSCONFIG_PUBLIC

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1075


commit 78c7d8b01a0ca55a17a8e66e2ce16b0f9f58e895
Author: Jean-Baptiste Onofré 
Date:   2016-10-10T19:15:04Z

Define DataSourceConfiguration as public in JdbcIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/6] incubator-beam git commit: Remove direct references to iobase.Native*

[robertwb]

Remove direct references to iobase.Native*


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/60e271b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/60e271b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/60e271b5

Branch: refs/heads/python-sdk
Commit: 60e271b5fe6e42f241b20554ddafd410e87735eb
Parents: 807013a
Author: Robert Bradshaw 
Authored: Thu Oct 6 17:50:41 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:30:00 2016 -0700

--
 sdks/python/apache_beam/io/bigquery.py|  9 ++--
 sdks/python/apache_beam/io/fileio.py  | 17 +++---
 sdks/python/apache_beam/io/fileio_test.py | 73 ++
 sdks/python/apache_beam/io/iobase.py  |  8 +--
 sdks/python/apache_beam/io/pubsub.py  |  5 +-
 sdks/python/apache_beam/pipeline_test.py  |  2 +-
 6 files changed, 60 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/60e271b5/sdks/python/apache_beam/io/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/bigquery.py 
b/sdks/python/apache_beam/io/bigquery.py
index 5508eaa..60d85df 100644
--- a/sdks/python/apache_beam/io/bigquery.py
+++ b/sdks/python/apache_beam/io/bigquery.py
@@ -117,6 +117,7 @@ from apache_beam.internal import auth
 from apache_beam.internal.json_value import from_json_value
 from apache_beam.internal.json_value import to_json_value
 from apache_beam.io import iobase
+from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
 from apache_beam.utils import retry
 from apache_beam.utils.options import GoogleCloudOptions
 
@@ -280,7 +281,7 @@ def _parse_table_reference(table, dataset=None, 
project=None):
 # BigQuerySource, BigQuerySink.
 
 
-class BigQuerySource(iobase.NativeSource):
+class BigQuerySource(dataflow_io.NativeSource):
   """A source based on a BigQuery table."""
 
   def __init__(self, table=None, dataset=None, project=None, query=None,
@@ -345,7 +346,7 @@ class BigQuerySource(iobase.NativeSource):
 source=self, test_bigquery_client=test_bigquery_client)
 
 
-class BigQuerySink(iobase.NativeSink):
+class BigQuerySink(dataflow_io.NativeSink):
   """A sink based on a BigQuery table."""
 
   def __init__(self, table, dataset=None, project=None, schema=None,
@@ -459,7 +460,7 @@ class BigQuerySink(iobase.NativeSink):
 # BigQueryReader, BigQueryWriter.
 
 
-class BigQueryReader(iobase.NativeSourceReader):
+class BigQueryReader(dataflow_io.NativeSourceReader):
   """A reader for a BigQuery source."""
 
   def __init__(self, source, test_bigquery_client=None):
@@ -516,7 +517,7 @@ class BigQueryReader(iobase.NativeSourceReader):
   yield row
 
 
-class BigQueryWriter(iobase.NativeSinkWriter):
+class BigQueryWriter(dataflow_io.NativeSinkWriter):
   """The sink writer for a BigQuerySink."""
 
   def __init__(self, sink, test_bigquery_client=None, buffer_size=None):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/60e271b5/sdks/python/apache_beam/io/fileio.py
--
diff --git a/sdks/python/apache_beam/io/fileio.py 
b/sdks/python/apache_beam/io/fileio.py
index 54ca891..c248f12 100644
--- a/sdks/python/apache_beam/io/fileio.py
+++ b/sdks/python/apache_beam/io/fileio.py
@@ -32,6 +32,7 @@ import weakref
 from apache_beam import coders
 from apache_beam.io import iobase
 from apache_beam.io import range_trackers
+from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
 
 __all__ = ['TextFileSource', 'TextFileSink']
 
@@ -106,7 +107,7 @@ class CompressionTypes(object):
 return cls.UNCOMPRESSED
 
 
-class NativeFileSource(iobase.NativeSource):
+class NativeFileSource(dataflow_io.NativeSource):
   """A source implemented by Dataflow service from a GCS or local file or 
files.
 
   This class is to be only inherited by sources natively implemented by Cloud
@@ -185,7 +186,7 @@ class NativeFileSource(iobase.NativeSource):
 return NativeFileSourceReader(self)
 
 
-class NativeFileSourceReader(iobase.NativeSourceReader,
+class NativeFileSourceReader(dataflow_io.NativeSourceReader,
  coders.observable.ObservableMixin):
   """The source reader for a NativeFileSource.
 
@@ -302,7 +303,7 @@ class NativeFileSourceReader(iobase.NativeSourceReader,
 raise NotImplementedError
 
   def get_progress(self):
-return iobase.ReaderProgress(position=iobase.ReaderPosition(
+return dataflow_io.ReaderProgress(position=dataflow_io.ReaderPosition(
 byte_offset=self.range_tracker.last_record_start))
 
   def request_dynamic_split(self, dynamic_split_request):
@@ -328,7 +329,7 @@ class 

[GitHub] incubator-beam pull request #1067: Move native sources/sinks out of iobase.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1067


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/6] incubator-beam git commit: Avoid circular imports.

[robertwb]

Avoid circular imports.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e2dc6855
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e2dc6855
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e2dc6855

Branch: refs/heads/python-sdk
Commit: e2dc685516d3869a4c2ebfe7d389c5d4c948745b
Parents: 7852075
Author: Robert Bradshaw 
Authored: Thu Oct 6 16:37:45 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:30:00 2016 -0700

--
 sdks/python/apache_beam/runners/direct_runner.py | 9 +++--
 1 file changed, 7 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e2dc6855/sdks/python/apache_beam/runners/direct_runner.py
--
diff --git a/sdks/python/apache_beam/runners/direct_runner.py 
b/sdks/python/apache_beam/runners/direct_runner.py
index a62ddf7..9eb587c 100644
--- a/sdks/python/apache_beam/runners/direct_runner.py
+++ b/sdks/python/apache_beam/runners/direct_runner.py
@@ -30,8 +30,6 @@ import logging
 
 from apache_beam import coders
 from apache_beam import error
-from apache_beam.io import fileio
-from apache_beam.io import iobase
 from apache_beam.pvalue import DictPCollectionView
 from apache_beam.pvalue import EmptySideInput
 from apache_beam.pvalue import IterablePCollectionView
@@ -244,6 +242,9 @@ class DirectPipelineRunner(PipelineRunner):
   transform_node.full_label] += len(read_result)
   self._cache.cache_output(transform_node, read_result)
 
+# pylint: disable=import-at-top
+from apache_beam.io import iobase
+
 if isinstance(source, iobase.BoundedSource):
   # Getting a RangeTracker for the default range of the source and reading
   # the full source using that.
@@ -257,6 +258,10 @@ class DirectPipelineRunner(PipelineRunner):
   @skip_if_cached
   def run__NativeWrite(self, transform_node):
 sink = transform_node.transform.sink
+
+# pylint: disable=import-at-top
+from apache_beam.io import fileio
+
 if isinstance(sink, fileio.NativeTextFileSink):
   assert sink.num_shards in (0, 1)
   if sink.shard_name_template:

[6/6] incubator-beam git commit: Closes #1067

[robertwb]

Closes #1067


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fd27f684
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fd27f684
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fd27f684

Branch: refs/heads/python-sdk
Commit: fd27f6848d5f5b6096c1538bcfeacb5beaf0068f
Parents: 90004a0 893fa2d
Author: Robert Bradshaw 
Authored: Mon Oct 10 12:04:52 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 12:04:52 2016 -0700

--
 sdks/python/apache_beam/io/bigquery.py  |  10 +-
 sdks/python/apache_beam/io/fileio.py|  17 +-
 sdks/python/apache_beam/io/fileio_test.py   |  74 ++---
 sdks/python/apache_beam/io/iobase.py| 299 +-
 sdks/python/apache_beam/io/pubsub.py|   6 +-
 sdks/python/apache_beam/pipeline_test.py|   2 +-
 .../apache_beam/runners/dataflow/__init__.py|   0
 .../runners/dataflow/native_io/__init__.py  |   0
 .../runners/dataflow/native_io/iobase.py| 310 +++
 .../python/apache_beam/runners/direct_runner.py |   9 +-
 .../runners/inprocess/transform_evaluator.py|   5 +-
 11 files changed, 385 insertions(+), 347 deletions(-)
--

[5/6] incubator-beam git commit: Move explicit references to _NativeWrite.

[robertwb]

Move explicit references to _NativeWrite.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/807013a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/807013a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/807013a9

Branch: refs/heads/python-sdk
Commit: 807013a963ecd9f1aaa8551508b2ef45be595380
Parents: e2dc685
Author: Robert Bradshaw 
Authored: Thu Oct 6 17:10:07 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:30:00 2016 -0700

--
 sdks/python/apache_beam/io/iobase.py| 9 -
 .../apache_beam/runners/inprocess/transform_evaluator.py| 5 +++--
 2 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/807013a9/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index b83d7eb..f070b39 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -44,7 +44,6 @@ from apache_beam.transforms import ptransform
 from apache_beam.transforms import window
 
 from apache_beam.runners.dataflow.native_io.iobase import *
-from apache_beam.runners.dataflow.native_io.iobase import _NativeWrite
 
 
 # Encapsulates information about a bundle of a source generated when method
@@ -705,11 +704,11 @@ class Write(ptransform.PTransform):
 self.sink = sink
 
   def apply(self, pcoll):
-from apache_beam.io import iobase
-if isinstance(self.sink, iobase.NativeSink):
+from apache_beam.runners.dataflow.native_io import iobase as native_iobase
+if isinstance(self.sink, native_iobase.NativeSink):
   # A native sink
-  return pcoll | 'native_write' >> _NativeWrite(self.sink)
-elif isinstance(self.sink, iobase.Sink):
+  return pcoll | 'native_write' >> native_iobase._NativeWrite(self.sink)
+elif isinstance(self.sink, Sink):
   # A custom sink
   return pcoll | WriteImpl(self.sink)
 elif isinstance(self.sink, ptransform.PTransform):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/807013a9/sdks/python/apache_beam/runners/inprocess/transform_evaluator.py
--
diff --git a/sdks/python/apache_beam/runners/inprocess/transform_evaluator.py 
b/sdks/python/apache_beam/runners/inprocess/transform_evaluator.py
index 9aeda46..7d9166c 100644
--- a/sdks/python/apache_beam/runners/inprocess/transform_evaluator.py
+++ b/sdks/python/apache_beam/runners/inprocess/transform_evaluator.py
@@ -29,6 +29,7 @@ from apache_beam.runners.common import DoFnRunner
 from apache_beam.runners.common import DoFnState
 from apache_beam.runners.inprocess.inprocess_watermark_manager import 
InProcessWatermarkManager
 from apache_beam.runners.inprocess.inprocess_transform_result import 
InProcessTransformResult
+from apache_beam.runners.dataflow.native_io.iobase import _NativeWrite  # 
pylint: disable=protected-access
 from apache_beam.transforms import core
 from apache_beam.transforms import sideinputs
 from apache_beam.transforms.window import GlobalWindows
@@ -54,7 +55,7 @@ class TransformEvaluatorRegistry(object):
 core.ParDo: _ParDoEvaluator,
 core.GroupByKeyOnly: _GroupByKeyOnlyEvaluator,
 sideinputs.CreatePCollectionView: _CreatePCollectionViewEvaluator,
-io.iobase._NativeWrite: _NativeWriteEvaluator,  # pylint: 
disable=protected-access
+_NativeWrite: _NativeWriteEvaluator,
 }
 
   def for_application(
@@ -98,7 +99,7 @@ class TransformEvaluatorRegistry(object):
 """
 return isinstance(applied_ptransform.transform,
   (core.GroupByKeyOnly, sideinputs.CreatePCollectionView,
-   io.iobase._NativeWrite))  # pylint: 
disable=protected-access
+   _NativeWrite))
 
 
 class _TransformEvaluator(object):

[1/6] incubator-beam git commit: Import Native* in iobase for backwards compatibility.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 90004a09e -> fd27f6848


Import Native* in iobase for backwards compatibility.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/893fa2d4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/893fa2d4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/893fa2d4

Branch: refs/heads/python-sdk
Commit: 893fa2d4b531e23e41f98f817f45cd11e49c200c
Parents: 60e271b
Author: Robert Bradshaw 
Authored: Thu Oct 6 17:57:18 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:30:00 2016 -0700

--
 sdks/python/apache_beam/io/bigquery.py  | 1 -
 sdks/python/apache_beam/io/fileio_test.py   | 1 -
 sdks/python/apache_beam/io/iobase.py| 7 +--
 sdks/python/apache_beam/io/pubsub.py| 1 -
 .../python/apache_beam/runners/dataflow/native_io/iobase.py | 9 -
 sdks/python/apache_beam/runners/direct_runner.py| 6 +++---
 6 files changed, 8 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/893fa2d4/sdks/python/apache_beam/io/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/bigquery.py 
b/sdks/python/apache_beam/io/bigquery.py
index 60d85df..4eecaa5 100644
--- a/sdks/python/apache_beam/io/bigquery.py
+++ b/sdks/python/apache_beam/io/bigquery.py
@@ -116,7 +116,6 @@ from apache_beam import coders
 from apache_beam.internal import auth
 from apache_beam.internal.json_value import from_json_value
 from apache_beam.internal.json_value import to_json_value
-from apache_beam.io import iobase
 from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
 from apache_beam.utils import retry
 from apache_beam.utils.options import GoogleCloudOptions

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/893fa2d4/sdks/python/apache_beam/io/fileio_test.py
--
diff --git a/sdks/python/apache_beam/io/fileio_test.py 
b/sdks/python/apache_beam/io/fileio_test.py
index 77d6c45..b518b97 100644
--- a/sdks/python/apache_beam/io/fileio_test.py
+++ b/sdks/python/apache_beam/io/fileio_test.py
@@ -29,7 +29,6 @@ import zlib
 import apache_beam as beam
 from apache_beam import coders
 from apache_beam.io import fileio
-from apache_beam.io import iobase
 from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
 
 # TODO: Add tests for file patterns (ie not just individual files) for both

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/893fa2d4/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index ac20732..8239e26 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -43,8 +43,6 @@ from apache_beam.transforms import core
 from apache_beam.transforms import ptransform
 from apache_beam.transforms import window
 
-#from apache_beam.runners.dataflow.native_io.iobase import *
-
 
 # Encapsulates information about a bundle of a source generated when method
 # BoundedSource.split() is invoked.
@@ -813,3 +811,8 @@ class _RoundRobinKeyFn(core.DoFn):
 if self.counter >= self.count:
   self.counter -= self.count
 yield self.counter, context.element
+
+
+# For backwards compatibility.
+# pylint: disable=wrong-import-position
+from apache_beam.runners.dataflow.native_io.iobase import *

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/893fa2d4/sdks/python/apache_beam/io/pubsub.py
--
diff --git a/sdks/python/apache_beam/io/pubsub.py 
b/sdks/python/apache_beam/io/pubsub.py
index 1f5989a..da81742 100644
--- a/sdks/python/apache_beam/io/pubsub.py
+++ b/sdks/python/apache_beam/io/pubsub.py
@@ -23,7 +23,6 @@ pipelines, during remote execution.
 from __future__ import absolute_import
 
 from apache_beam import coders
-from apache_beam.io import iobase
 from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/893fa2d4/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py 
b/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
index bccca9f..9621f4c 100644
--- a/sdks/python/apache_beam/runners/dataflow/native_io/iobase.py
+++ 

[3/6] incubator-beam git commit: Move dataflow native sinks and sources into dataflow directory.

[robertwb]

Move dataflow native sinks and sources into dataflow directory.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/78520758
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/78520758
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/78520758

Branch: refs/heads/python-sdk
Commit: 78520758abc3b1c2b38e26f3ffd64e01870de067
Parents: 90004a0
Author: Robert Bradshaw 
Authored: Thu Oct 6 16:52:57 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:30:00 2016 -0700

--
 sdks/python/apache_beam/io/iobase.py| 287 +
 .../apache_beam/runners/dataflow/__init__.py|   0
 .../runners/dataflow/native_io/__init__.py  |   0
 .../runners/dataflow/native_io/iobase.py| 319 +++
 4 files changed, 321 insertions(+), 285 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/78520758/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index 4305fb6..b83d7eb 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -43,263 +43,8 @@ from apache_beam.transforms import core
 from apache_beam.transforms import ptransform
 from apache_beam.transforms import window
 
-
-def _dict_printable_fields(dict_object, skip_fields):
-  """Returns a list of strings for the interesting fields of a dict."""
-  return ['%s=%r' % (name, value)
-  for name, value in dict_object.iteritems()
-  # want to output value 0 but not None nor []
-  if (value or value == 0)
-  and name not in skip_fields]
-
-_minor_fields = ['coder', 'key_coder', 'value_coder',
- 'config_bytes', 'elements',
- 'append_trailing_newlines', 'strip_trailing_newlines',
- 'compression_type']
-
-
-class NativeSource(object):
-  """A source implemented by Dataflow service.
-
-  This class is to be only inherited by sources natively implemented by Cloud
-  Dataflow service, hence should not be sub-classed by users.
-
-  This class is deprecated and should not be used to define new sources.
-  """
-
-  def reader(self):
-"""Returns a NativeSourceReader instance associated with this source."""
-raise NotImplementedError
-
-  def __repr__(self):
-return '<{name} {vals}>'.format(
-name=self.__class__.__name__,
-vals=', '.join(_dict_printable_fields(self.__dict__,
-  _minor_fields)))
-
-
-class NativeSourceReader(object):
-  """A reader for a source implemented by Dataflow service."""
-
-  def __enter__(self):
-"""Opens everything necessary for a reader to function properly."""
-raise NotImplementedError
-
-  def __exit__(self, exception_type, exception_value, traceback):
-"""Cleans up after a reader executed."""
-raise NotImplementedError
-
-  def __iter__(self):
-"""Returns an iterator over all the records of the source."""
-raise NotImplementedError
-
-  @property
-  def returns_windowed_values(self):
-"""Returns whether this reader returns windowed values."""
-return False
-
-  def get_progress(self):
-"""Returns a representation of how far the reader has read.
-
-Returns:
-  A SourceReaderProgress object that gives the current progress of the
-  reader.
-"""
-return
-
-  def request_dynamic_split(self, dynamic_split_request):
-"""Attempts to split the input in two parts.
-
-The two parts are named the "primary" part and the "residual" part. The
-current 'NativeSourceReader' keeps processing the primary part, while the
-residual part will be processed elsewhere (e.g. perhaps on a different
-worker).
-
-The primary and residual parts, if concatenated, must represent the
-same input as the current input of this 'NativeSourceReader' before this
-call.
-
-The boundary between the primary part and the residual part is
-specified in a framework-specific way using 'DynamicSplitRequest' e.g.,
-if the framework supports the notion of positions, it might be a
-position at which the input is asked to split itself (which is not
-necessarily the same position at which it *will* split itself); it
-might be an approximate fraction of input, or something else.
-
-This function returns a 'DynamicSplitResult', which encodes, in a
-framework-specific way, the information sufficient to construct a
-description of the resulting primary and residual inputs. For example, it
-might, again, be a position demarcating these parts, or it might be a pair
-of fully-specified 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1490

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1490

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1074: Move GcsPathValidatorTest to package matc...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1074

Move GcsPathValidatorTest to package matching class under test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
GcsPathValidatorTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1074.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1074


commit a37ec0ae9d07915fe33e932578628b02b3161d02
Author: Kenneth Knowles 
Date:   2016-10-10T18:28:11Z

Move GcsPathValidatorTest to package matching class under test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-736) BigQueryTornadoesIT broken, blocking nightly release.

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-736:
-

 Summary: BigQueryTornadoesIT broken, blocking nightly release.
 Key: BEAM-736
 URL: https://issues.apache.org/jira/browse/BEAM-736
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Jason Kuster
Assignee: Pei He


Build break begins here: 
https://builds.apache.org/job/beam_PostCommit_MavenVerify/1471/

listing 3 potential culprit commits.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1035: [BEAM] Making sure that GcsBufferedReader...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1035


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1035: [BEAM] Making sure that GcsBufferedReader...

[pabloem]

GitHub user pabloem reopened a pull request:

https://github.com/apache/incubator-beam/pull/1035

[BEAM] Making sure that GcsBufferedReader implements the iterator protocol

The change adds only the `__iter__` and `next` functions to the 
`GcsBufferedReader` class, so that it implements the iterator protocol in 
Python.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1035.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1035






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-721) Travis CI fails to run Python tox tests on Mac

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15562929#comment-15562929
 ] 

ASF GitHub Bot commented on BEAM-721:
-

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1068


> Travis CI fails to run Python tox tests on Mac
> --
>
> Key: BEAM-721
> URL: https://issues.apache.org/jira/browse/BEAM-721
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
> Environment: Mac
>Reporter: Pablo Estrada
>Assignee: Frances Perry
>
> Some Travis CI runs on Mac are failing because the test script can not find 
> tox.
> See: https://travis-ci.org/apache/incubator-beam/jobs/165306424#L86
> The travis.yml file does attempt to install tox (See: 
> https://github.com/apache/incubator-beam/blob/python-sdk/.travis.yml#L66)
> Looking at the logs, it seems that tox is available in a different directory 
> (/usr/local), and TOX_HOME is set to $HOME/Library/Python/2.7/bin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1068: [BEAM-721] Testing tox intermittent failu...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/incubator-beam/pull/1068


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1068

[robertwb]

Closes #1068


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/90004a09
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/90004a09
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/90004a09

Branch: refs/heads/python-sdk
Commit: 90004a09e6cc3fff717806aa026610f262e3ad84
Parents: 7d0758b 1176049
Author: Robert Bradshaw 
Authored: Mon Oct 10 10:27:11 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:27:11 2016 -0700

--
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Fixes issue with Travis CI and Mac images.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 7d0758b64 -> 90004a09e


Fixes issue with Travis CI and Mac images.

Tox was be installed in some Mac images, but not others.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1176049b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1176049b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1176049b

Branch: refs/heads/python-sdk
Commit: 1176049b131df06b7f98ee29a39fe1ead25cb33e
Parents: 7d0758b
Author: Pablo 
Authored: Fri Oct 7 15:19:48 2016 -0700
Committer: Robert Bradshaw 
Committed: Mon Oct 10 10:26:17 2016 -0700

--
 .travis.yml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1176049b/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 8d12668..d9d5e1e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -63,7 +63,8 @@ before_install:
 
 install:
   - if [ ! "$TEST_PYTHON" ]; then travis_retry mvn -B install clean -U 
-DskipTests=true; fi
-  - if [ "$TEST_PYTHON" ]; then travis_retry pip install tox --user `whoami`; 
fi
+  - if [ "$TEST_PYTHON" ] && pip list | grep tox; then TOX_FILE=`which tox` ; 
export TOX_HOME=`dirname $TOX_FILE`; fi
+  - if [ "$TEST_PYTHON" ] && ! pip list | grep tox; then travis_retry pip 
install tox --user `whoami`; fi
 
 script:
   - if [ "$TEST_PYTHON" ]; then travis_retry $TOX_HOME/tox -e $TOX_ENV -c 
sdks/python/tox.ini; fi

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1489

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1489

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: BigQueryIO: port trivial fixes from Dataflow version.

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 2492604e4 -> 570de74da


BigQueryIO: port trivial fixes from Dataflow version.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/39b9de5f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/39b9de5f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/39b9de5f

Branch: refs/heads/master
Commit: 39b9de5feab7be37f88e44e99784375a8ae82bc7
Parents: 2492604
Author: Pei He 
Authored: Mon Oct 3 21:19:37 2016 -0700
Committer: Luke Cwik 
Committed: Mon Oct 10 09:01:29 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 21 +++-
 1 file changed, 7 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/39b9de5f/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
index 716fe39..3d1aba6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
@@ -268,12 +268,6 @@ public class BigQueryIO {
 
   private static final Pattern TABLE_SPEC = 
Pattern.compile(DATASET_TABLE_REGEXP);
 
-  // TODO: make this private and remove improper access from 
BigQueryIOTranslator.
-  public static final String SET_PROJECT_FROM_OPTIONS_WARNING =
-  "No project specified for BigQuery table \"%1$s.%2$s\". Assuming it is 
in \"%3$s\". If the"
-  + " table is in a different project please specify it as a part of the 
BigQuery table"
-  + " definition.";
-
   private static final String RESOURCE_NOT_FOUND_ERROR =
   "BigQuery %1$s not found for table \"%2$s\" . Please create the %1$s 
before pipeline"
   + " execution. If the %1$s is created by an earlier stage of the 
pipeline, this"
@@ -614,6 +608,7 @@ public class BigQueryIO {
 JobReference jobRef = new JobReference()
 .setProjectId(executingProject)
 .setJobId(getExtractJobId(jobIdToken));
+
 Job extractJob = bqServices.getJobService(bqOptions)
 .getJob(jobRef);
 
@@ -805,8 +800,7 @@ public class BigQueryIO {
 BigQueryServices bqServices,
 String executingProject) {
   super(jobIdToken, extractDestinationDir, bqServices, executingProject);
-  checkNotNull(table, "table");
-  this.jsonTable = toJsonString(table);
+  this.jsonTable = toJsonString(checkNotNull(table, "table"));
   this.tableSizeBytes = new AtomicReference<>();
 }
 
@@ -960,6 +954,7 @@ public class BigQueryIO {
   super.populateDisplayData(builder);
   builder.add(DisplayData.item("query", query));
 }
+
 private synchronized JobStatistics dryRunQueryIfNeeded(BigQueryOptions 
bqOptions)
 throws InterruptedException, IOException {
   if (dryRunJobStats.get() == null) {
@@ -1755,10 +1750,8 @@ public class BigQueryIO {
   new StreamWithDeDup(getTable(), tableRefFunction, getSchema(), 
bqServices));
 }
 
-TableReference table = fromJsonString(jsonTableRef, 
TableReference.class);
-if (Strings.isNullOrEmpty(table.getProjectId())) {
-  table.setProjectId(options.getProject());
-}
+TableReference table = getTableWithDefaultProject(options);
+
 String jobIdToken = "beam_job_" + randomUUIDString();
 String tempLocation = options.getTempLocation();
 String tempFilePrefix;
@@ -2653,7 +2646,7 @@ public class BigQueryIO {
 public void populateDisplayData(DisplayData.Builder builder) {
   super.populateDisplayData(builder);
 
-  builder.addIfNotNull(DisplayData.item("tableSpec", tableSpec));
+  builder.addIfNotNull(DisplayData.item("table", tableSpec));
   if (tableRefFunction != null) {
 builder.add(DisplayData.item("tableFn", tableRefFunction.getClass())
   .withLabel("Table Reference Function"));
@@ -2745,7 +2738,7 @@ public class BigQueryIO {
 UNKNOWN,
   }
 
-  private static Status parseStatus(Job job) {
+  private static Status parseStatus(@Nullable Job job) {
 if (job == null) {
   return Status.UNKNOWN;
 }

[GitHub] incubator-beam pull request #1042: BigQueryIO: port trivial fixes from Dataf...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1042


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: BigQueryIO: port trivial fixes from Dataflow version.

[lcwik]

BigQueryIO: port trivial fixes from Dataflow version.

This closes #1042


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/570de74d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/570de74d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/570de74d

Branch: refs/heads/master
Commit: 570de74da05082f1a97b54191356f13c7fea5657
Parents: 2492604 39b9de5
Author: Luke Cwik 
Authored: Mon Oct 10 09:02:08 2016 -0700
Committer: Luke Cwik 
Committed: Mon Oct 10 09:02:08 2016 -0700

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 21 +++-
 1 file changed, 7 insertions(+), 14 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1488

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1487

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1487

[Apache Jenkins Server]

See