[jira] [Reopened] (BEAM-4841) Test Auto JIRA Subtask 789

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot reopened BEAM-4841:
-

> Test Auto JIRA Subtask 789
> --
>
> Key: BEAM-4841
> URL: https://issues.apache.org/jira/browse/BEAM-4841
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
> Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #757

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #770

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT #263

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT_HDFS #158

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #439

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #440

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #763

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #791

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #540

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #450

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #871

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #534

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Java_Cron #128

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4769] Fix issue when performing coder inference for wildcard

[qinyeli] Added a round trip through proto in Interactive runner.

[ekirpichov] Avoids use of discouraged Duration(long) ctor

[qinyeli] Interactive Runner -- keep_alive

--
[...truncated 16.45 MB...]
INFO: 2018-07-21T00:19:31.220Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.271Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:364#1d275f544daf228c
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.319Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.368Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.409Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.450Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.488Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.539Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.583Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.618Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.664Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.713Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 21, 2018 12:19:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-21T00:19:31.761Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #439

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Spark #1978

[Apache Jenkins Server]

See 

[beam] 01/01: Merge pull request #6001 from qinyeli/master

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 57b60e36497d8b7935be6558f05413e34974a7df
Merge: 3d77e2b 5a7fbb8
Author: Charles Chen 
AuthorDate: Fri Jul 20 16:39:50 2018 -0700

Merge pull request #6001 from qinyeli/master

Interactive Beam -- keep alive

 .../runners/interactive/interactive_runner.py  | 36 --
 .../runners/interactive/interactive_runner_test.py | 18 +++
 2 files changed, 52 insertions(+), 2 deletions(-)

[beam] branch master updated (3d77e2b -> 57b60e3)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3d77e2b  Merge pull request #6013 from jkff/duration-ms
 add fbd878c  Added a round trip through proto in Interactive runner.
 add 5a7fbb8  Interactive Runner -- keep_alive
 new 57b60e3  Merge pull request #6001 from qinyeli/master

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/interactive/interactive_runner.py  | 36 --
 .../runners/interactive/interactive_runner_test.py | 18 +++
 2 files changed, 52 insertions(+), 2 deletions(-)

[jira] [Closed] (BEAM-4841) Test Auto JIRA Subtask 789

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4841.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> Test Auto JIRA Subtask 789
> --
>
> Key: BEAM-4841
> URL: https://issues.apache.org/jira/browse/BEAM-4841
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
> Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4841) Test Auto JIRA Subtask 789

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4841:
---

 Summary: Test Auto JIRA Subtask 789
 Key: BEAM-4841
 URL: https://issues.apache.org/jira/browse/BEAM-4841
 Project: Beam
  Issue Type: Sub-task
  Components: dependencies
Reporter: Beam JIRA Bot


Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4840) Test Auto JIRA Subtask 123

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4840.
---
   Resolution: Fixed
Fix Version/s: Not applicable

This issue is used to test Beam Auto Jira tools. No actions are required.

> Test Auto JIRA Subtask 123
> --
>
> Key: BEAM-4840
> URL: https://issues.apache.org/jira/browse/BEAM-4840
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
> Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4840) Test Auto JIRA Subtask 123

[Beam JIRA Bot (JIRA)]

Beam JIRA Bot created BEAM-4840:
---

 Summary: Test Auto JIRA Subtask 123
 Key: BEAM-4840
 URL: https://issues.apache.org/jira/browse/BEAM-4840
 Project: Beam
  Issue Type: Sub-task
  Components: dependencies
Reporter: Beam JIRA Bot


Test JIRA issue creation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-4511:
--
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-1251

> Create a tox environment that uses Py3 interpreter for pre/post commit test 
> suites, once codebase supports Py3. 
> 
>
> Key: BEAM-4511
> URL: https://issues.apache.org/jira/browse/BEAM-4511
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3709) Implement the portable lifted Combiner transforms in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3709?focusedWorklogId=125675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125675
 ]

ASF GitHub Bot logged work on BEAM-3709:


Author: ASF GitHub Bot
Created on: 20/Jul/18 22:54
Start Date: 20/Jul/18 22:54
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #6014: [BEAM-3709] 
Implementing new combine URNs in python.
URL: https://github.com/apache/beam/pull/6014
 
 
   Added implementations for the new combine URNs to the bundle processor.
   These implementations are pretty much identical to the implementations
   for existing Python URNs, except for the one for Combine Grouped Values
   which is new.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125675)
Time Spent: 10m
Remaining Estimate: 0h

> Implement the portable lifted Combiner transforms in Python SDK
> ---
>
> Key: BEAM-3709
> URL: https://issues.apache.org/jira/browse/BEAM-3709
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, 

[beam] branch master updated (e4c7e11 -> 3d77e2b)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e4c7e11  [BEAM-4769] Fix issue when performing coder inference for 
wildcard types.
 add bc7005c  Avoids use of discouraged Duration(long) ctor
 new 3d77e2b  Merge pull request #6013 from jkff/duration-ms

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] 01/01: Merge pull request #6013 from jkff/duration-ms

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 3d77e2b61b36f119068526da6104c7015d70c9ac
Merge: e4c7e11 bc7005c
Author: Charles Chen 
AuthorDate: Fri Jul 20 15:49:10 2018 -0700

Merge pull request #6013 from jkff/duration-ms

Avoids use of discouraged Duration(long) ctor

 .../runners/fnexecution/splittabledofn/SDFFeederViaStateAndTimers.java  | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=125666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125666
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 20/Jul/18 22:04
Start Date: 20/Jul/18 22:04
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #503: [BEAM-4471] Update Java 
Quickstart for Windows
URL: https://github.com/apache/beam-site/pull/503#issuecomment-406738441
 
 
   #507 is merged, so you can add the shell- now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125666)
Time Spent: 40m  (was: 0.5h)

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (f28f99c -> 98ecf88)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from f28f99c  Prepare repository for deployment.
 add 1ea2a7f  Fix shell tabs, add new shell tab switcher
 add 3a39ed6  This closes #507
 new 98ecf88  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/quickstart-py/index.html | 37 ++--
 content/js/language-switch.js|  1 +
 src/get-started/quickstart-py.md | 34 +
 src/js/language-switch.js|  1 +
 4 files changed, 39 insertions(+), 34 deletions(-)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 98ecf887ed01b37de60cc3ef1cdf0a8a88fe5f85
Author: Mergebot 
AuthorDate: Fri Jul 20 22:01:00 2018 +

Prepare repository for deployment.
---
 content/get-started/quickstart-py/index.html | 37 ++--
 content/js/language-switch.js|  1 +
 2 files changed, 20 insertions(+), 18 deletions(-)

diff --git a/content/get-started/quickstart-py/index.html 
b/content/get-started/quickstart-py/index.html
index b74852c..f761339 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -256,10 +256,11 @@ limitations under the License.
 If you do not have pip version 7.0.0 
or newer, run the following command to
 install it. This command might require administrative privileges.
 
-pip install --upgrade pip
+pip 
install --upgrade pip
 
 
-PS python -m 
pip install --upgrade pip
+
+PS python -m pip install --upgrade pip
 
 
 
@@ -270,10 +271,11 @@ for initial experiments. If you do not have virt
 newer, run the following command to install it. This command might require
 administrative privileges.
 
-pip install --upgrade virtualenv
+pip 
install --upgrade virtualenv
 
 
-PS python -m 
pip install --upgrade virtualenv
+
+PS python -m pip install --upgrade virtualenv
 
 
 
@@ -281,10 +283,11 @@ administrative privileges.
 setuptools is installed on your 
machine. If you do not have setuptools
 version 17.1 or newer, run the following command to install it.
 
-pip install --upgrade setuptools
+pip 
install --upgrade setuptools
 
 
-PS python -m 
pip install --upgrade setuptools
+
+PS python -m pip install --upgrade setuptools
 
 
 
@@ -294,10 +297,11 @@ version 17.1 or newer, run the following command to 
install it.
 
 A virtual environment is a directory tree containing its own Python 
distribution. To create a virtual environment, create a directory and run:
 
-virtualenv /path/to/directory
+virtualenv /path/to/directory
 
 
-PS virtualenv 
C:\path\to\directory
+
+PS virtualenv C:\path\to\directory
 
 
 
@@ -307,12 +311,11 @@ environment’s directories.
 
 To activate a virtual environment in Bash, run:
 
-{.unix}
-. 
/path/to/directory/bin/activate
+. 
/path/to/directory/bin/activate
 
 
-{.powershell}
-PS 
C:\path\to\directory\Scripts\activate.ps1
+
+PS C:\path\to\directory\Scripts\activate.ps1
 
 
 
@@ -324,10 +327,11 @@ environment’s directories.
 
 Install the latest Python SDK from PyPI:
 
-pip install apache-beam
+pip 
install apache-beam
 
 
-PS python -m 
pip install apache-beam
+
+PS python -m pip install apache-beam
 
 
 
@@ -369,10 +373,7 @@ environment’s directories.
 
 For example, run wordcount.py with 
the following command:
 
-python -m apache_beam.examples.wordcount --input 
/path/to/inputfile --output /path/to/write/counts
-
-
-python -m 
apache_beam.examples.wordcount --input C:\path\to\inputfile --output 
C:\path\to\write\counts
+python -m apache_beam.examples.wordcount --input 
/path/to/inputfile --output /path/to/write/counts
 
 
 
diff --git a/content/js/language-switch.js b/content/js/language-switch.js
index 3e7c9e7..6c30e88 100644
--- a/content/js/language-switch.js
+++ b/content/js/language-switch.js
@@ -127,4 +127,5 @@ $(document).ready(function() {
 
 Switcher({"class-prefix":"language","default":"java"}).render();
 Switcher({"class-prefix":"runner","default":"direct"}).render();
+Switcher({"class-prefix":"shell","default":"unix"}).render();
 });

[beam-site] branch mergebot updated (701e5f0 -> 3a39ed6)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 701e5f0  This closes #504
 add f28f99c  Prepare repository for deployment.
 new 1ea2a7f  Fix shell tabs, add new shell tab switcher
 new 3a39ed6  This closes #507

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/community/contact-us/index.html |  6 +++---
 src/get-started/quickstart-py.md| 34 +
 src/js/language-switch.js   |  1 +
 3 files changed, 22 insertions(+), 19 deletions(-)

[beam-site] 02/02: This closes #507

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 3a39ed64458ee2eef26153d002d2fa6b00abfc60
Merge: f28f99c 1ea2a7f
Author: Mergebot 
AuthorDate: Fri Jul 20 21:57:42 2018 +

This closes #507

 src/get-started/quickstart-py.md | 34 ++
 src/js/language-switch.js|  1 +
 2 files changed, 19 insertions(+), 16 deletions(-)

[beam-site] 01/02: Fix shell tabs, add new shell tab switcher

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 1ea2a7f4e88ea56b45aa0737b1ac2f473d60db30
Author: melissa 
AuthorDate: Fri Jul 20 14:20:01 2018 -0700

Fix shell tabs, add new shell tab switcher
---
 src/get-started/quickstart-py.md | 34 ++
 src/js/language-switch.js|  1 +
 2 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index e9edfea..b199c5e 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -46,11 +46,12 @@ pip --version
 If you do not have `pip` version 7.0.0 or newer, run the following command to
 install it. This command might require administrative privileges.
 
-{:.unix}
+{:.shell-unix}
 ```
 pip install --upgrade pip
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> python -m pip install --upgrade pip
 ```
@@ -63,11 +64,12 @@ for initial experiments. If you do not have `virtualenv` 
version 13.1.0 or
 newer, run the following command to install it. This command might require
 administrative privileges.
 
-{:.unix}
+{:.shell-unix}
 ```
 pip install --upgrade virtualenv
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> python -m pip install --upgrade virtualenv
 ```
@@ -76,11 +78,12 @@ If you do not want to use a Python virtual environment (not 
recommended), ensure
 `setuptools` is installed on your machine. If you do not have `setuptools`
 version 17.1 or newer, run the following command to install it.
 
-{:.unix}
+{:.shell-unix}
 ```
 pip install --upgrade setuptools
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> python -m pip install --upgrade setuptools
 ```
@@ -91,11 +94,12 @@ PS> python -m pip install --upgrade setuptools
 
 A virtual environment is a directory tree containing its own Python 
distribution. To create a virtual environment, create a directory and run:
 
-{:.unix}
+{:.shell-unix}
 ```
 virtualenv /path/to/directory
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> virtualenv C:\path\to\directory
 ```
@@ -106,11 +110,12 @@ environment's directories.
 
 To activate a virtual environment in Bash, run:
 
-{.unix}
+{:.shell-unix}
 ```
 . /path/to/directory/bin/activate
 ```
-{.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> C:\path\to\directory\Scripts\activate.ps1
 ```
@@ -123,11 +128,12 @@ For instructions using other shells, see the [virtualenv 
documentation](https://
 
 Install the latest Python SDK from PyPI:
 
-{:.unix}
+{:.shell-unix}
 ```
 pip install apache-beam
 ```
-{:.powershell}
+
+{:.shell-PowerShell}
 ```
 PS> python -m pip install apache-beam
 ```
@@ -160,10 +166,6 @@ For example, run `wordcount.py` with the following command:
 ```
 python -m apache_beam.examples.wordcount --input /path/to/inputfile --output 
/path/to/write/counts
 ```
-{:.runner-direct-powershell}
-```
-python -m apache_beam.examples.wordcount --input C:\path\to\inputfile --output 
C:\path\to\write\counts
-```
 
 {:.runner-apex}
 ```
diff --git a/src/js/language-switch.js b/src/js/language-switch.js
index 3e7c9e7..6c30e88 100644
--- a/src/js/language-switch.js
+++ b/src/js/language-switch.js
@@ -127,4 +127,5 @@ $(document).ready(function() {
 
 Switcher({"class-prefix":"language","default":"java"}).render();
 Switcher({"class-prefix":"runner","default":"direct"}).render();
+Switcher({"class-prefix":"shell","default":"unix"}).render();
 });

[jira] [Updated] (BEAM-4839) EOF Exception writing non-english Characters to Spanner

[Tom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom updated BEAM-4839:
--
Summary: EOF Exception writing non-english Characters to Spanner  (was: EOF 
Exception with non-english Characters writing to Spanner)

> EOF Exception writing non-english Characters to Spanner
> ---
>
> Key: BEAM-4839
> URL: https://issues.apache.org/jira/browse/BEAM-4839
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-dataflow
>Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: GCP and Local (High Sierra)
>Reporter: Tom
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: Not applicable
>
>
> I am having an issue with Apache Beam ^2.3 and Google Cloud Platform Spanner.
> In short, I'm trying to write data into Spanner. Some of this data contains 
> non-English characters, which blows up the dataflow job when using Beam 2.3 
> or higher.
> Currently, I'm trying to use Apache Beam 2.5, google-api-client 1.23 and Java 
> 1.8.
> This error occurs using Apache Beam 2.3, and not when using 2.2. When using 
> Apache Beam 2.2, we have to drop the google-http/api-client to 1.22 from 1.23.
> {quote}2.5.0{quote}
> {quote}
> 
>  org.apache.beam
>  beam-sdks-java-core
>  ${beam.version}
> 
> 
>  org.apache.beam
>  beam-runners-google-cloud-dataflow-java
>  ${beam.version}
> 
> 
>  org.apache.beam
>  beam-sdks-java-io-google-cloud-platform
>  ${beam.version}
> 
> {quote}
> The google libraries
> {quote}
> 1.23.0
> 0.26.0-alpha
> {quote}
> {quote}
> 
>  com.google.cloud
>  google-cloud
>  ${google-package.version}
> 
> 
>  com.google.http-client
>  google-http-client
>  ${google-clients.version}
>  
>  
>  
>  com.google.guava
>  guava-jdk5
>  
>  
>  
> {quote}
> Here's the stack trace. You'll see that in this quick sample runner we tried 
> inserting 4 rows. Everything runs fine until I attempt to write mutations to 
> the table.
> {quote}
> [INFO] Scanning for projects...
> [INFO]
> [INFO] < com.testing:dataflowtwofive 
> >-
> [INFO] Building dataflowtwofive 0.1
> [INFO] [ jar 
> ]-
> [WARNING] The POM for com.google.oauth-client:google-oauth-client:jar:1.23.0 
> is invalid, transitive dependencies (if any) will not be available, enable 
> debug logging for more details
> [WARNING] The POM for 
> com.google.http-client:google-http-client-jackson:jar:1.23.0 is invalid, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details
> [WARNING] The POM for 
> com.google.apis:google-api-services-storage:jar:v1-rev114-1.23.0 is invalid, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> dataflowtwofive ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 0 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ 
> dataflowtwofive ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ dataflowtwofive ---
> Jul 19, 2018 3:22:41 PM runners.SimpleTestRunner main
> WARNING: ATTN: Coder for mutations: [class com.google.cloud.spanner.Mutation]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=f2360671-557d-4406-b693-c0d66..., LAHQ=a, LAISO=a2, LASPEZ=a3, SPRAS=a4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=cba60659-6986-4dc3-a523-116e2..., LAHQ=á, LAISO=c2, LASPEZ=c3, SPRAS=c4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=889d9d14-df57-4d72-8748-8de0f..., LAHQ=d, LAISO=d2, LASPEZ=d3, SPRAS=d4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=888a2a76-aff7-4f64-9e24-2c231..., LAHQ=b, LAISO=b2, LASPEZ=b3, SPRAS=b4}]
> [WARNING]
> java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>  at 

[jira] [Created] (BEAM-4839) 2.5 Throws EOF Exception with non-english Characters

[Tom (JIRA)]

Tom created BEAM-4839:
-

 Summary: 2.5 Throws EOF Exception with non-english Characters
 Key: BEAM-4839
 URL: https://issues.apache.org/jira/browse/BEAM-4839
 Project: Beam
  Issue Type: Bug
  Components: runner-core, runner-dataflow
Affects Versions: 2.5.0, 2.4.0, 2.3.0
 Environment: GCP and Local (High Sierra)
Reporter: Tom
Assignee: Kenneth Knowles
 Fix For: Not applicable


I am having an issue with Apache Beam ^2.3 and Google Cloud Platform Spanner.

In short, I'm trying to write data into Spanner. Some of this data contains 
non-English characters, which blows up the dataflow job when using Beam 2.3 or 
higher.

Currently, I'm trying to use Apache Beam 2.5, google-api-client 1.23 and Java 
1.8.

This error occurs using Apache Beam 2.3, and not when using 2.2. When using 
Apache Beam 2.2, we have to drop the google-http/api-client to 1.22 from 1.23.

{quote}2.5.0{quote}


{quote}

 org.apache.beam
 beam-sdks-java-core
 ${beam.version}



 org.apache.beam
 beam-runners-google-cloud-dataflow-java
 ${beam.version}



 org.apache.beam
 beam-sdks-java-io-google-cloud-platform
 ${beam.version}

{quote}

The google libraries

{quote}
1.23.0
0.26.0-alpha
{quote}

{quote}

 com.google.cloud
 google-cloud
 ${google-package.version}



 com.google.http-client
 google-http-client
 ${google-clients.version}
 
 
 
 com.google.guava
 guava-jdk5
 
 
 
{quote}

Here's the stack trace. You'll see that in this quick sample runner we tried 
inserting 4 rows. Everything runs fine until I attempt to write mutations to 
the table.

{quote}
[INFO] Scanning for projects...
[INFO]
[INFO] < com.testing:dataflowtwofive >-
[INFO] Building dataflowtwofive 0.1
[INFO] [ jar ]-
[WARNING] The POM for com.google.oauth-client:google-oauth-client:jar:1.23.0 is 
invalid, transitive dependencies (if any) will not be available, enable debug 
logging for more details
[WARNING] The POM for 
com.google.http-client:google-http-client-jackson:jar:1.23.0 is invalid, 
transitive dependencies (if any) will not be available, enable debug logging 
for more details
[WARNING] The POM for 
com.google.apis:google-api-services-storage:jar:v1-rev114-1.23.0 is invalid, 
transitive dependencies (if any) will not be available, enable debug logging 
for more details
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
dataflowtwofive ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ 
dataflowtwofive ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ dataflowtwofive ---
Jul 19, 2018 3:22:41 PM runners.SimpleTestRunner main
WARNING: ATTN: Coder for mutations: [class com.google.cloud.spanner.Mutation]

Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
WARNING: ATTN: mutation: 
[\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
GUID=f2360671-557d-4406-b693-c0d66..., LAHQ=a, LAISO=a2, LASPEZ=a3, SPRAS=a4}]

Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
WARNING: ATTN: mutation: 
[\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
GUID=cba60659-6986-4dc3-a523-116e2..., LAHQ=á, LAISO=c2, LASPEZ=c3, SPRAS=c4}]

Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
WARNING: ATTN: mutation: 
[\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
GUID=889d9d14-df57-4d72-8748-8de0f..., LAHQ=d, LAISO=d2, LASPEZ=d3, SPRAS=d4}]

Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
WARNING: ATTN: mutation: 
[\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
GUID=888a2a76-aff7-4f64-9e24-2c231..., LAHQ=b, LAISO=b2, LASPEZ=b3, SPRAS=b4}]

[WARNING]
java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke (Method.java:498)
 at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:293)
 at java.lang.Thread.run (Thread.java:748)
Caused by: org.apache.beam.sdk.Pipeline$PipelineExecutionException: 
java.lang.RuntimeException: java.io.EOFException
 at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish
 (DirectRunner.java:349)
 at 
org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish
 (DirectRunner.java:319)

[jira] [Updated] (BEAM-4839) EOF Exception with non-english Characters writing to Spanner

[Tom (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom updated BEAM-4839:
--
Summary: EOF Exception with non-english Characters writing to Spanner  
(was: 2.5 Throws EOF Exception with non-english Characters)

> EOF Exception with non-english Characters writing to Spanner
> 
>
> Key: BEAM-4839
> URL: https://issues.apache.org/jira/browse/BEAM-4839
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-dataflow
>Affects Versions: 2.3.0, 2.4.0, 2.5.0
> Environment: GCP and Local (High Sierra)
>Reporter: Tom
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: Not applicable
>
>
> I am having an issue with Apache Beam ^2.3 and Google Cloud Platform Spanner.
> In short, I'm trying to write data into Spanner. Some of this data contains 
> non-English characters, which blows up the dataflow job when using Beam 2.3 
> or higher.
> Currently, I'm trying to use Apache Beam 2.5, google-api-client 1.23 and Java 
> 1.8.
> This error occurs using Apache Beam 2.3, and not when using 2.2. When using 
> Apache Beam 2.2, we have to drop the google-http/api-client to 1.22 from 1.23.
> {quote}2.5.0{quote}
> {quote}
> 
>  org.apache.beam
>  beam-sdks-java-core
>  ${beam.version}
> 
> 
>  org.apache.beam
>  beam-runners-google-cloud-dataflow-java
>  ${beam.version}
> 
> 
>  org.apache.beam
>  beam-sdks-java-io-google-cloud-platform
>  ${beam.version}
> 
> {quote}
> The google libraries
> {quote}
> 1.23.0
> 0.26.0-alpha
> {quote}
> {quote}
> 
>  com.google.cloud
>  google-cloud
>  ${google-package.version}
> 
> 
>  com.google.http-client
>  google-http-client
>  ${google-clients.version}
>  
>  
>  
>  com.google.guava
>  guava-jdk5
>  
>  
>  
> {quote}
> Here's the stack trace. You'll see that in this quick sample runner we tried 
> inserting 4 rows. Everything runs fine until I attempt to write mutations to 
> the table.
> {quote}
> [INFO] Scanning for projects...
> [INFO]
> [INFO] < com.testing:dataflowtwofive 
> >-
> [INFO] Building dataflowtwofive 0.1
> [INFO] [ jar 
> ]-
> [WARNING] The POM for com.google.oauth-client:google-oauth-client:jar:1.23.0 
> is invalid, transitive dependencies (if any) will not be available, enable 
> debug logging for more details
> [WARNING] The POM for 
> com.google.http-client:google-http-client-jackson:jar:1.23.0 is invalid, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details
> [WARNING] The POM for 
> com.google.apis:google-api-services-storage:jar:v1-rev114-1.23.0 is invalid, 
> transitive dependencies (if any) will not be available, enable debug logging 
> for more details
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
> dataflowtwofive ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 0 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ 
> dataflowtwofive ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ dataflowtwofive ---
> Jul 19, 2018 3:22:41 PM runners.SimpleTestRunner main
> WARNING: ATTN: Coder for mutations: [class com.google.cloud.spanner.Mutation]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=f2360671-557d-4406-b693-c0d66..., LAHQ=a, LAISO=a2, LASPEZ=a3, SPRAS=a4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=cba60659-6986-4dc3-a523-116e2..., LAHQ=á, LAISO=c2, LASPEZ=c3, SPRAS=c4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=889d9d14-df57-4d72-8748-8de0f..., LAHQ=d, LAISO=d2, LASPEZ=d3, SPRAS=d4}]
> Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement
> WARNING: ATTN: mutation: 
> [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.76500Z, 
> ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.80900Z, 
> GUID=888a2a76-aff7-4f64-9e24-2c231..., LAHQ=b, LAISO=b2, LASPEZ=b3, SPRAS=b4}]
> [WARNING]
> java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:62)
>  at 

[jira] [Work logged] (BEAM-2660) Set PubsubIO batch size using builder

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2660?focusedWorklogId=125658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125658
 ]

ASF GitHub Bot logged work on BEAM-2660:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:28
Start Date: 20/Jul/18 21:28
Worklog Time Spent: 10m 
  Work Description: cjmcgraw commented on issue #3619: [BEAM-2660] Set 
PubsubIO batch size using builder
URL: https://github.com/apache/beam/pull/3619#issuecomment-406731653
 
 
   @aromanenko-dev pubsub is google cloud specific. But this change is not 
runner specific


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125658)
Time Spent: 1h 50m  (was: 1h 40m)

> Set PubsubIO batch size using builder
> -
>
> Key: BEAM-2660
> URL: https://issues.apache.org/jira/browse/BEAM-2660
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Carl McGraw
>Assignee: Chamikara Jayalath
>Priority: Major
>  Labels: gcp, java, pubsub, sdk
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> PubsubIO doesn't allow users to set the publish batch size. Instead the value 
> is hard coded in both the BoundedPubsubWriter and the UnboundedPubsubSink. 
> google's pub/sub is bound to a maximum of 10mb per request size. My company 
> has run into problems with events that are individually smaller than 1mb, but 
> when batched in the 100 or 2000 default batch sizes causes pubsub to fail to 
> send the event.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-4788) Test Auto JIRA subtask456

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reopened BEAM-4788:
-

> Test Auto JIRA subtask456
> -
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> updated couple feilds of this issue by using jira api



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=125656=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125656
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:22
Start Date: 20/Jul/18 21:22
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204172671
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -45,9 +46,12 @@
   private final ListeningExecutorService executor;
   private final ServerConfiguration configuration;
   private final ServerFactory serverFactory;
+  private GrpcFnServer jobServer;
+  private GrpcFnServer 
artifactStagingServer;
 
-  private static class ServerConfiguration {
-@Option(name = "--job-host", required = true, usage = "The job server host 
string")
+  /** Configuration for the jobServer. */
+  public static class ServerConfiguration {
+@Option(name = "--job-host", usage = "The job server host string")
 
 Review comment:
   That seems to be possible.
   Comparing the approach 
   1. Optional port and query the port back
   2. Passing a socket to the flink driver
   
   I feel option 1 seems to be simpler while keeping initialization flexible.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125656)
Time Spent: 1h 20m  (was: 1h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125652
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:15
Start Date: 20/Jul/18 21:15
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #5989: [BEAM-4838] Add 
dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#issuecomment-406728886
 
 
   @lgajowy Thank you for review, comments addressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125652)
Time Spent: 1h 20m  (was: 1h 10m)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125651
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:14
Start Date: 20/Jul/18 21:14
Worklog Time Spent: 10m 
  Work Description: angoenka closed pull request #6007: [BEAM-4834] 
Identify loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
index 9f41a6663f0..702a8f8f1e3 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
@@ -176,6 +176,28 @@ public final NodeT apply(NodeT input) {
 return computeTopologicalOrder(orderedNetwork);
   }
 
+  /**
+   * Do a simple DFS traversal to find cycle in the graph. Not using 
Graphs.hasCycle as it does not
+   * provide the found cycle.
+   */
+  private static  void checkCycle(Network network) 
{
+ArrayDeque parents = new ArrayDeque<>();
+HashSet parentsSet = new HashSet<>();
+network.nodes().forEach(node -> dfs(network, node, parents, parentsSet));
+  }
+
+  private static  void dfs(
+  Network network, NodeT node, ArrayDeque parents, 
Set parentsSet) {
+if (parentsSet.contains(node)) {
+  checkArgument(!parents.contains(node), "Network has a cycle %s on node 
%s", parents, node);
+}
+parents.addLast(node);
+parentsSet.add(node);
+network.successors(node).forEach(successor -> dfs(network, successor, 
parents, parentsSet));
+parentsSet.remove(node);
+parents.removeLast();
+  }
+
   /**
* Compute the topological order for a {@link Network}.
*
@@ -193,6 +215,7 @@ public final NodeT apply(NodeT input) {
 "Only networks without self loops are supported, given %s",
 network);
 
+checkCycle(network);
 // Linked hashset will prevent duplicates from appearing and will maintain 
insertion order.
 LinkedHashSet nodes = new LinkedHashSet<>(network.nodes().size());
 Queue processingOrder = new ArrayDeque<>();
diff --git 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
index 2dd8834585f..20f7d9700a1 100644
--- 
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
+++ 
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java
@@ -83,6 +83,46 @@ public void testTopologicalSort() {
 }
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testTopologicalSortWithCycle() {
+MutableNetwork network = createEmptyNetwork();
+network.addNode("A");
+network.addNode("B");
+network.addNode("C");
+network.addNode("D");
+network.addNode("E");
+
+network.addEdge("A", "B", "AB");
+network.addEdge("B", "C", "BC");
+network.addEdge("C", "D", "CD");
+network.addEdge("D", "E", "DE");
+// Cycle between B -> C -> D ->
+network.addEdge("D", "B", "DB");
+
+Iterable sortedNodes = Networks.topologicalOrder(network);
+Map nodeToPosition = new 
HashMap<>(Iterables.size(sortedNodes));
+int i = 0;
+for (String node : sortedNodes) {
+  nodeToPosition.put(node, i);
+  i += 1;
+}
+
+for (String node : network.nodes()) {
+  for (String descendant :
+  Sets.difference(
+  Networks.reachableNodes(network, ImmutableSet.of(node), 
Collections.emptySet()),
+  Sets.newHashSet(node))) {
+assertThat(
+String.format(
+"Expected position of node %s to be before descendant %s,"
++ " order returned %s for network %s",
+node, descendant, sortedNodes, network),
+nodeToPosition.get(descendant),
+greaterThan(nodeToPosition.get(node)));
+  }
+}
+  }
+
   @Test
   public void testTopologicalSortWithSuborder() {
 // This cast is required to narrow the type accepted by the comparator


 


This is an automated message from the Apache Git 

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=125650=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125650
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:11
Start Date: 20/Jul/18 21:11
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on a change in pull request #5935: 
[BEAM-4176] Initial implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#discussion_r204170154
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkJobServerDriver.java
 ##
 @@ -45,9 +46,12 @@
   private final ListeningExecutorService executor;
   private final ServerConfiguration configuration;
   private final ServerFactory serverFactory;
+  private GrpcFnServer jobServer;
+  private GrpcFnServer 
artifactStagingServer;
 
-  private static class ServerConfiguration {
-@Option(name = "--job-host", required = true, usage = "The job server host 
string")
+  /** Configuration for the jobServer. */
+  public static class ServerConfiguration {
+@Option(name = "--job-host", usage = "The job server host string")
 
 Review comment:
   Is this a local port? If so, it's quite easy to do in Java by allocating a 
new TCP server socket at port 0. You can then query it for the chosen port. 
This works as long as you are able to pass that socket directly to the service 
implementation. (Note that you can do this with gRPC and we have a 
Beam-specific tool that does this for you).
   
   If you're trying to find an unused port for a non-Java server or for a 
server that will run separately (i.e., for which you do not directly control 
the socket), you can use the same server allocation trick and note the chosen 
port number. You can then kill the server socket and pass that port into your 
new server's configuration. Note that doing this approach introduces a race 
condition but will work in most cases (especially with retries).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125650)
Time Spent: 1h 10m  (was: 1h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125649
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:07
Start Date: 20/Jul/18 21:07
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6007: [BEAM-4834] Identify 
loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007#issuecomment-406727242
 
 
   Without identifying loop, topological sort will go into infinite loop.
   To make sure we fail in a timely and meaningful manner, we should fail on 
loop.
   And we can revisit to see how can we support loops in the pipeline.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125649)
Time Spent: 1h  (was: 50m)

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Cyclic graphs will never finish the topological sort so we should check the 
> cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=125648=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125648
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 20/Jul/18 21:05
Start Date: 20/Jul/18 21:05
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5935: [BEAM-4176] Initial 
implementation for running portable runner tests
URL: https://github.com/apache/beam/pull/5935#issuecomment-406726774
 
 
   Ping for the review!
   cc: @bsidhom @lukecwik @axelmagn @ryan-williams 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125648)
Time Spent: 1h  (was: 50m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f28f99c2d686b7f91b982f4532a948a63c5ff037
Author: Mergebot 
AuthorDate: Fri Jul 20 20:51:01 2018 +

Prepare repository for deployment.
---
 content/community/contact-us/index.html | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/content/community/contact-us/index.html 
b/content/community/contact-us/index.html
index bad1306..08e51aa 100644
--- a/content/community/contact-us/index.html
+++ b/content/community/contact-us/index.html
@@ -205,15 +205,15 @@ whichever one seems best!
   
 
   https://lists.apache.org/list.html?u...@beam.apache.org;>user@ 
mailing list
-  User support and questions (mailto:user-subscr...@beam.apache.org;>Subscribe)
+  User support and questions (mailto:user-subscr...@beam.apache.org;>Subscribe, mailto:user-unsubscr...@beam.apache.org;>Unsubscribe)
 
 
   https://lists.apache.org/list.html?d...@beam.apache.org;>dev@ mailing 
list
-  Development discussions (mailto:dev-subscr...@beam.apache.org;>Subscribe)
+  Development discussions (mailto:dev-subscr...@beam.apache.org;>Subscribe, mailto:dev-unsubscr...@beam.apache.org;>Unsubscribe)
 
 
   https://lists.apache.org/list.html?commits@beam.apache.org;>commits@ 
mailing list
-  Firehose of commits, bugs, pull requests, etc. (mailto:commits-subscr...@beam.apache.org;>Subscribe)
+  Firehose of commits, bugs, pull requests, etc. (mailto:commits-subscr...@beam.apache.org;>Subscribe, mailto:commits-unsubscr...@beam.apache.org;>Unsubscribe)
 
 
   https://issues.apache.org/jira/browse/BEAM;>JIRA bug 
tracker

[beam-site] branch asf-site updated (990b9eb -> f28f99c)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 990b9eb  Prepare repository for deployment.
 add c636154  Add courtesy unsubscribe links to mailing lists
 add 701e5f0  This closes #504
 new f28f99c  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/community/contact-us/index.html | 6 +++---
 src/community/contact-us.md | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)

[beam-site] 01/02: Add courtesy unsubscribe links to mailing lists

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c6361549d540be759ebb00b733d4c10f072ad140
Author: Rafael Fernandez 
AuthorDate: Thu Jul 19 19:22:14 2018 -0700

Add courtesy unsubscribe links to mailing lists

It's not obvious for a casual user that unsubscribe instructions are 
included in the headers of mails sent by Pony mail...
---
 src/community/contact-us.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/community/contact-us.md b/src/community/contact-us.md
index a508e42..e0639ad 100644
--- a/src/community/contact-us.md
+++ b/src/community/contact-us.md
@@ -29,9 +29,9 @@ whichever one seems best!
 
 | How to contact us | When to use it |
 | - | ---|
-| [user@](https://lists.apache.org/list.html?u...@beam.apache.org) mailing 
list | User support and questions 
([Subscribe](mailto:user-subscr...@beam.apache.org)) |
-| [dev@](https://lists.apache.org/list.html?d...@beam.apache.org) mailing list 
| Development discussions ([Subscribe](mailto:dev-subscr...@beam.apache.org)) |
-| [commits@](https://lists.apache.org/list.html?commits@beam.apache.org) 
mailing list | Firehose of commits, bugs, pull requests, etc. 
([Subscribe](mailto:commits-subscr...@beam.apache.org)) |
+| [user@](https://lists.apache.org/list.html?u...@beam.apache.org) mailing 
list | User support and questions 
([Subscribe](mailto:user-subscr...@beam.apache.org), 
[Unsubscribe](mailto:user-unsubscr...@beam.apache.org)) |
+| [dev@](https://lists.apache.org/list.html?d...@beam.apache.org) mailing list 
| Development discussions ([Subscribe](mailto:dev-subscr...@beam.apache.org), 
[Unsubscribe](mailto:dev-unsubscr...@beam.apache.org)) |
+| [commits@](https://lists.apache.org/list.html?commits@beam.apache.org) 
mailing list | Firehose of commits, bugs, pull requests, etc. 
([Subscribe](mailto:commits-subscr...@beam.apache.org), 
[Unsubscribe](mailto:commits-unsubscr...@beam.apache.org)) |
 | [JIRA bug tracker](https://issues.apache.org/jira/browse/BEAM) | Report bugs 
/ discover known issues |
 | [StackOverflow](http://stackoverflow.com/questions/tagged/apache-beam) | Ask 
and answer user support questions |
 | [Slack](https://s.apache.org/beam-slack-channel) | Chat with users and 
developers ([Join](https://s.apache.org/slack-invite)) |

[beam-site] 02/02: This closes #504

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 701e5f02da973b062c206b92f2b4f80afbe012c8
Merge: 990b9eb c636154
Author: Mergebot 
AuthorDate: Fri Jul 20 20:48:07 2018 +

This closes #504

 src/community/contact-us.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

[beam-site] branch mergebot updated (d00682c -> 701e5f0)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from d00682c  This closes #505
 add 990b9eb  Prepare repository for deployment.
 new c636154  Add courtesy unsubscribe links to mailing lists
 new 701e5f0  This closes #504

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html |  8 --
 content/get-started/quickstart-py/index.html | 37 ++--
 src/community/contact-us.md  |  6 ++---
 3 files changed, 39 insertions(+), 12 deletions(-)

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=125645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125645
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 20/Jul/18 20:46
Start Date: 20/Jul/18 20:46
Worklog Time Spent: 10m 
  Work Description: ThatRfernand commented on issue #503: [BEAM-4471] 
Update Java Quickstart for Windows
URL: https://github.com/apache/beam-site/pull/503#issuecomment-406722234
 
 
   'bot is back online -- thank you @jasonkuster !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125645)
Time Spent: 0.5h  (was: 20m)

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=125644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125644
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 20/Jul/18 20:42
Start Date: 20/Jul/18 20:42
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #505: [BEAM-4471] Update 
Python QuickStart with Windows instructions
URL: https://github.com/apache/beam-site/pull/505
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index 3447aa461..e9edfeaaf 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -46,9 +46,14 @@ pip --version
 If you do not have `pip` version 7.0.0 or newer, run the following command to
 install it. This command might require administrative privileges.
 
+{:.unix}
 ```
 pip install --upgrade pip
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade pip
+```
 
 
 ### Install Python virtual environment
@@ -58,17 +63,27 @@ for initial experiments. If you do not have `virtualenv` 
version 13.1.0 or
 newer, run the following command to install it. This command might require
 administrative privileges.
 
+{:.unix}
 ```
 pip install --upgrade virtualenv
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade virtualenv
+```
 
 If you do not want to use a Python virtual environment (not recommended), 
ensure
 `setuptools` is installed on your machine. If you do not have `setuptools`
 version 17.1 or newer, run the following command to install it.
 
+{:.unix}
 ```
 pip install --upgrade setuptools
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade setuptools
+```
 
 ## Get Apache Beam
 
@@ -76,9 +91,14 @@ pip install --upgrade setuptools
 
 A virtual environment is a directory tree containing its own Python 
distribution. To create a virtual environment, create a directory and run:
 
+{:.unix}
 ```
 virtualenv /path/to/directory
 ```
+{:.powershell}
+```
+PS> virtualenv C:\path\to\directory
+```
 
 A virtual environment needs to be activated for each shell that is to use it.
 Activating it sets some environment variables that point to the virtual
@@ -86,11 +106,16 @@ environment's directories.
 
 To activate a virtual environment in Bash, run:
 
+{.unix}
 ```
 . /path/to/directory/bin/activate
 ```
+{.powershell}
+```
+PS> C:\path\to\directory\Scripts\activate.ps1
+```
 
-That is, source the script `bin/activate` under the virtual environment 
directory you created.
+That is, execute the `activate` script under the virtual environment directory 
you created.
 
 For instructions using other shells, see the [virtualenv 
documentation](https://virtualenv.pypa.io/en/stable/userguide/#activate-script).
 
@@ -98,9 +123,14 @@ For instructions using other shells, see the [virtualenv 
documentation](https://
 
 Install the latest Python SDK from PyPI:
 
+{:.unix}
 ```
 pip install apache-beam
 ```
+{:.powershell}
+```
+PS> python -m pip install apache-beam
+```
 
  Extra requirements
 
@@ -130,6 +160,10 @@ For example, run `wordcount.py` with the following command:
 ```
 python -m apache_beam.examples.wordcount --input /path/to/inputfile --output 
/path/to/write/counts
 ```
+{:.runner-direct-powershell}
+```
+python -m apache_beam.examples.wordcount --input C:\path\to\inputfile --output 
C:\path\to\write\counts
+```
 
 {:.runner-apex}
 ```


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125644)
Time Spent: 20m  (was: 10m)

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 990b9eb77d0c0eba7aa95bf7e82d7572e8ed9e48
Author: Mergebot 
AuthorDate: Fri Jul 20 20:42:35 2018 +

Prepare repository for deployment.
---
 content/documentation/io/built-in/index.html |  8 --
 content/get-started/quickstart-py/index.html | 37 ++--
 2 files changed, 36 insertions(+), 9 deletions(-)

diff --git a/content/documentation/io/built-in/index.html 
b/content/documentation/io/built-in/index.html
index a4aa05f..9a8de18 100644
--- a/content/documentation/io/built-in/index.html
+++ b/content/documentation/io/built-in/index.html
@@ -358,6 +358,10 @@ limitations under the License.
 https://issues.apache.org/jira/browse/BEAM-607;>BEAM-607
   
   
+Apache KafkaPython
+https://issues.apache.org/jira/browse/BEAM-3788;>BEAM-3788
+  
+  
 Apache KuduJava
 https://issues.apache.org/jira/browse/BEAM-2661;>BEAM-2661
 
@@ -390,8 +394,8 @@ limitations under the License.
 https://issues.apache.org/jira/browse/BEAM-1946;>BEAM-1946
   
   
-Apache KafkaPython
-https://issues.apache.org/jira/browse/BEAM-3788;>BEAM-3788
+Amazon SQSJava
+https://issues.apache.org/jira/browse/BEAM-4828;>BEAM-4828
   
 
 
diff --git a/content/get-started/quickstart-py/index.html 
b/content/get-started/quickstart-py/index.html
index a504f20..b74852c 100644
--- a/content/get-started/quickstart-py/index.html
+++ b/content/get-started/quickstart-py/index.html
@@ -256,7 +256,10 @@ limitations under the License.
 If you do not have pip version 7.0.0 
or newer, run the following command to
 install it. This command might require administrative privileges.
 
-pip install 
--upgrade pip
+pip install --upgrade pip
+
+
+PS python -m 
pip install --upgrade pip
 
 
 
@@ -267,7 +270,10 @@ for initial experiments. If you do not have virt
 newer, run the following command to install it. This command might require
 administrative privileges.
 
-pip install 
--upgrade virtualenv
+pip install --upgrade virtualenv
+
+
+PS python -m 
pip install --upgrade virtualenv
 
 
 
@@ -275,7 +281,10 @@ administrative privileges.
 setuptools is installed on your 
machine. If you do not have setuptools
 version 17.1 or newer, run the following command to install it.
 
-pip install 
--upgrade setuptools
+pip install --upgrade setuptools
+
+
+PS python -m 
pip install --upgrade setuptools
 
 
 
@@ -285,7 +294,10 @@ version 17.1 or newer, run the following command to 
install it.
 
 A virtual environment is a directory tree containing its own Python 
distribution. To create a virtual environment, create a directory and run:
 
-virtualenv 
/path/to/directory
+virtualenv /path/to/directory
+
+
+PS virtualenv 
C:\path\to\directory
 
 
 
@@ -295,11 +307,16 @@ environment’s directories.
 
 To activate a virtual environment in Bash, run:
 
+{.unix}
 . 
/path/to/directory/bin/activate
 
 
+{.powershell}
+PS 
C:\path\to\directory\Scripts\activate.ps1
+
+
 
-That is, source the script bin/activate under the virtual environment 
directory you created.
+That is, execute the activate script 
under the virtual environment directory you created.
 
 For instructions using other shells, see the https://virtualenv.pypa.io/en/stable/userguide/#activate-script;>virtualenv
 documentation.
 
@@ -307,7 +324,10 @@ environment’s directories.
 
 Install the latest Python SDK from PyPI:
 
-pip install 
apache-beam
+pip install apache-beam
+
+
+PS python -m 
pip install apache-beam
 
 
 
@@ -349,7 +369,10 @@ environment’s directories.
 
 For example, run wordcount.py with 
the following command:
 
-python -m apache_beam.examples.wordcount --input 
/path/to/inputfile --output /path/to/write/counts
+python -m apache_beam.examples.wordcount --input 
/path/to/inputfile --output /path/to/write/counts
+
+
+python -m 
apache_beam.examples.wordcount --input C:\path\to\inputfile --output 
C:\path\to\write\counts
 
 
 

[beam-site] branch asf-site updated (48cc1c3 -> 990b9eb)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 48cc1c3  [BEAM-4828] Updated documentation, added SQS BEAM-4828 to 
In-Progress
 add feacae2  Update Python quickstart with Windows instructions
 add d00682c  This closes #505
 new 990b9eb  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/io/built-in/index.html |  8 --
 content/get-started/quickstart-py/index.html | 37 ++--
 src/get-started/quickstart-py.md | 36 ++-
 3 files changed, 71 insertions(+), 10 deletions(-)

[beam-site] branch mergebot updated (8b99952 -> d00682c)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 8b99952  This closes #501
 add 48cc1c3  [BEAM-4828] Updated documentation, added SQS BEAM-4828 to 
In-Progress
 new feacae2  Update Python quickstart with Windows instructions
 new d00682c  This closes #505

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (8b99952)
\
 N -- N -- N   refs/heads/mergebot (d00682c)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/get-started/quickstart-py.md | 36 +++-
 1 file changed, 35 insertions(+), 1 deletion(-)

[beam-site] 02/02: This closes #505

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d00682cc1e1b344d0924bbf6f8230be9290a7681
Merge: 48cc1c3 feacae2
Author: Mergebot 
AuthorDate: Fri Jul 20 20:39:38 2018 +

This closes #505

 src/get-started/quickstart-py.md | 36 +++-
 1 file changed, 35 insertions(+), 1 deletion(-)

[beam-site] 01/02: Update Python quickstart with Windows instructions

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit feacae2a9c0aa069fa60cb43b41f1017fa2fa6d6
Author: Rafael Fernandez 
AuthorDate: Fri Jul 20 11:48:55 2018 -0700

Update Python quickstart with Windows instructions

PowerShell-specific instructions for Beamers using Microsoft Windows
---
 src/get-started/quickstart-py.md | 36 +++-
 1 file changed, 35 insertions(+), 1 deletion(-)

diff --git a/src/get-started/quickstart-py.md b/src/get-started/quickstart-py.md
index 3447aa4..e9edfea 100644
--- a/src/get-started/quickstart-py.md
+++ b/src/get-started/quickstart-py.md
@@ -46,9 +46,14 @@ pip --version
 If you do not have `pip` version 7.0.0 or newer, run the following command to
 install it. This command might require administrative privileges.
 
+{:.unix}
 ```
 pip install --upgrade pip
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade pip
+```
 
 
 ### Install Python virtual environment
@@ -58,17 +63,27 @@ for initial experiments. If you do not have `virtualenv` 
version 13.1.0 or
 newer, run the following command to install it. This command might require
 administrative privileges.
 
+{:.unix}
 ```
 pip install --upgrade virtualenv
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade virtualenv
+```
 
 If you do not want to use a Python virtual environment (not recommended), 
ensure
 `setuptools` is installed on your machine. If you do not have `setuptools`
 version 17.1 or newer, run the following command to install it.
 
+{:.unix}
 ```
 pip install --upgrade setuptools
 ```
+{:.powershell}
+```
+PS> python -m pip install --upgrade setuptools
+```
 
 ## Get Apache Beam
 
@@ -76,9 +91,14 @@ pip install --upgrade setuptools
 
 A virtual environment is a directory tree containing its own Python 
distribution. To create a virtual environment, create a directory and run:
 
+{:.unix}
 ```
 virtualenv /path/to/directory
 ```
+{:.powershell}
+```
+PS> virtualenv C:\path\to\directory
+```
 
 A virtual environment needs to be activated for each shell that is to use it.
 Activating it sets some environment variables that point to the virtual
@@ -86,11 +106,16 @@ environment's directories.
 
 To activate a virtual environment in Bash, run:
 
+{.unix}
 ```
 . /path/to/directory/bin/activate
 ```
+{.powershell}
+```
+PS> C:\path\to\directory\Scripts\activate.ps1
+```
 
-That is, source the script `bin/activate` under the virtual environment 
directory you created.
+That is, execute the `activate` script under the virtual environment directory 
you created.
 
 For instructions using other shells, see the [virtualenv 
documentation](https://virtualenv.pypa.io/en/stable/userguide/#activate-script).
 
@@ -98,9 +123,14 @@ For instructions using other shells, see the [virtualenv 
documentation](https://
 
 Install the latest Python SDK from PyPI:
 
+{:.unix}
 ```
 pip install apache-beam
 ```
+{:.powershell}
+```
+PS> python -m pip install apache-beam
+```
 
  Extra requirements
 
@@ -130,6 +160,10 @@ For example, run `wordcount.py` with the following command:
 ```
 python -m apache_beam.examples.wordcount --input /path/to/inputfile --output 
/path/to/write/counts
 ```
+{:.runner-direct-powershell}
+```
+python -m apache_beam.examples.wordcount --input C:\path\to\inputfile --output 
C:\path\to\write\counts
+```
 
 {:.runner-apex}
 ```

[jira] [Work logged] (BEAM-4471) Documentation for Windows SDK users

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?focusedWorklogId=125640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125640
 ]

ASF GitHub Bot logged work on BEAM-4471:


Author: ASF GitHub Bot
Created on: 20/Jul/18 20:38
Start Date: 20/Jul/18 20:38
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #505: [BEAM-4471] Update 
Python QuickStart with Windows instructions
URL: https://github.com/apache/beam-site/pull/505#issuecomment-406720384
 
 
   @asfgit merge


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125640)
Time Spent: 10m
Remaining Estimate: 0h

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3386) Dependency conflict when Calcite is included in a project.

[Kai Jiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Jiang updated BEAM-3386:

Affects Version/s: 2.6.0

> Dependency conflict when Calcite is included in a project.
> --
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0
>Reporter: Austin Haas
>Priority: Critical
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and 
> the Beam SQL extension, then the following error is thrown when trying to run 
> Beam code.
> ClassCastException 
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot 
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl. 
> (CalciteConnectionImpl.java:120)
> 
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. 
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection 
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection 
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect 
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
>  (Frameworks.java:145)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
>  (Frameworks.java:106)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
>  (PlannerImpl.java:140)
> 
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
>  (PlannerImpl.java:170)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4807) Upgrade calcite to 1.17.0

[Kai Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551230#comment-16551230
 ] 

Kai Jiang commented on BEAM-4807:
-

I also want to see if BEAM-3386 is still a problem when we upgrade to 1.17.0

> Upgrade calcite to 1.17.0
> -
>
> Key: BEAM-4807
> URL: https://issues.apache.org/jira/browse/BEAM-4807
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> We should upgrade calcite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4807) Upgrade calcite to 1.17.0

[Kai Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551230#comment-16551230
 ] 

Kai Jiang edited comment on BEAM-4807 at 7/20/18 8:29 PM:
--

I also want to see if BEAM-3386 is still a problem after we upgrade to 1.17.0


was (Author: vectorijk):
I also want to see if BEAM-3386 is still a problem when we upgrade to 1.17.0

> Upgrade calcite to 1.17.0
> -
>
> Key: BEAM-4807
> URL: https://issues.apache.org/jira/browse/BEAM-4807
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> We should upgrade calcite.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=125638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125638
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 20/Jul/18 20:16
Start Date: 20/Jul/18 20:16
Worklog Time Spent: 10m 
  Work Description: ElliottBrossard commented on issue #5755: [BEAM-4417] 
Support BigQuery's NUMERIC type using Java
URL: https://github.com/apache/beam/pull/5755#issuecomment-406715160
 
 
   Yes, this should be fixed now. The current version of the Tornadoes test 
doesn't exercise NUMERIC, but the unit tests now use the proper Avro encoding 
(BYTES+DECIMAL) in correspondence with NUMERIC to verify that BigQueryIO is 
able to read those values.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125638)
Time Spent: 7h 20m  (was: 7h 10m)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.7.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3321) Update gax-grpc dependency to latest

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3321?focusedWorklogId=125637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125637
 ]

ASF GitHub Bot logged work on BEAM-3321:


Author: ASF GitHub Bot
Created on: 20/Jul/18 20:15
Start Date: 20/Jul/18 20:15
Worklog Time Spent: 10m 
  Work Description: pongad closed pull request #4707: [BEAM-3321] update 
google-cloud-spanner dependency
URL: https://github.com/apache/beam/pull/4707
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index 8e70a5bb274..e50c3202fe9 100644
--- a/build.gradle
+++ b/build.gradle
@@ -32,7 +32,7 @@ def bigtable_proto_version = "1.0.0-pre3"
 def google_clients_version = "1.22.0"
 def google_auth_version = "0.7.1"
 def grpc_version = "1.2.0"
-def protobuf_version = "3.2.0"
+def protobuf_version = "3.5.1"
 def guava_version = "20.0"
 def netty_version = "4.1.8.Final"
 def grpc_google_common_protos = "0.1.9"
@@ -78,7 +78,7 @@ ext.library = [
 error_prone_annotations: 
"com.google.errorprone:error_prone_annotations:2.0.15",
 findbugs_annotations: 
"com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1",
 findbugs_jsr305: "com.google.code.findbugs:jsr305:3.0.1",
-gax_grpc: "com.google.api:gax-grpc:0.20.0",
+gax: "com.google.api:gax:1.19.0",
 google_api_client: 
"com.google.api-client:google-api-client:$google_clients_version",
 google_api_client_jackson2: 
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
 google_api_client_java6: 
"com.google.api-client:google-api-client-java6:$google_clients_version",
@@ -91,10 +91,10 @@ ext.library = [
 google_api_services_storage: 
"com.google.apis:google-api-services-storage:v1-rev71-$google_clients_version",
 google_auth_library_credentials: 
"com.google.auth:google-auth-library-credentials:$google_auth_version",
 google_auth_library_oauth2_http: 
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-google_cloud_core: "com.google.cloud:google-cloud-core:1.0.2",
-google_cloud_core_grpc: 
"com.google.cloud:google-cloud-core-grpc:$grpc_version",
+google_cloud_core: "com.google.cloud:google-cloud-core:1.16.0",
+google_cloud_core_grpc: "com.google.cloud:google-cloud-core-grpc:1.16.0",
 google_cloud_dataflow_java_proto_library_all: 
"com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
-google_cloud_spanner: "com.google.cloud:google-cloud-spanner:0.20.0-beta",
+google_cloud_spanner: "com.google.cloud:google-cloud-spanner:0.34.0-beta",
 google_http_client: 
"com.google.http-client:google-http-client:$google_clients_version",
 google_http_client_jackson: 
"com.google.http-client:google-http-client-jackson:$google_clients_version",
 google_http_client_jackson2: 
"com.google.http-client:google-http-client-jackson2:$google_clients_version",
diff --git a/pom.xml b/pom.xml
index 7fe67cb6380..829564a5d2a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -125,7 +125,7 @@
 0.7.1
 1.22.0
 1.4.5
-1.0.2
+1.16.0
 
0.5.160304
 20.0
 1.2.0
@@ -145,10 +145,10 @@
 1.9.5
 4.1.8.Final
 1.1.33.Fork26
-3.2.0
+3.5.1
 v1-rev10-1.22.0
 1.7.25
-0.20.0-beta
+0.34.0-beta
 2.2.1
 4.3.5.RELEASE
 3.1.4
@@ -184,7 +184,7 @@
 -Werror
 
-Xpkginfo:always
 nothing
-0.20.0
+1.19.0
 
 
 kubectl
@@ -908,8 +908,8 @@
 
   
 com.google.api
-gax-grpc
-${gax-grpc.version}
+gax
+${gax.version}
   
 
   
@@ -1135,7 +1135,7 @@
   
 com.google.cloud
 google-cloud-core-grpc
-${grpc.version}
+${google-cloud-core.version}
   
   
 com.google.cloud.bigtable
diff --git a/sdks/java/io/google-cloud-platform/build.gradle 
b/sdks/java/io/google-cloud-platform/build.gradle
index 6c8a8980722..a64322dc7ea 100644
--- a/sdks/java/io/google-cloud-platform/build.gradle
+++ b/sdks/java/io/google-cloud-platform/build.gradle
@@ -38,7 +38,7 @@ dependencies {
   shadow library.java.jackson_databind
   shadow library.java.grpc_core
   shadow library.java.google_api_services_bigquery
-  shadow library.java.gax_grpc
+  shadow library.java.gax
   shadow library.java.google_cloud_core_grpc
   shadow library.java.google_api_services_pubsub
   shadow library.java.grpc_google_cloud_pubsub_v1
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 7aae03b90c7..cceeea25945 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -189,7 +189,7 @@

[jira] [Resolved] (BEAM-4769) Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4769.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable
> ---
>
> Key: BEAM-4769
> URL: https://issues.apache.org/jira/browse/BEAM-4769
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Dzmitry Lazerka
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Hi,
> I'm getting the following exception, probably when Beam cannot find a coder 
> for a wildcard generic type (although I don't think I use any, gotta find).
> {code:java}
> Exception in thread "main" java.lang.ClassCastException: 
> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.reflect.Types$WildcardTypeImpl
>  cannot be cast to java.lang.reflect.TypeVariable
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:625)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromParameterizedType(CoderRegistry.java:658)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:620)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromParameterizedType(CoderRegistry.java:658)
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:620)
>  at org.apache.beam.sdk.coders.CoderRegistry.getCoder(CoderRegistry.java:254)
>  at 
> org.apache.beam.sdk.values.PCollection.inferCoderOrFail(PCollection.java:149)
>  at 
> org.apache.beam.sdk.values.PCollection.finishSpecifyingOutput(PCollection.java:89)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy.setOutput(TransformHierarchy.java:208)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:538)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:472)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:286)
>  at org.apache.beam.sdk.transforms.ParDo$SingleOutput.expand(ParDo.java:649)
>  at org.apache.beam.sdk.transforms.ParDo$SingleOutput.expand(ParDo.java:584)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
>  at org.apache.beam.sdk.transforms.MapElements.expand(MapElements.java:123)
>  at org.apache.beam.sdk.transforms.MapElements.expand(MapElements.java:36)
>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:491)
>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:299)
> {code} 
> The error seems to be in the following lines of CoderRegistry.java @ 629:
> {code:java|title=CoderRegistry.java|borderStyle=solid}
> } else if (type instanceof TypeVariable) {
>      coder = getCoderFromFactories(typeDescriptor, Collections.emptyList());
> } else if (type instanceof WildcardType) {
>      // No coder for an unknown generic type.
>      throw new CannotProvideCoderException(
>      String.format("Cannot provide a coder for type variable %s"
>      + " (declared by %s) because the actual type is unknown due to 
> erasure.",
>      type,
>      ((TypeVariable) type).getGenericDeclaration()),
>      ReasonCode.TYPE_ERASURE);
> {code}
> It tries to cast a WildcardType to TypeVariable, although that should've 
> been caught above on line 621.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4769) Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4769?focusedWorklogId=125626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125626
 ]

ASF GitHub Bot logged work on BEAM-4769:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:51
Start Date: 20/Jul/18 18:51
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5967: [BEAM-4769] Fix 
issue when performing coder inference for wildcard types.
URL: https://github.com/apache/beam/pull/5967
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
index 167ff84f01c..865b2488608 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java
@@ -621,11 +621,7 @@ private static boolean isNullOrEmpty(Collection c) {
 } else if (type instanceof WildcardType) {
   // No coder for an unknown generic type.
   throw new CannotProvideCoderException(
-  String.format(
-  "Cannot provide a coder for type variable %s"
-  + " (declared by %s) because the actual type is unknown due 
to erasure.",
-  type, ((TypeVariable) type).getGenericDeclaration()),
-  ReasonCode.TYPE_ERASURE);
+  String.format("Cannot provide a coder for wildcard type %s.", type), 
ReasonCode.UNKNOWN);
 } else {
   throw new RuntimeException("Internal error: unexpected kind of Type: " + 
type);
 }
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
index de561dd1779..4b21fd29fbf 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CoderRegistryTest.java
@@ -27,6 +27,7 @@
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.io.Serializable;
+import java.lang.reflect.ParameterizedType;
 import java.lang.reflect.Type;
 import java.util.Collections;
 import java.util.List;
@@ -149,6 +150,21 @@ public void testParameterizedDefaultCoderUnknown() throws 
Exception {
 registry.getCoder(listUnknownToken);
   }
 
+  @Test
+  public void testParameterizedWildcardTypeIsUnknown() throws Exception {
+CoderRegistry registry = CoderRegistry.createDefault();
+TypeDescriptor> wildcardUnknownToken =
+new TypeDescriptor>() {};
+
+thrown.expect(CannotProvideCoderException.class);
+thrown.expectMessage(
+String.format(
+"Cannot provide coder for parameterized type %s: Cannot provide a 
coder for wildcard type %s.",
+wildcardUnknownToken,
+((ParameterizedType) 
wildcardUnknownToken.getType()).getActualTypeArguments()[0]));
+registry.getCoder(wildcardUnknownToken);
+  }
+
   @Test
   public void testTypeParameterInferenceForward() throws Exception {
 CoderRegistry registry = CoderRegistry.createDefault();


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125626)
Time Spent: 0.5h  (was: 20m)

> Types$WildcardTypeImpl cannot be cast to java.lang.reflect.TypeVariable
> ---
>
> Key: BEAM-4769
> URL: https://issues.apache.org/jira/browse/BEAM-4769
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Dzmitry Lazerka
>Assignee: Luke Cwik
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Hi,
> I'm getting the following exception, probably when Beam cannot find a coder 
> for a wildcard generic type (although I don't think I use any, gotta find).
> {code:java}
> Exception in thread "main" java.lang.ClassCastException: 
> org.apache.beam.repackaged.beam_sdks_java_core.com.google.common.reflect.Types$WildcardTypeImpl
>  cannot be cast to java.lang.reflect.TypeVariable
>  at 
> org.apache.beam.sdk.coders.CoderRegistry.getCoderFromTypeDescriptor(CoderRegistry.java:625)
>  at 
> 

[beam] branch master updated (df16fcd -> e4c7e11)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from df16fcd  Add automation script for verifying release build (#5973)
 add 481956a  [BEAM-4769] Fix issue when performing coder inference for 
wildcard types.
 new e4c7e11  [BEAM-4769] Fix issue when performing coder inference for 
wildcard types.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/coders/CoderRegistry.java   |  6 +-
 .../org/apache/beam/sdk/coders/CoderRegistryTest.java| 16 
 2 files changed, 17 insertions(+), 5 deletions(-)

[beam] 01/01: [BEAM-4769] Fix issue when performing coder inference for wildcard types.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e4c7e11e5fb5bc43ffbaf1463ea06b22b6119815
Merge: df16fcd 481956a
Author: Lukasz Cwik 
AuthorDate: Fri Jul 20 11:51:48 2018 -0700

[BEAM-4769] Fix issue when performing coder inference for wildcard types.

 .../java/org/apache/beam/sdk/coders/CoderRegistry.java   |  6 +-
 .../org/apache/beam/sdk/coders/CoderRegistryTest.java| 16 
 2 files changed, 17 insertions(+), 5 deletions(-)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125624
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:50
Start Date: 20/Jul/18 18:50
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204138278
 
 

 ##
 File path: devtools/JenkinsOnDocker/fetchplugins.sh
 ##
 @@ -0,0 +1,12 @@
+#!/bin/bash
+curl https://cwiki.apache.org/confluence/display/INFRA/Jenkins+Plugin+Upgrades 
> plugins.txt
+sed -i ':a;N;$!ba;s/\n/ /g' plugins.txt
+grep -Po -m 1 'https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125623
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:50
Start Date: 20/Jul/18 18:50
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204138266
 
 

 ##
 File path: devtools/JenkinsOnDocker/fetchplugins.sh
 ##
 @@ -0,0 +1,12 @@
+#!/bin/bash
+curl https://cwiki.apache.org/confluence/display/INFRA/Jenkins+Plugin+Upgrades 
> plugins.txt
+sed -i ':a;N;$!ba;s/\n/ /g' plugins.txt
 
 Review comment:
   done(removed)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125623)
Time Spent: 1h  (was: 50m)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125622
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:49
Start Date: 20/Jul/18 18:49
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204138211
 
 

 ##
 File path: devtools/JenkinsOnDocker/README.md
 ##
 @@ -0,0 +1,142 @@
+# Local Jenkins Setup
 
 Review comment:
   done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125622)
Time Spent: 50m  (was: 40m)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125618
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:45
Start Date: 20/Jul/18 18:45
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204137036
 
 

 ##
 File path: devtools/JenkinsOnDocker/fetchplugins.sh
 ##
 @@ -0,0 +1,12 @@
+#!/bin/bash
 
 Review comment:
   done. (removed)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125618)
Time Spent: 40m  (was: 0.5h)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125615
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:43
Start Date: 20/Jul/18 18:43
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204136430
 
 

 ##
 File path: devtools/JenkinsOnDocker/README.md
 ##
 @@ -0,0 +1,142 @@
+# Local Jenkins Setup
+
+Setting up a local Jenkins instance is useful for testing Jenkins job changes
+without affecting the production instance. Using Docker makes the setup very
+straightforward.
+
+## Jenkins on Docker
+
+### Requirements
+If you haven't yet, [install Docker on your 
machine.](https://docs.docker.com/install/)
+
+
+### Setup using provided scripts
+
+**WARNING: Not for production use.**
+
+*INFO: Changing admin credentials is advised.*
+
+You can utilize scripts in this folder to build Docker container with Jenkins,
+pre-installed plugin and some basic configuration.
+
+```bash
+fetchplugins.sh
+docker build -t beamjenkins .
+docker run -p 127.0.0.1:8080:8080 beamjenkins:latest
+```
+* fetchplugin.sh -- fetches list of plugins from
+  [infra wiki 
list](https://cwiki.apache.org/confluence/display/INFRA/Jenkins+Plugin+Upgrades).
+* docker build -- builds image with name beamjenkins based on dockerfile 
located
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125615)
Time Spent: 10m
Remaining Estimate: 0h

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125616=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125616
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:43
Start Date: 20/Jul/18 18:43
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204136441
 
 

 ##
 File path: devtools/JenkinsOnDocker/dockerfile
 ##
 @@ -0,0 +1,22 @@
+FROM jenkins/jenkins:lts
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125616)
Time Spent: 20m  (was: 10m)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4838) Prepare instructions for running Jenkins locally.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4838?focusedWorklogId=125617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125617
 ]

ASF GitHub Bot logged work on BEAM-4838:


Author: ASF GitHub Bot
Created on: 20/Jul/18 18:43
Start Date: 20/Jul/18 18:43
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #5989: 
[BEAM-4838] Add dockerfile for standalone Jenkins. Plugins included.
URL: https://github.com/apache/beam/pull/5989#discussion_r204136454
 
 

 ##
 File path: devtools/JenkinsOnDocker/dockerfile
 ##
 @@ -0,0 +1,22 @@
+FROM jenkins/jenkins:lts
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125617)
Time Spent: 0.5h  (was: 20m)

> Prepare instructions for running Jenkins locally.
> -
>
> Key: BEAM-4838
> URL: https://issues.apache.org/jira/browse/BEAM-4838
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We need a way to test Jenkins Job DSL scripts without breaking our build 
> server. One of the approaches available is to spin up Jenkins on your local 
> machine with all plugins available.
> This has already been used by some developers in their workflow.
> We want to share instruction on spinning up local Jenkins with other 
> developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4838) Prepare instructions for running Jenkins locally.

[Mikhail Gryzykhin (JIRA)]

Mikhail Gryzykhin created BEAM-4838:
---

 Summary: Prepare instructions for running Jenkins locally.
 Key: BEAM-4838
 URL: https://issues.apache.org/jira/browse/BEAM-4838
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Mikhail Gryzykhin
Assignee: Mikhail Gryzykhin


We need a way to test Jenkins Job DSL scripts without breaking our build 
server. One of the approaches available is to spin up Jenkins on your local 
machine with all plugins available.

This has already been used by some developers in their workflow.

We want to share instruction on spinning up local Jenkins with other developers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #756

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins897186046553976097.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6358351207876256461.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1054471722323745681.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5974130140951832206.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2790163277059841195.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins641354282215659812.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #769

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3426455608173996771.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3635452461830186560.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3479673260620300125.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins727885114527424.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8123825965671907897.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4353269326431699045.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already satisfied: PyYAML==3.12 in 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #262

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins895829130089940.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins487508618068391255.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5040789759250009969.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5525125802202552465.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins8949166698037697135.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins3260979616432805537.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT_HDFS #157

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3133458409106769918.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins605118801020533718.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8446940957345549130.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-parquetioit-hdfs-157
namespace "beam-performancetests-parquetioit-hdfs-157" created
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins901810286464564774.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-parquetioit-hdfs-157
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1328271240868627241.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1959624728979366569.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5034163512419450561.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7974342853976728652.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 

Build failed in Jenkins: beam_PreCommit_Java_Cron #127

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
[...truncated 16.66 MB...]
INFO: 2018-07-20T18:18:30.650Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.694Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.738Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.778Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.822Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.875Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.926Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:30.967Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:31.007Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:31.052Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:31.087Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Jul 20, 2018 6:18:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T18:18:31.137Z: Unzipping flatten s13-u58 for input 
s14.org.apache.beam.sdk.values.PCollection.:364#f0cbc4d341b04049-c56
Jul 20, 2018 6:18:32 PM 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #790

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam13 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1271839718911251162.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins642382732311365.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins678550873778830166.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4883097783182132296.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5109903818165288745.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2711900341666834520.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already satisfied: PyYAML==3.12 in 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #762

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins914396980318609384.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins864810086107772971.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1355724674091012293.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7037801332330674259.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins9144418372833376648.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins3109615618280163807.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #439

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4089857844854999530.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6419626862791119028.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1245396790949475449.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-compressed-textioit-hdfs-439
namespace "beam-performancetests-compressed-textioit-hdfs-439" created
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5084502137724857212.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-compressed-textioit-hdfs-439
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5972919961173544119.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7895378163240176385.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6917507615070254856.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5405102912342676916.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #438

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins425566485228207987.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6680636057436967139.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5272688382935138080.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-xmlioit-hdfs-438
namespace "beam-performancetests-xmlioit-hdfs-438" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2996862629122989541.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-xmlioit-hdfs-438
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8880930077561623040.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7945145642798220115.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3883233722844379071.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins485514157782174373.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #449

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1155844803706293852.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1531848244992202105.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7438813510165437332.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-textioit-hdfs-449
namespace "beam-performancetests-textioit-hdfs-449" created
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1911985947641214210.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-textioit-hdfs-449
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3519566466416522740.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5511625355003936202.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3109656818716312466.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1360118900296906080.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have 

[jira] [Resolved] (BEAM-3482) Java serialiazation exception when using BigQueryIO

[Pablo Estrada (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada resolved BEAM-3482.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Java serialiazation exception when using BigQueryIO
> ---
>
> Key: BEAM-3482
> URL: https://issues.apache.org/jira/browse/BEAM-3482
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.3.0
>Reporter: Shashank Prabhakara
>Assignee: Chamikara Jayalath
>Priority: Major
>  Labels: easyfix, newbie
> Fix For: 2.5.0
>
> Attachments: m.patch
>
>
> When writing data to BQ using BigQueryIO, we get the following exception when 
> checkpointing. To reproduce, use BigQueryIO.writeTableRows() using 
> sufficiently large dataset with latest code from master branch. Attached 
> patch to ticket.
>  The following stack trace is when using using ApexRunner:
> ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set 
> [OperatorDeployInfo[id=20,name=BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)/ParMultiDo(Anonymous),type=GENERIC,checkpoint=\{,
>  0, 0},inputs=[OperatorDeployInfo.Inpu
>  
> tDeployInfo[portName=input,streamId=stream9,sourceNodeId=19,sourcePortName=output,locality=CONTAINER_LOCAL,partitionMask=0,partitionKeys=]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream56,bufferServer=
>  stopped running due to
>  an exception.
>  com.esotericsoftware.kryo.KryoException: Error during Java serialization.
>  Serialization trace:
>  doFn (org.apache.beam.runners.apex.translation.operators.ApexParDoOperator)
>      at 
> com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:33)
>      at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>      at 
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
>      at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
>      at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
>      at 
> com.datatorrent.common.util.FSStorageAgent.store(FSStorageAgent.java:190)
>      at 
> com.datatorrent.common.util.AsyncFSStorageAgent.save(AsyncFSStorageAgent.java:101)
>      at com.datatorrent.stram.engine.Node.checkpoint(Node.java:521)
>      at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:461)
>      at 
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
>  Caused by: java.io.NotSerializableException: 
> org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations$SideInputAccessorViaProcessContext
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
>      at 
> com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:30)
>      ... 9 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3482) Java serialiazation exception when using BigQueryIO

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551092#comment-16551092
 ] 

Pablo Estrada commented on BEAM-3482:
-

Seems like this is fixed. Is it?

> Java serialiazation exception when using BigQueryIO
> ---
>
> Key: BEAM-3482
> URL: https://issues.apache.org/jira/browse/BEAM-3482
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.3.0
>Reporter: Shashank Prabhakara
>Assignee: Chamikara Jayalath
>Priority: Major
>  Labels: easyfix, newbie
> Fix For: 2.5.0
>
> Attachments: m.patch
>
>
> When writing data to BQ using BigQueryIO, we get the following exception when 
> checkpointing. To reproduce, use BigQueryIO.writeTableRows() using 
> sufficiently large dataset with latest code from master branch. Attached 
> patch to ticket.
>  The following stack trace is when using using ApexRunner:
> ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set 
> [OperatorDeployInfo[id=20,name=BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)/ParMultiDo(Anonymous),type=GENERIC,checkpoint=\{,
>  0, 0},inputs=[OperatorDeployInfo.Inpu
>  
> tDeployInfo[portName=input,streamId=stream9,sourceNodeId=19,sourcePortName=output,locality=CONTAINER_LOCAL,partitionMask=0,partitionKeys=]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream56,bufferServer=
>  stopped running due to
>  an exception.
>  com.esotericsoftware.kryo.KryoException: Error during Java serialization.
>  Serialization trace:
>  doFn (org.apache.beam.runners.apex.translation.operators.ApexParDoOperator)
>      at 
> com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:33)
>      at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523)
>      at 
> com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61)
>      at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495)
>      at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599)
>      at 
> com.datatorrent.common.util.FSStorageAgent.store(FSStorageAgent.java:190)
>      at 
> com.datatorrent.common.util.AsyncFSStorageAgent.save(AsyncFSStorageAgent.java:101)
>      at com.datatorrent.stram.engine.Node.checkpoint(Node.java:521)
>      at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:461)
>      at 
> com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428)
>  Caused by: java.io.NotSerializableException: 
> org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations$SideInputAccessorViaProcessContext
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>      at 
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>      at 
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>      at 
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>      at 
> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
>      at 
> com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:30)
>      ... 9 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #539

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1899549359115680522.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins2085427973927767758.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1938318491931795105.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-hadoopinputformat-539
namespace "beam-performancetests-hadoopinputformat-539" created
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins8487137472468906555.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-hadoopinputformat-539
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins2507483932445701580.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins627692465442533513.sh
+ rm -rf .env
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1367500277086332087.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins9063804501194264059.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.

Build failed in Jenkins: beam_PerformanceTests_Spark #1977

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3065246188662805805.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6701142626838387451.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins4176053764439608332.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins204843741089038177.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6715557444724031288.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7295933271535762166.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 

[jira] [Commented] (BEAM-4803) Beam spark runner not working properly with kafka

[Vivek Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551079#comment-16551079
 ] 

Vivek Agarwal commented on BEAM-4803:
-

[~jbonofre] yes spark runners for streaming need to create microbatches over 
unbounded data source, but shouldn't the readers(kafka consumers in this case) 
be cached? Looking at SourceDStream.java(in beam spark-runner code), I could 
find 
{code:java}
// Reader cache expiration interval. 50% of batch interval is added to 
accommodate latency.
this.readerCacheInterval = 1.5 * sparkOptions.getBatchIntervalMillis();{code}
which is used to specify the time for which reader objects will be cached when 
creating microbatches. Since it is just using batchInterval, we can't really 
increase it without converting streaming to batch processing. I think there 
should be a way to separately specify readerCacheInterval, no? as clearly 
repeated kerberos authentication is a no go.

In kafka for spark streaming 
[documentation|https://spark.apache.org/docs/2.2.0/streaming-kafka-0-10-integration.html#locationstrategies]
 its mentioned that default behaviour is to cache consumers rather than 
creating for each batch. Please comment, thanks.

> Beam spark runner not working properly with kafka
> -
>
> Key: BEAM-4803
> URL: https://issues.apache.org/jira/browse/BEAM-4803
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka, runner-spark
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Vivek Agarwal
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>
> We are running a beam stream processing job on a spark runner, which reads 
> from a kafka topic using kerberos authentication. We are using java-io-kafka 
> v2.4.0 to read from kafka topic in the pipeline. The issue is that the 
> kafkaIO client is continuously creating a new kafka consumer with specified 
> config, doing kerberos login every time. Also, there are spark streaming jobs 
> which get spawned for the unbounded source, every second or so even when 
> there is no data in the kafka topic. Log has these jobs-
> INFO SparkContext: Starting job: dstr...@sparkunboumdedsource.java:172
> We can see in the logs
> INFO MicrobatchSource: No cached reader found for split: 
> [org.apache.beam.sdk.io.kafka.KafkaUnboundedSource@2919a728]. Creating new 
> reader at checkpoint mark...
> And then it creates new consumer doing fresh kerberos login, which is 
> creating issues.
> We are unsure of what should be correct behavior here and why so many spark 
> streaming jobs are getting created. We tried the beam code with flink runner 
> and did not find this issue there. Can someone point to the correct settings 
> for using unbounded kafka source with spark runner using beam? 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4788) Test Auto JIRA subtask456

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-4788:

Description: updated couple feilds of this issue by using jira api  (was: 
No actions required)
Component/s: testing
Summary: Test Auto JIRA subtask456  (was: Test Auto JIRA Subtask 123)

> Test Auto JIRA subtask456
> -
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> updated couple feilds of this issue by using jira api



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #533

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins8421403386530226484.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins1628755361812089650.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins2196140764774670479.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins2401932435036822850.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, but you'll 
have oauth2client 4.1.2 which is incompatible.
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins4486513396571826587.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins4515428057530958030.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.2.2)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.2.0)
Requirement already satisfied: PyYAML==3.12 in 

Build failed in Jenkins: beam_PerformanceTests_JDBC #870

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins534798288352524544.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8753179807497947556.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6017659103157741801.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-jdbc-870
namespace "beam-performancetests-jdbc-870" created
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7151927186763497176.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-jdbc-870
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8021643134618639634.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8937423270417032906.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4134736527395263131.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5753669424455757853.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but you'll have 
httplib2 0.11.3 which is incompatible.
googledatastore 7.0.1 has requirement 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #438

[Apache Jenkins Server]

See 


Changes:

[garrettjonesgoogle] Bumping google cloud versions

[supercclank] Set BatchPath on google.api.services.storage.Storage Storage 
Builder

[supercclank] Change to using java.nio.file.Paths over string concatenation for 
batch

[katarzyna.kucharczyk] Changed order of installing perfkit requirements and 
beam requirements

[thw] Remove dead link from README.

[aaltay] Add automation script for verifying release build (#5973)

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision df16fcddd9fe64f77d5bdaab1f1a732132645486 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f df16fcddd9fe64f77d5bdaab1f1a732132645486
Commit message: "Add automation script for verifying release build (#5973)"
 > git rev-list --no-walk 2834b2ddc5a7cdc553adf0cafb1e7850181e0e68 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3587539233426927642.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8725924838504492009.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5308813761317940458.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-avroioit-hdfs-438
namespace "beam-performancetests-avroioit-hdfs-438" created
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8372396431506245692.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-avroioit-hdfs-438
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7447308050732493660.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1718917902719660794.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4847823594931228882.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6414608388863118138.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-datastore-v1 0.90.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
incompatible.
proto-google-cloud-pubsub-v1 0.15.4 has requirement 
oauth2client<4.0dev,>=2.0.0, but you'll have 

[beam] branch master updated: Add automation script for verifying release build (#5973)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new df16fcd  Add automation script for verifying release build (#5973)
df16fcd is described below

commit df16fcddd9fe64f77d5bdaab1f1a732132645486
Author: Boyuan Zhang <36090911+boyua...@users.noreply.github.com>
AuthorDate: Fri Jul 20 10:55:14 2018 -0700

Add automation script for verifying release build (#5973)

* Add automation script for verifying release build

* Address Ahmet's comments
---
 release/src/main/scripts/verify_release_build.sh | 127 +++
 1 file changed, 127 insertions(+)

diff --git a/release/src/main/scripts/verify_release_build.sh 
b/release/src/main/scripts/verify_release_build.sh
new file mode 100755
index 000..dd0ab74
--- /dev/null
+++ b/release/src/main/scripts/verify_release_build.sh
@@ -0,0 +1,127 @@
+#!/bin/bash
+#
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#See the License for the specific language governing permissions and
+#limitations under the License.
+#
+
+# This script will run pre-installations and run release build.
+
+set -e
+
+GIT_REPO_URL=https://github.com/apache/beam.git
+LOCAL_CLONE_DIR=release_build
+BEAM_ROOT_DIR=beam
+
+echo "Which branch you want to verify release build: "
+read branch
+
+echo "=Environment Variables="
+echo "working branch: ${branch}"
+echo "local repo dir: ~/${LOCAL_CLONE_DIR}/${BEAM_ROOT_DIR}"
+
+echo "Checking Requirement==="
+
+echo "Checking pip==="
+if [[ -z `which pip` ]]; then
+  echo "There is no pip installed on your machine."
+  echo "Would you like to install pip with root permission? [y|N]"
+  read confirmation
+  if [[ $confirmation != "y" ]]; then
+echo "Refused to install pip on your machine. Exit."
+exit
+  else
+echo "==Installing pip="
+curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
+sudo python get-pip.py
+rm get-pip.py
+  fi
+else
+  pip --version
+fi
+
+echo "Checking virtualenv"
+if [[ -z `which virtualenv` ]]; then
+  echo "There is no virtualenv installed on your machine."
+  echo "Would you like to install virtualenv with root perrission? [y|N]"
+  read confirmation
+  if [[ $confirmation != "y" ]]; then
+echo "Refused to install virtualenv on your machine. Exit."
+exit
+  else
+echo "==Installing virtualenv==="
+sudo `which pip` install --upgrade virtualenv
+  fi
+else
+  virtualenv --version
+fi
+
+echo "=Checking cython==="
+if [[ -z `which cython` ]]; then
+  echo "There is no cython installed on your machine."
+  echo "Would you like to install cython with root permission? [y|N]"
+  read confirmation
+  if [[ $confirmation != "y" ]]; then
+echo "Refused to install cython on your machine. Exit."
+exit
+  else
+echo "==Installing cython==="
+sudo `which pip` install cython
+sudo apt-get install gcc
+sudo apt-get install python-dev
+  fi
+else
+  cython --version
+fi
+
+echo "==Checking /usr/bin/time"
+if [[ `which time` != "/usr/bin/time" ]]; then
+  echo "There is no usr/bin/time installed on your machine."
+  echo "Would you like to install time on your machine with root permission? 
[y|N]"
+  read confirmation
+  if [[ $confirmation != "y" ]]; then
+echo "Refused to install time on your machine. Exit."
+exit
+  else
+echo "==Installing time="
+sudo apt-get install time
+alias time='/usr/bin/time'
+  fi
+else
+  which time
+fi
+
+echo "==Starting Clone Repo=="
+mkdir ${LOCAL_CLONE_DIR}
+cd  ${LOCAL_CLONE_DIR}
+git clone ${GIT_REPO_URL}
+cd ${BEAM_ROOT_DIR}
+git checkout ${branch}
+echo "==="
+
+echo "==Starting Release Build==="
+git 

[jira] [Resolved] (BEAM-4795) Support Spanner library 0.53

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4795.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Support Spanner library 0.53
> 
>
> Key: BEAM-4795
> URL: https://issues.apache.org/jira/browse/BEAM-4795
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Niel Markwick
>Assignee: Garrett Jones
>Priority: Minor
> Fix For: 2.7.0
>
>
> Beam 2.5 uses Spanner library 0.20
> Spanner (at time of writing) is up to version 0.53 with many additional 
> features... (eg commit timestamps).
> It is impossible to use this spanner library version due to Beam using 
> functions which have been removed between 0.20 and 0.53... 
> Please consider updating the spanner (and other GCP) library versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4788) Test Auto JIRA Subtask 123

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4788.
---
Resolution: Fixed

> Test Auto JIRA Subtask 123
> --
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> No actions required



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3321) Update gax-grpc dependency to latest

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3321.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Update gax-grpc dependency to latest
> 
>
> Key: BEAM-3321
> URL: https://issues.apache.org/jira/browse/BEAM-3321
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Beam uses 0.20 latest release is 1.15: 
> https://mvnrepository.com/artifact/com.google.api/gax-grpc
> This results in compatibility issues when Beam and Datastore client are used 
> together: 
> https://lists.apache.org/thread.html/8a99f4c0dbb1138f1067ef73b2ba7f44f035b309af41b1876ffc1ff2@%3Cuser.beam.apache.org%3E
> Assigning to Mairbek who added this dependency recently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3098) Upgrade Java grpc version

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3098.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3321) Update gax-grpc dependency to latest

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3321?focusedWorklogId=125590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125590
 ]

ASF GitHub Bot logged work on BEAM-3321:


Author: ASF GitHub Bot
Created on: 20/Jul/18 17:48
Start Date: 20/Jul/18 17:48
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5988: [BEAM-3321, 
BEAM-3098, BEAM-4795] Bumping google cloud versions
URL: https://github.com/apache/beam/pull/5988
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 1461ace929f..23e96a27e28 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -296,23 +296,25 @@ class BeamModulePlugin implements Plugin {
 def bigtable_version = "1.0.0"
 def bigtable_proto_version = "1.0.0-pre3"
 def google_clients_version = "1.23.0"
-def google_auth_version = "0.7.1"
-def grpc_version = "1.2.0"
-def protobuf_version = "3.2.0"
+def google_auth_version = "0.10.0"
+def grpc_version = "1.13.1"
+def protobuf_version = "3.6.0"
 def guava_version = "20.0"
-def netty_version = "4.1.8.Final"
-def grpc_google_common_protos = "0.1.9"
+def netty_version = "4.1.25.Final"
+def proto_google_common_protos = "1.12.0"
 def hamcrest_version = "1.3"
 def hadoop_version = "2.7.3"
 def jackson_version = "2.9.5"
 def spark_version = "2.3.1"
-def pubsub_grpc_version = "0.1.18"
+def spanner_grpc_version = "0.19.0"
+def pubsub_grpc_version = "1.18.0"
 def apex_core_version = "3.7.0"
 def apex_malhar_version = "3.4.0"
 def postgres_version = "42.2.2"
 def jaxb_api_version = "2.2.12"
 def kafka_version = "1.0.0"
 def quickcheck_version = "0.8"
+def google_cloud_core_version = "1.36.0"
 
 // A map of maps containing common libraries used per language. To use:
 // dependencies {
@@ -342,14 +344,14 @@ class BeamModulePlugin implements Plugin {
 commons_io_2x   : 
"commons-io:commons-io:2.5",
 commons_lang3   : 
"org.apache.commons:commons-lang3:3.6",
 commons_math3   : 
"org.apache.commons:commons-math3:3.6.1",
-datastore_v1_proto_client   : 
"com.google.cloud.datastore:datastore-v1-proto-client:1.4.0",
-datastore_v1_protos : 
"com.google.cloud.datastore:datastore-v1-protos:1.3.0",
+datastore_v1_proto_client   : 
"com.google.cloud.datastore:datastore-v1-proto-client:1.6.0",
+datastore_v1_protos : 
"com.google.cloud.datastore:datastore-v1-protos:1.5.0",
 error_prone_annotations : 
"com.google.errorprone:error_prone_annotations:2.0.15",
-gax_grpc: 
"com.google.api:gax-grpc:0.20.0",
+gax_grpc: 
"com.google.api:gax-grpc:1.29.0",
 google_api_client   : 
"com.google.api-client:google-api-client:$google_clients_version",
 google_api_client_jackson2  : 
"com.google.api-client:google-api-client-jackson2:$google_clients_version",
 google_api_client_java6 : 
"com.google.api-client:google-api-client-java6:$google_clients_version",
-google_api_common   : 
"com.google.api:api-common:1.0.0-rc2",
+google_api_common   : 
"com.google.api:api-common:1.6.0",
 google_api_services_bigquery: 
"com.google.apis:google-api-services-bigquery:v2-rev374-$google_clients_version",
 google_api_services_clouddebugger   : 
"com.google.apis:google-api-services-clouddebugger:v2-rev233-$google_clients_version",
 google_api_services_cloudresourcemanager: 
"com.google.apis:google-api-services-cloudresourcemanager:v1-rev477-$google_clients_version",
@@ -358,10 +360,10 @@ class BeamModulePlugin implements Plugin {
 google_api_services_storage : 
"com.google.apis:google-api-services-storage:v1-rev124-$google_clients_version",
 google_auth_library_credentials : 
"com.google.auth:google-auth-library-credentials:$google_auth_version",
 google_auth_library_oauth2_http : 
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
-

[beam] 01/01: [BEAM-3321, BEAM-3098, BEAM-4795] Bumping google cloud versions

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 440c40a4a04bcf7c56d978dab1aab22b7b780bec
Merge: 996dc4d e0110c4
Author: Lukasz Cwik 
AuthorDate: Fri Jul 20 10:48:51 2018 -0700

[BEAM-3321, BEAM-3098, BEAM-4795] Bumping google cloud versions

 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 68 +++---
 .../beam/sdk/io/gcp/spanner/SpannerConfig.java |  5 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  6 +-
 .../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java  | 22 ---
 .../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 12 ++--
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 22 ---
 6 files changed, 75 insertions(+), 60 deletions(-)

[beam] branch master updated (996dc4d -> 440c40a)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 996dc4d  Remove dead link from README.
 add e0110c4  Bumping google cloud versions
 new 440c40a  [BEAM-3321, BEAM-3098, BEAM-4795] Bumping google cloud 
versions

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 68 +++---
 .../beam/sdk/io/gcp/spanner/SpannerConfig.java |  5 +-
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  6 +-
 .../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java  | 22 ---
 .../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 12 ++--
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 22 ---
 6 files changed, 75 insertions(+), 60 deletions(-)