[jira] [Work logged] (BEAM-4835) Add more flexible options for data loading to BigQueryIO.Write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4835?focusedWorklogId=125350=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125350
 ]

ASF GitHub Bot logged work on BEAM-4835:


Author: ASF GitHub Bot
Created on: 20/Jul/18 05:33
Start Date: 20/Jul/18 05:33
Worklog Time Spent: 10m 
  Work Description: gene-telligent opened a new pull request #6008: 
[BEAM-4835] Adding in additional options for BigQueryIO insert statements
URL: https://github.com/apache/beam/pull/6008
 
 
   Adds in two flags for BigQueryIO.Write, skipInvalidRows and 
ignoreUnknownValues. These are passed through to the HTTP API. More details in 
the [corresponding JIRA 
ticket.](https://issues.apache.org/jira/browse/BEAM-4835)
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125350)
Time Spent: 10m
Remaining Estimate: 0h

> Add more flexible options for data loading to BigQueryIO.Write
> --
>
> Key: BEAM-4835
> URL: https://issues.apache.org/jira/browse/BEAM-4835
> Project: Beam
>  Issue Type: Improvement
>  Components: 

[jira] [Created] (BEAM-4835) Add more flexible options for data loading to BigQueryIO.Write

[Gene Peters (JIRA)]

Gene Peters created BEAM-4835:
-

 Summary: Add more flexible options for data loading to 
BigQueryIO.Write
 Key: BEAM-4835
 URL: https://issues.apache.org/jira/browse/BEAM-4835
 Project: Beam
  Issue Type: Improvement
  Components: io-java-gcp
Reporter: Gene Peters
Assignee: Chamikara Jayalath


As part of the BigQuery API, there are a few options exposed to end-users which 
allow for more flexible data loading.

For both 
[streaming|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setIgnoreUnknownValues-java.lang.Boolean-]
 and 
[batch|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/JobConfigurationLoad.html#setIgnoreUnknownValues-java.lang.Boolean-]
 inserts, the flag "ignoreUnknownValues" can be set, which indicates if 
BigQuery should accept rows that contain values that do not match the schema. 

[In 
addition,|https://developers.google.com/resources/api-libraries/documentation/bigquery/v2/java/latest/com/google/api/services/bigquery/model/TableDataInsertAllRequest.html#setSkipInvalidRows-java.lang.Boolean-]
 streaming inserts allow for the option of accepting an inserted batch of rows 
even if some of of the rows are invalid.

I've made the necessary code changes to make this available within 
BigQueryIO.Write and will be attaching the pull request to this ticket for 
review. Both flags are off by default.

Let me know if you have any questions or feedback about this!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125348
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 05:02
Start Date: 20/Jul/18 05:02
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6006: [BEAM-4808][SQL] add 
e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406489731
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125348)
Time Spent: 1h 50m  (was: 1h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125346=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125346
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 04:59
Start Date: 20/Jul/18 04:59
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406489289
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125346)
Time Spent: 1h 40m  (was: 1.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125344
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 04:56
Start Date: 20/Jul/18 04:56
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406473976
 
 
   Run java preCommit
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125344)
Time Spent: 1h 20m  (was: 1h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125345
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 04:56
Start Date: 20/Jul/18 04:56
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406488727
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125345)
Time Spent: 1.5h  (was: 1h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125343
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 04:54
Start Date: 20/Jul/18 04:54
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406488727
 
 
   run java postcommit
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125343)
Time Spent: 1h 10m  (was: 1h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125332
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 03:42
Start Date: 20/Jul/18 03:42
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6007: [BEAM-4834] Identify 
loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007#issuecomment-406480454
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125332)
Time Spent: 40m  (was: 0.5h)

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Cyclic graphs will never finish the topological sort so we should check the 
> cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125325
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 02:54
Start Date: 20/Jul/18 02:54
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406473976
 
 
   Run java preCommit
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125325)
Time Spent: 1h  (was: 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #450

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins3784546003534278490.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins6861100187432251710.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins5310037475365633265.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-mongodbio-it-450
namespace "beam-performancetests-mongodbio-it-450" created
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7247491461284273550.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-mongodbio-it-450
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins9221980604691504906.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins5893775144763198231.sh
+ rm -rf .env
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins1525343043285574781.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins1185574331265587084.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7394312006113243325.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins1444370549478064374.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125301
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:43
Start Date: 20/Jul/18 00:43
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6006: [BEAM-4808][SQL] add 
e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406455419
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125301)
Time Spent: 50m  (was: 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4832) Concurrent Writes in Data channels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4832?focusedWorklogId=125297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125297
 ]

ASF GitHub Bot logged work on BEAM-4832:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:39
Start Date: 20/Jul/18 00:39
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6003: [BEAM-4832] Fix 
concurrent race in datamgr.go
URL: https://github.com/apache/beam/pull/6003#issuecomment-406454924
 
 
   Good call, Done!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125297)
Time Spent: 40m  (was: 0.5h)

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation yeilded the panic below, which triggered concurrent 
> modifications of the data channel maps.
>  
> The fix is properly guarding all writes to the Data channel maps, in 
> particular:
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
> Should be
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {  
> 

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125296
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:38
Start Date: 20/Jul/18 00:38
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6006: [BEAM-4808][SQL] add 
e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406454733
 
 
   FAILURE

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125296)
Time Spent: 40m  (was: 0.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #753

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7859670474358658850.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2074744482718364678.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2787010559474343330.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins626723809899662465.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7948156072487233671.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins826756654803787.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #766

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5757321793785259713.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5189875111550719659.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins135413268217935103.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins1918043972930392502.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8351820442774688904.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3824784014386049934.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125293
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:31
Start Date: 20/Jul/18 00:31
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6007: [BEAM-4834] Identify 
loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007#issuecomment-406453786
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125293)
Time Spent: 0.5h  (was: 20m)

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Cyclic graphs will never finish the topological sort so we should check the 
> cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4832) Concurrent Writes in Data channels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4832?focusedWorklogId=125292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125292
 ]

ASF GitHub Bot logged work on BEAM-4832:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:26
Start Date: 20/Jul/18 00:26
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #6003: [BEAM-4832] Fix 
concurrent race in datamgr.go
URL: https://github.com/apache/beam/pull/6003#issuecomment-406453049
 
 
   Please add comments to the mutex to indicate that it protects the readers 
and writers maps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125292)
Time Spent: 0.5h  (was: 20m)

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation yeilded the panic below, which triggered concurrent 
> modifications of the data channel maps.
>  
> The fix is properly guarding all writes to the Data channel maps, in 
> particular:
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
> Should be
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {  
> 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #259

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins7014121201278485977.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins1507439249885817237.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins8182657405119869865.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins4181473182017996307.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins2114807034027533093.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins2155237861700031420.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

Build failed in Jenkins: beam_PreCommit_Java_Cron #124

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
[...truncated 16.37 MB...]
INFO: 2018-07-20T00:19:00.893Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:00.944Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:364#1d275f544daf228c
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:00.988Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.036Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.082Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.137Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.178Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.217Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.253Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.297Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.326Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.370Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 20, 2018 12:19:05 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-20T00:19:01.418Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #778

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam13 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7131337087305091436.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8092599008138803864.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8724020036533236134.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7836504532092509209.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7006014397908353548.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1496926454678047433.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT_HDFS #154

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8639237850907070594.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6864170402578446781.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6600042018070870039.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-parquetioit-hdfs-154
namespace "beam-performancetests-parquetioit-hdfs-154" created
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2511613020226154189.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-parquetioit-hdfs-154
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6808392698786102125.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins731509699571479778.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1634226197610801908.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7462741708196316852.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5282368028423012135.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4683780841275722718.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #759

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6111889592627881240.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins7183639501168346468.sh
+ rm -rf .env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins772438178919697511.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5040256769810216600.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5742921418602932263.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5948931480302286912.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #436

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8920992676523719316.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6097053804917533185.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7401532941956433510.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-compressed-textioit-hdfs-436
namespace "beam-performancetests-compressed-textioit-hdfs-436" created
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2386062541118122279.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-compressed-textioit-hdfs-436
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7990645803359857802.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3986756723617480538.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5158069921187886812.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7663006266610262563.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8316350458215125177.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 

[jira] [Commented] (BEAM-4832) Concurrent Writes in Data channels

[Bill Neubauer (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550056#comment-16550056
 ] 

Bill Neubauer commented on BEAM-4832:
-

The comments for the mutex should indicate that it protects the readers and 
writers maps. That will help future reviewers maintain the locking invariants.

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation yeilded the panic below, which triggered concurrent 
> modifications of the data channel maps.
>  
> The fix is properly guarding all writes to the Data channel maps, in 
> particular:
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
> Should be
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {  
> [c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=198=xref_jump_to_def=c=205012539].[mu|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=96=xref_jump_to_def=mu=205012539].[Lock|https://cs.corp.google.com/piper///depot/google3/third_party/go/gc/src/sync/mutex.go?l=72=xref_jump_to_def=Lock=205012539]()
>   
> 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #435

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5041235359169071047.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6591843205527662172.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2028268006842829817.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-xmlioit-hdfs-435
namespace "beam-performancetests-xmlioit-hdfs-435" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8690548796771667078.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-xmlioit-hdfs-435
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3530095379385410958.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins151317527341084635.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8665814514415086715.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins393496193137850815.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4018958584195723655.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4339524442501207383.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #446

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8853947903190849280.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7247993095443956934.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1014720200870397096.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-textioit-hdfs-446
namespace "beam-performancetests-textioit-hdfs-446" created
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2203819654429193483.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-textioit-hdfs-446
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4384612114209580075.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins9168363747922715813.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2457285435403225541.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7693003019667280177.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1167606097930914311.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5867669338273773617.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #535

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins7040872982862938990.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins6163123683414681864.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins7607111520399213742.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-hadoopinputformat-535
namespace "beam-performancetests-hadoopinputformat-535" created
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins6644517011025666463.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-hadoopinputformat-535
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins7035852227243632070.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins2836472724981303912.sh
+ rm -rf .env
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins8792526416535801369.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins5324281738854189608.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1577586122083092265.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins6961261101008092930.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 

Build failed in Jenkins: beam_PerformanceTests_Spark #1974

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3787075178260564397.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5038119115218167667.sh
+ rm -rf .env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5831100105724113616.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6955147825445577880.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7451195538029071570.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2259344983594308914.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from 

[jira] [Assigned] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka reassigned BEAM-4834:
--

Assignee: Ankur Goenka  (was: Kenneth Knowles)

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Cyclic graphs will never finish the topological sort so we should check the 
> cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125289
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:11
Start Date: 20/Jul/18 00:11
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #6007: [BEAM-4834] 
Identify loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125289)
Time Spent: 10m
Remaining Estimate: 0h

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Kenneth Knowles
>  

[jira] [Work logged] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4834?focusedWorklogId=125290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125290
 ]

ASF GitHub Bot logged work on BEAM-4834:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:11
Start Date: 20/Jul/18 00:11
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6007: [BEAM-4834] Identify 
loops before doing topological sort of pipeline network
URL: https://github.com/apache/beam/pull/6007#issuecomment-406450874
 
 
   R; @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125290)
Time Spent: 20m  (was: 10m)

> Validate cycles in org.apache.beam.runners.core.construction.graph.Network 
> before doing topological sort
> 
>
> Key: BEAM-4834
> URL: https://issues.apache.org/jira/browse/BEAM-4834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ankur Goenka
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Cyclic graphs will never finish the topological sort so we should check the 
> cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125288
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:10
Start Date: 20/Jul/18 00:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406450681
 
 
   R: @akedin @apilloud  


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125288)
Time Spent: 0.5h  (was: 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125287
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:10
Start Date: 20/Jul/18 00:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-406450624
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125287)
Time Spent: 20m  (was: 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4834) Validate cycles in org.apache.beam.runners.core.construction.graph.Network before doing topological sort

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-4834:
--

 Summary: Validate cycles in 
org.apache.beam.runners.core.construction.graph.Network before doing 
topological sort
 Key: BEAM-4834
 URL: https://issues.apache.org/jira/browse/BEAM-4834
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Ankur Goenka
Assignee: Kenneth Knowles


Cyclic graphs will never finish the topological sort so we should check the 
cycle before doing the topological sort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #530

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins5541823732980380055.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins6013590192034407611.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins6368512422942104785.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins2801923367963557911.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins4643500477953834047.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins1211005476155737474.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl

Build failed in Jenkins: beam_PerformanceTests_JDBC #864

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins3134373611484234921.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8158563824985112558.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7279512122801736832.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-jdbc-864
namespace "beam-performancetests-jdbc-864" created
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8530206595724531260.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-jdbc-864
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins397398473611759988.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6617362698399179411.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1716040388451611229.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5286414359809798159.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8562840997787030780.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins2932700663127457247.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #435

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4653] Add support to the Java SDK harness to execute timers.

[amaliujia] BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages

[lcwik] Address PR comments.

[lcwik] Address PR comments.

[jxlewis] Added ClientConfiguration to AwsOptions and S3FileSystem

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 0e18bf4c81e09c193e113c74cac7301dc26dac9e (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Commit message: "Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT"
 > git rev-list --no-walk e33852830d863cfa9f9b3250e81afe29b8321d4b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6622742086621075261.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8045500271777531383.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6473770013710275328.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-avroioit-hdfs-435
namespace "beam-performancetests-avroioit-hdfs-435" created
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6190848513186538346.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-avroioit-hdfs-435
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3714961899312108140.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8207166967409531888.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8884937331115213891.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4713174701112090332.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7965233106226853384.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5578795317558120745.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages 

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=125286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125286
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:06
Start Date: 20/Jul/18 00:06
Worklog Time Spent: 10m 
  Work Description: amaliujia opened a new pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006
 
 
   Add integration tests for BeamSqlLine.
   
   Right now, there are some wired behaviors when running queries with windows. 
So this PR only includes tests without using windows. I am still investigating 
TUBLE/HOP queries.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125286)
Time Spent: 10m
Remaining Estimate: 0h

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=125285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125285
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 20/Jul/18 00:04
Start Date: 20/Jul/18 00:04
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6005: [BEAM-4833][WIP] Add 
support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005#issuecomment-406449753
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125285)
Time Spent: 20m  (was: 10m)

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: BEAM-4833
> URL: https://issues.apache.org/jira/browse/BEAM-4833
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: holdenk
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> It's pretty common that Python scripts require extra dependencies, even the 
> tensorflow model analysis TFMA example requires a different version of TF 
> than the one we install by default. While users can roll their own container 
> or edit the Dockerfile, it would probably be useful to provide an easier path 
> to integrating their dependencies.
> While we support automatically installing the dependencies at runtime on the 
> workers, this can be very slow, especially for things like tensorflow, arrow, 
> or other numeric heavy code.
> Another alternative could be a simple script to augment the existing base 
> image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4471) Documentation for Windows SDK users

[Rafael Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550040#comment-16550040
 ] 

Rafael Fernandez commented on BEAM-4471:


I've sent a PR updating Java quickstart instructions.

https://github.com/apache/beam-site/pull/503

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=125277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125277
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 19/Jul/18 23:45
Start Date: 19/Jul/18 23:45
Worklog Time Spent: 10m 
  Work Description: holdenk opened a new pull request #6005: 
[BEAM-4833][WIP] Add support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005
 
 
   Attempts to make it easier for folks to add their own custom requirements 
for use with Python and the portable runners. Some alternative options exist 
that could also make it easier, e.g. adding a python or shell script which 
generates a docker file based on the "vanilla" one as the starting point (or 
similar gradle target), or not doing this at all.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ X ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [  ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125277)
Time Spent: 10m
Remaining Estimate: 0h

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: 

[jira] [Work logged] (BEAM-4257) Add error reason and table destination to BigQueryIO streaming failed inserts

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4257?focusedWorklogId=125262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125262
 ]

ASF GitHub Bot logged work on BEAM-4257:


Author: ASF GitHub Bot
Created on: 19/Jul/18 22:16
Start Date: 19/Jul/18 22:16
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5341: [BEAM-4257] 
Increases BigQuery streaming error information
URL: https://github.com/apache/beam/pull/5341#issuecomment-406431360
 
 
   @calonso I apologize for the delay, I simply forgot to re-review this. The 
code looks good, and thanks for the detailed unit tests!
   
   There appears to be a compilation breakage. It might be because you are 
based against an old version of master, so I would try to rebase (git fetch 
--all git rebase origin/master) and see if that fixes things. I'll merge this 
PR once compilation and tests pass.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125262)
Time Spent: 4h  (was: 3h 50m)

> Add error reason and table destination to BigQueryIO streaming failed inserts
> -
>
> Key: BEAM-4257
> URL: https://issues.apache.org/jira/browse/BEAM-4257
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Carlos Alonso
>Assignee: Carlos Alonso
>Priority: Minor
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> When using `BigQueryIO.Write` and getting `WriteResult.getFailedInserts()` we 
> get a `PCollection` which is fine, but in order to properly work on 
> the errors downstream having extended information such as the `InsertError` 
> fields and the `TableReference` it was routed to would be really valuable.
>  
> My suggestion is to create a new object that contains all that information 
> and return a `PCollection` of those instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3032) Add RedshiftIO

[Jacob Marble (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549936#comment-16549936
 ] 

Jacob Marble commented on BEAM-3032:


Thanks, guys, that's really nice of you.

https://github.com/apache/beam/pull/6004

> Add RedshiftIO
> --
>
> Key: BEAM-3032
> URL: https://issues.apache.org/jira/browse/BEAM-3032
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
> Environment: AWS Redshift
>Reporter: Jacob Marble
>Assignee: Jacob Marble
>Priority: Minor
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> I would like to add a RedshiftIO Java extension to perform bulk read/write 
> to/from AWS Redshift via the UNLOAD and COPY Redshift SQL commands. This 
> requires S3, which is the subject of BEAM-2500.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4832) Concurrent Writes in Data channels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4832?focusedWorklogId=125256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125256
 ]

ASF GitHub Bot logged work on BEAM-4832:


Author: ASF GitHub Bot
Created on: 19/Jul/18 21:54
Start Date: 19/Jul/18 21:54
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6003: [BEAM-4832] Fix 
concurrent race in datamgr.go
URL: https://github.com/apache/beam/pull/6003#issuecomment-406426217
 
 
   R: @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125256)
Time Spent: 20m  (was: 10m)

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation yeilded the panic below, which triggered concurrent 
> modifications of the data channel maps.
>  
> The fix is properly guarding all writes to the Data channel maps, in 
> particular:
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
> Should be
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {  
> 

[jira] [Work logged] (BEAM-4832) Concurrent Writes in Data channels

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4832?focusedWorklogId=125255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125255
 ]

ASF GitHub Bot logged work on BEAM-4832:


Author: ASF GitHub Bot
Created on: 19/Jul/18 21:53
Start Date: 19/Jul/18 21:53
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #6003: [BEAM-4832] 
Fix concurrent race in datamgr.go
URL: https://github.com/apache/beam/pull/6003
 
 
   Concurrent access to the datachannel maps wasn't guarded when deleting 
readers.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125255)
Time Spent: 10m
Remaining Estimate: 0h

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation 

[jira] [Assigned] (BEAM-4832) Concurrent Writes in Data channels

[Robert Burke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke reassigned BEAM-4832:
--

Assignee: Robert Burke  (was: Henning Rohde)

> Concurrent Writes in Data channels
> --
>
> Key: BEAM-4832
> URL: https://issues.apache.org/jira/browse/BEAM-4832
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>
> A user was having issue with streaming Go pipelines on Dataflow. 
> Investigation yeilded the panic below, which triggered concurrent 
> modifications of the data channel maps.
>  
> The fix is properly guarding all writes to the Data channel maps, in 
> particular:
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
> Should be
>  
> func 
> ([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
>  
> *[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
>  
> [removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
>  string) {  
> [c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=198=xref_jump_to_def=c=205012539].[mu|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=96=xref_jump_to_def=mu=205012539].[Lock|https://cs.corp.google.com/piper///depot/google3/third_party/go/gc/src/sync/mutex.go?l=72=xref_jump_to_def=Lock=205012539]()
>   
> delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
>  
> [id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])
>   
> 

Jenkins build is back to normal : beam_PostCommit_Java_Nexmark_Dataflow #3

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4658) Update pipeline representation in runner support libraries to handle timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4658?focusedWorklogId=125253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125253
 ]

ASF GitHub Bot logged work on BEAM-4658:


Author: ASF GitHub Bot
Created on: 19/Jul/18 21:34
Start Date: 19/Jul/18 21:34
Worklog Time Spent: 10m 
  Work Description: lukecwik opened a new pull request #6002: [BEAM-4658] 
Modify RemoteBundle interface to allow for multiple inputs.
URL: https://github.com/apache/beam/pull/6002
 
 
   This is towards supporting timers as PCollections.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125253)
Time Spent: 10m
Remaining Estimate: 0h

> Update pipeline representation in runner support libraries to handle timers
> ---
>
> Key: BEAM-4658
> URL: https://issues.apache.org/jira/browse/BEAM-4658
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 

[jira] [Work logged] (BEAM-3446) RedisIO non-prefix read operations

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3446?focusedWorklogId=125251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125251
 ]

ASF GitHub Bot logged work on BEAM-3446:


Author: ASF GitHub Bot
Created on: 19/Jul/18 21:25
Start Date: 19/Jul/18 21:25
Worklog Time Spent: 10m 
  Work Description: iemejia commented on a change in pull request #5841: 
[BEAM-3446] Fixes RedisIO non-prefix read operations
URL: https://github.com/apache/beam/pull/5841#discussion_r203876000
 
 

 ##
 File path: 
sdks/java/io/redis/src/main/java/org/apache/beam/sdk/io/redis/RedisIO.java
 ##
 @@ -279,28 +314,73 @@ public void processElement(ProcessContext 
processContext) throws Exception {
   while (!finished) {
 ScanResult scanResult = jedis.scan(cursor, scanParams);
 List keys = scanResult.getResult();
-
-Pipeline pipeline = jedis.pipelined();
-if (keys != null) {
-  for (String key : keys) {
-pipeline.get(key);
-  }
-  List values = pipeline.syncAndReturnAll();
-  for (int i = 0; i < values.size(); i++) {
-processContext.output(KV.of(keys.get(i), (String) values.get(i)));
-  }
+for (String k : keys) {
+  processContext.output(k);
 }
-
 cursor = scanResult.getStringCursor();
 if ("0".equals(cursor)) {
   finished = true;
 }
   }
 }
+  }
+  /** A {@link DoFn} requesting Redis server to get key/value pairs. */
+  private static class ReadFn extends BaseReadFn> {
+private int batchSize;
+private List bufferedKeys;
+BoundedWindow window;
+Instant lastMsg;
+
+@StartBundle
+public void startBundle(StartBundleContext context) {
+  bufferedKeys = new ArrayList<>();
+}
 
-@Teardown
-public void teardown() {
-  jedis.close();
+ReadFn(RedisConnectionConfiguration connectionConfiguration, int 
batchSize) {
+  super(connectionConfiguration);
+  this.batchSize = batchSize;
+}
+
+private int getBatchSize() {
+  return batchSize;
+}
+
+@ProcessElement
+public void processElement(ProcessContext processContext, BoundedWindow 
window)
+throws Exception {
+  String key = processContext.element();
+  bufferedKeys.add(key);
+  this.window = window;
+  this.lastMsg = processContext.timestamp();
+  if (bufferedKeys.size() > getBatchSize()) {
+List> kvs = fetchAndFlush();
+for (KV kv : kvs) {
+  processContext.output(kv);
+}
+  }
+}
+
+private List> fetchAndFlush() {
+  String[] keys = new String[bufferedKeys.size()];
+  bufferedKeys.toArray(keys);
+  List results = jedis.mget(keys);
+  assert bufferedKeys.size() == results.size();
+  List> kvs = new ArrayList<>(bufferedKeys.size());
+  for (int i = 0; i < bufferedKeys.size(); i++) {
+if (results.get(i) != null) {
+  kvs.add(KV.of(bufferedKeys.get(i), results.get(i)));
+}
+  }
+  bufferedKeys = new ArrayList<>();
+  return kvs;
+}
+
+@FinishBundle
+public void finishBundle(FinishBundleContext context) throws Exception {
+  List> kvs = fetchAndFlush();
+  for (KV kv : kvs) {
+context.output(kv, lastMsg, window);
 
 Review comment:
   Oh so silly of me I have misread the motivation on keeping the window, you 
are right, it makes total sense, in that case probably it is a better idea to 
store the elements in a Map with the window as key and the list of elements and 
use the window.maxTimeStamp (you don't need the lastMsg) and flush when enough 
elements, Similar to what is done here (but with the count logic): 
   
https://github.com/apache/beam/blob/70b653187d566da7eea2590f17a36bbf22ef8bed/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L825-L844


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125251)
Time Spent: 3.5h  (was: 3h 20m)

> RedisIO non-prefix read operations
> --
>
> Key: BEAM-3446
> URL: https://issues.apache.org/jira/browse/BEAM-3446
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-redis
>Reporter: Vinay varma
>Assignee: Vinay varma
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Read operation in RedisIO is for prefix based look ups. While this can be 
> used for exact key matches as well, the number of operations limits the 
> 

[jira] [Created] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[holdenk (JIRA)]

holdenk created BEAM-4833:
-

 Summary: Add support for users specifying a requirements.txt for 
their Python portable container
 Key: BEAM-4833
 URL: https://issues.apache.org/jira/browse/BEAM-4833
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: holdenk
Assignee: holdenk


It's pretty common that Python scripts require extra dependencies, even the 
tensorflow model analysis TFMA example requires a different version of TF than 
the one we install by default. While users can roll their own container or edit 
the Dockerfile, it would probably be useful to provide an easier path to 
integrating their dependencies.

While we support automatically installing the dependencies at runtime on the 
workers, this can be very slow, especially for things like tensorflow, arrow, 
or other numeric heavy code.

Another alternative could be a simple script to augment the existing base image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4471) Documentation for Windows SDK users

[Rafael Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafael Fernandez reassigned BEAM-4471:
--

Assignee: Rafael Fernandez  (was: Melissa Pashniak)

> Documentation for Windows SDK users
> ---
>
> Key: BEAM-4471
> URL: https://issues.apache.org/jira/browse/BEAM-4471
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Udi Meiri
>Assignee: Rafael Fernandez
>Priority: Major
>
> This is a request to document how to use Apache Beam on Windows machines.
> At minimum, add Windows-specific steps to the Java quickstart guide:
> [https://beam.apache.org/get-started/quickstart-java/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (70b6531 -> 0e18bf4)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 70b6531  [BEAM-4653] Add support to the Java SDK harness to execute 
timers.
 add a920902  BEAM-4820: Add timeout to Future.get when waiting for Pubsub 
messages
 new 0e18bf4  Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0e18bf4c81e09c193e113c74cac7301dc26dac9e
Merge: 70b6531 a920902
Author: Ahmet Altay 
AuthorDate: Thu Jul 19 13:43:52 2018 -0700

Merge pull request #5985 from amaliujia/rui_wang-fix_pubsubIT

[BEAM-4820][SQL] Add timeout to Future.get when waiting for Pubsub messages

 .../beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4820) Add timeout to Future.get when waiting for Pubsub messages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4820?focusedWorklogId=125242=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125242
 ]

ASF GitHub Bot logged work on BEAM-4820:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:43
Start Date: 19/Jul/18 20:43
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5985: [BEAM-4820][SQL] Add 
timeout to Future.get when waiting for Pubsub messages
URL: https://github.com/apache/beam/pull/5985
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
index 4e1c70aeb9c..196405a72ff 100644
--- 
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
+++ 
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
@@ -40,6 +40,7 @@
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
 import java.util.stream.Collectors;
 import org.apache.beam.sdk.extensions.sql.impl.BeamCalciteSchema;
 import org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv;
@@ -262,7 +263,7 @@ public void testSQLLimit() throws Exception {
 // wait one minute to allow subscription creation.
 Thread.sleep(60 * 1000);
 eventsTopic.publish(messages);
-assertThat(queryResult.get().size(), equalTo(3));
+assertThat(queryResult.get(2, TimeUnit.MINUTES).size(), equalTo(3));
 pool.shutdown();
   }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125242)
Time Spent: 1h 40m  (was: 1.5h)

> Add timeout to Future.get when waiting for Pubsub messages
> --
>
> Key: BEAM-4820
> URL: https://issues.apache.org/jira/browse/BEAM-4820
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Should have a timeout when waiting for Pubsub messages because if unbounded 
> limit is broken, the Future.get() will be blocking and never return.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.6.0 updated (7cf23fc -> 137c058)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch release-2.6.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7cf23fc  Merge pull request #5980 from boyuanzz/fix_release_branch
 add c986ed9  Add version bound to numpy.
 new 137c058  Merge pull request #5999 from tvalentyn/numpy_version_lock

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] 01/01: Merge pull request #5999 from tvalentyn/numpy_version_lock

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch release-2.6.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 137c05860dea29c9223725034d05225a737f4599
Merge: 7cf23fc c986ed9
Author: Ahmet Altay 
AuthorDate: Thu Jul 19 13:42:44 2018 -0700

Merge pull request #5999 from tvalentyn/numpy_version_lock

Add version bound to numpy.

 sdks/python/setup.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Resolved] (BEAM-4469) Setting up Jekyll on Windows is difficult

[Rafael Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafael Fernandez resolved BEAM-4469.

   Resolution: Fixed
Fix Version/s: Not applicable

> Setting up Jekyll on Windows is difficult
> -
>
> Key: BEAM-4469
> URL: https://issues.apache.org/jira/browse/BEAM-4469
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Minor
> Fix For: Not applicable
>
>
> This bug is to capture 
> [conversation|https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit?pli=1=B50kLXQ]
>  on the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] doc. [~altay] 
> mentioned that setting up Jekyll is not easy, particularly on Windows. I'd 
> like to understand the issues more and see how we can improve them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4469) Setting up Jekyll on Windows is difficult

[Rafael Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549823#comment-16549823
 ] 

Rafael Fernandez edited comment on BEAM-4469 at 7/19/18 8:34 PM:
-

I spun up a Windows VM (2016) on x64. Here are my observations:

 
 * Following [https://jekyllrb.com/docs/windows/].
 * Attempted installing latest (*rubyinstaller-devkit-2.5.1-2-x64.exe*). 
Windows Defender claims the file contains a virus and deletes it. I believe it 
is unsigned.
 * As per the RubyInstaller site's instructions ,*2.4.4-2* works just fine. I 
installed MSYS2 and MINGW development toolchain (option 3).
 * I made sure the *ruby/bin* directory was in the system path (this can be 
done in System Properties / Advanced > Environment Variables)
 * *gem install jekyll bundler* worked as expected.
 * *jekyll -v* output:
 {{PS C:\Users\rfernand> jekyll -v}}
 {{jekyll 3.8.3}}
 {{PS C:\Users\rfernand>}}
 * *jekyll new jekyll-website* successfully installed the blog.

 

A comment in the discussion doc ([~thomasWeise]) acknowledges the process is 
not that painful (agree), and merely suggests it can be made even simpler with 
a container.


was (Author: rfernand):
I spun up a Windows VM (2016) on x64. Here are my observations:

 
 * Following [https://jekyllrb.com/docs/windows/].
 * Attempted installing latest (*rubyinstaller-devkit-2.5.1-2-x64.exe*). 
Windows Defender claims the file contains a virus and deletes it. I believe it 
is unsigned.
 * As per the RubyInstaller site's instructions ,*2.4.4-2* works just fine. I 
installed MSYS2 and MINGW development toolchain (option 3).
 * I made sure the *ruby/bin* directory was in the system path (this can be 
done in System Properties / Advanced > Environment Variables)
 * *gem install jekyll bundler* worked as expected.
 * *jekyll -v* output:
 {{PS C:\Users\rfernand> jekyll -v}}
 {{jekyll 3.8.3}}
 {{PS C:\Users\rfernand>}}
 * *jekyll new jekyll-website* successfully installed the blog.

> Setting up Jekyll on Windows is difficult
> -
>
> Key: BEAM-4469
> URL: https://issues.apache.org/jira/browse/BEAM-4469
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Minor
>
> This bug is to capture 
> [conversation|https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit?pli=1=B50kLXQ]
>  on the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] doc. [~altay] 
> mentioned that setting up Jekyll is not easy, particularly on Windows. I'd 
> like to understand the issues more and see how we can improve them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4469) Setting up Jekyll on Windows is difficult

[Rafael Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549823#comment-16549823
 ] 

Rafael Fernandez edited comment on BEAM-4469 at 7/19/18 8:32 PM:
-

I spun up a Windows VM (2016) on x64. Here are my observations:

 
 * Following [https://jekyllrb.com/docs/windows/].
 * Attempted installing latest (*rubyinstaller-devkit-2.5.1-2-x64.exe*). 
Windows Defender claims the file contains a virus and deletes it. I believe it 
is unsigned.
 * As per the RubyInstaller site's instructions ,*2.4.4-2* works just fine. I 
installed MSYS2 and MINGW development toolchain (option 3).
 * I made sure the *ruby/bin* directory was in the system path (this can be 
done in System Properties / Advanced > Environment Variables)
 * *gem install jekyll bundler* worked as expected.
 * *jekyll -v* output:
 {{PS C:\Users\rfernand> jekyll -v}}
 {{jekyll 3.8.3}}
 {{PS C:\Users\rfernand>}}
 * *jekyll new jekyll-website* successfully installed the blog.


was (Author: rfernand):
I spun up a Windows VM (2016) on x64. Here are my observations:

 
 * Following [https://jekyllrb.com/docs/windows/].
 * Attempted installing latest (*rubyinstaller-devkit-2.5.1-2-x64.exe*). 
Windows Defender claims the file contains a virus and deletes it. I believe it 
is unsigned.
 * As per the RubyInstaller site's instructions ,*2.4.4-2* works just fine. I 
installed MSYS2 and MINGW development toolchain (option 3).
 * I made sure the *ruby/bin* directory was in the system path (this can be 
done in System Properties / Advanced > Environment Variables)
 * *gem install jekyll bundler* worked as expected.
 * *jekyll -v* output:
{{PS C:\Users\rfernand> jekyll -v}}
{{jekyll 3.8.3}}
{{PS C:\Users\rfernand>}}
 * jekyll new jekyll-website successfully installed the blog.

> Setting up Jekyll on Windows is difficult
> -
>
> Key: BEAM-4469
> URL: https://issues.apache.org/jira/browse/BEAM-4469
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Minor
>
> This bug is to capture 
> [conversation|https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit?pli=1=B50kLXQ]
>  on the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] doc. [~altay] 
> mentioned that setting up Jekyll is not easy, particularly on Windows. I'd 
> like to understand the issues more and see how we can improve them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4469) Setting up Jekyll on Windows is difficult

[Rafael Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549823#comment-16549823
 ] 

Rafael Fernandez commented on BEAM-4469:


I spun up a Windows VM (2016) on x64. Here are my observations:

 
 * Following [https://jekyllrb.com/docs/windows/].
 * Attempted installing latest (*rubyinstaller-devkit-2.5.1-2-x64.exe*). 
Windows Defender claims the file contains a virus and deletes it. I believe it 
is unsigned.
 * As per the RubyInstaller site's instructions ,*2.4.4-2* works just fine. I 
installed MSYS2 and MINGW development toolchain (option 3).
 * I made sure the *ruby/bin* directory was in the system path (this can be 
done in System Properties / Advanced > Environment Variables)
 * *gem install jekyll bundler* worked as expected.
 * *jekyll -v* output:
{{PS C:\Users\rfernand> jekyll -v}}
{{jekyll 3.8.3}}
{{PS C:\Users\rfernand>}}
 * jekyll new jekyll-website successfully installed the blog.

> Setting up Jekyll on Windows is difficult
> -
>
> Key: BEAM-4469
> URL: https://issues.apache.org/jira/browse/BEAM-4469
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Minor
>
> This bug is to capture 
> [conversation|https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit?pli=1=B50kLXQ]
>  on the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] doc. [~altay] 
> mentioned that setting up Jekyll is not easy, particularly on Windows. I'd 
> like to understand the issues more and see how we can improve them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4653) Java SDK harness should support user timers

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4653.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Java SDK harness should support user timers
> ---
>
> Key: BEAM-4653
> URL: https://issues.apache.org/jira/browse/BEAM-4653
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
> Fix For: 2.7.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Wire up the onTimer method in the Java SDK harness FnApiDoFnRunner connecting 
> it to the RemoteGrpcPort read/write that is responsible for 
> producing/consumer timers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4653) Java SDK harness should support user timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4653?focusedWorklogId=125236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125236
 ]

ASF GitHub Bot logged work on BEAM-4653:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:24
Start Date: 19/Jul/18 20:24
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5898: [BEAM-4653] Add 
support to the Java SDK harness to execute timers.
URL: https://github.com/apache/beam/pull/5898
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 6003aa23f0d..433533ed4fd 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -961,8 +961,24 @@ public TimerInternalsTimer(
 
 @Override
 public void set(Instant target) {
-  verifyAbsoluteTimeDomain();
-  verifyTargetTime(target);
+  // Verifies that the time domain of this timer is acceptable for 
absolute timers.
+  if (!TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
+throw new IllegalStateException(
+"Can only set relative timers in processing time domain. Use 
#setRelative()");
+  }
+
+  // Ensures that the target time is reasonable. For event time timers 
this means that the time
+  // should be prior to window GC time.
+  if (TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
+Instant windowExpiry = window.maxTimestamp().plus(allowedLateness);
+checkArgument(
+!target.isAfter(windowExpiry),
+"Attempted to set event time timer for %s but that is after"
++ " the expiration of window %s",
+target,
+windowExpiry);
+  }
+
   setUnderlyingTimer(target);
 }
 
@@ -1006,30 +1022,6 @@ private Instant minTargetAndGcTime(Instant target) {
   return target;
 }
 
-/**
- * Ensures that the target time is reasonable. For event time timers this 
means that the time
- * should be prior to window GC time.
- */
-private void verifyTargetTime(Instant target) {
-  if (TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
-Instant windowExpiry = window.maxTimestamp().plus(allowedLateness);
-checkArgument(
-!target.isAfter(windowExpiry),
-"Attempted to set event time timer for %s but that is after"
-+ " the expiration of window %s",
-target,
-windowExpiry);
-  }
-}
-
-/** Verifies that the time domain of this timer is acceptable for absolute 
timers. */
-private void verifyAbsoluteTimeDomain() {
-  if (!TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
-throw new IllegalStateException(
-"Cannot only set relative timers in processing time domain." + " 
Use #setRelative()");
-  }
-}
-
 /**
  * Sets the timer for the target time without checking anything about 
whether it is a reasonable
  * thing to do. For example, absolute processing time timers are not 
really sensible since the
diff --git a/sdks/java/harness/build.gradle b/sdks/java/harness/build.gradle
index bdf20048bab..ce6b2db3370 100644
--- a/sdks/java/harness/build.gradle
+++ b/sdks/java/harness/build.gradle
@@ -59,4 +59,5 @@ dependencies {
   testCompile library.java.junit
   testCompile library.java.mockito_core
   testCompile library.java.slf4j_jdk14
+  testCompile project(path: ":beam-sdks-java-core", configuration: 
"shadowTest")
 }
diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
index cb3c731545c..6ec31551288 100644
--- 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/BeamFnDataReadRunner.java
@@ -22,7 +22,7 @@
 
 import com.google.auto.service.AutoService;
 import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.Multimap;
+import com.google.common.collect.ListMultimap;
 import java.io.IOException;
 import java.util.Collection;
 import java.util.Map;
@@ -87,7 +87,7 @@
 Map pCollections,
 Map coders,
 Map windowingStrategies,
-Multimap>> 
pCollectionIdsToConsumers,
+ListMultimap>> 
pCollectionIdsToConsumers,
 Consumer addStartFunction,
 Consumer 

[jira] [Work logged] (BEAM-4653) Java SDK harness should support user timers

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4653?focusedWorklogId=125237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125237
 ]

ASF GitHub Bot logged work on BEAM-4653:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:24
Start Date: 19/Jul/18 20:24
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5898: 
[BEAM-4653] Add support to the Java SDK harness to execute timers.
URL: https://github.com/apache/beam/pull/5898#discussion_r203575993
 
 

 ##
 File path: 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
 ##
 @@ -433,4 +589,178 @@ public void updateWatermark(Instant watermark) {
   throw new UnsupportedOperationException("TODO: Add support for 
SplittableDoFn");
 }
   }
+
+  /** Provides arguments for a {@link DoFnInvoker} for {@link DoFn.OnTimer 
@OnTimer}. */
+  private class OnTimerContext extends DoFn.OnTimerContext
+  implements DoFnInvoker.ArgumentProvider {
+
+private OnTimerContext() {
+  context.doFn.super();
+}
+
+@Override
+public BoundedWindow window() {
+  return currentWindow;
+}
+
+@Override
+public PaneInfo paneInfo(DoFn doFn) {
+  throw new UnsupportedOperationException(
+  "Cannot access paneInfo outside of @ProcessElement methods.");
+}
+
+@Override
+public DoFn.StartBundleContext 
startBundleContext(DoFn doFn) {
+  throw new UnsupportedOperationException(
+  "Cannot access StartBundleContext outside of @StartBundle method.");
+}
+
+@Override
+public DoFn.FinishBundleContext finishBundleContext(
+DoFn doFn) {
+  throw new UnsupportedOperationException(
+  "Cannot access FinishBundleContext outside of @FinishBundle 
method.");
+}
+
+@Override
+public DoFn.ProcessContext processContext(DoFn doFn) {
+  throw new UnsupportedOperationException(
+  "Cannot access ProcessContext outside of @ProcessElement method.");
+}
+
+@Override
+public InputT element(DoFn doFn) {
+  throw new UnsupportedOperationException("Element parameters are not 
supported.");
+}
+
+@Override
+public Instant timestamp(DoFn doFn) {
+  return timestamp();
+}
+
+@Override
+public Row asRow(@Nullable String id) {
+  throw new UnsupportedOperationException(
+  "Cannot access element outside of @ProcessElement method.");
+}
+
+@Override
+public TimeDomain timeDomain(DoFn doFn) {
+  return timeDomain();
+}
+
+@Override
+public OutputReceiver outputReceiver(DoFn doFn) {
+  return DoFnOutputReceivers.windowedReceiver(this, null);
+}
+
+@Override
+public OutputReceiver outputRowReceiver(DoFn doFn) {
+  return DoFnOutputReceivers.rowReceiver(this, null, 
mainOutputSchemaCoder);
+}
+
+@Override
+public MultiOutputReceiver taggedOutputReceiver(DoFn 
doFn) {
+  return DoFnOutputReceivers.windowedMultiReceiver(this);
+}
+
+@Override
+public DoFn.OnTimerContext onTimerContext(DoFn doFn) {
+  return this;
+}
+
+@Override
+public RestrictionTracker restrictionTracker() {
+  throw new UnsupportedOperationException("RestrictionTracker parameters 
are not supported.");
+}
+
+@Override
+public State state(String stateId) {
+  StateDeclaration stateDeclaration = 
context.doFnSignature.stateDeclarations().get(stateId);
+  checkNotNull(stateDeclaration, "No state declaration found for %s", 
stateId);
+  StateSpec spec;
+  try {
+spec = (StateSpec) stateDeclaration.field().get(context.doFn);
+  } catch (IllegalAccessException e) {
+throw new RuntimeException(e);
+  }
+  return spec.bind(stateId, stateAccessor);
+}
+
+@Override
+public org.apache.beam.sdk.state.Timer timer(String timerId) {
+  checkState(
+  currentTimer.getValue() instanceof KV,
+  "Accessing timer in unkeyed context. Current timer is not a KV: %s.",
+  currentTimer);
+
+  return new FnApiTimer(timerId, (WindowedValue) currentTimer);
+}
+
+@Override
+public PipelineOptions getPipelineOptions() {
+  return context.pipelineOptions;
+}
+
+@Override
+public PipelineOptions pipelineOptions() {
+  return context.pipelineOptions;
+}
+
+@Override
+public void output(OutputT output) {
+  outputTo(
+  mainOutputConsumers,
+  WindowedValue.of(output, currentTimer.getTimestamp(), currentWindow, 
PaneInfo.NO_FIRING));
 
 Review comment:
   Currently replicating existing timer output behavior. I do believe we should 
get a copy of the PaneInfo that was on the timer.
   
   @takidau / @relax / @kennknowles Should the PaneInfo get propagated from the 
element that created the timer to the output of that timer when it fires?

[jira] [Created] (BEAM-4832) Concurrent Writes in Data channels

[Robert Burke (JIRA)]

Robert Burke created BEAM-4832:
--

 Summary: Concurrent Writes in Data channels
 Key: BEAM-4832
 URL: https://issues.apache.org/jira/browse/BEAM-4832
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Robert Burke
Assignee: Henning Rohde


A user was having issue with streaming Go pipelines on Dataflow. Investigation 
yeilded the panic below, which triggered concurrent modifications of the data 
channel maps.

 

The fix is properly guarding all writes to the Data channel maps, in particular:

 
func 
([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
 
*[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
 
[removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
 string) { 
delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
 
[id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])}
Should be
 
func 
([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Ac=c=xref_usages]
 
*[DataChannel|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=87=xref_jump_to_def=DataChannel=205012539])
 
[removeReader|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523method%252520DataChannel.removeReader=removeReader=xref_usages]([id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=kythe%253A%252F%252Fgoogle3%253Flang%253Dgo%253Fpath%253Dthird_party%252Fgolang%252Fapache_beam%252Fpkg%252Fbeam%252Fcore%252Fruntime%252Fharness%252Fharness%2523param%252520DataChannel.removeReader%25253Aid=id=xref_usages]
 string) {  
[c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=198=xref_jump_to_def=c=205012539].[mu|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=96=xref_jump_to_def=mu=205012539].[Lock|https://cs.corp.google.com/piper///depot/google3/third_party/go/gc/src/sync/mutex.go?l=72=xref_jump_to_def=Lock=205012539]()
  
delete([c|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=c=205012539].[readers|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=93=xref_jump_to_def=readers=205012539],
 
[id|https://cs.corp.google.com/piper///depot/google3/third_party/golang/apache_beam/pkg/beam/core/runtime/harness/datamgr.go?l=212=xref_jump_to_def=id=205012539])
  

[jira] [Work logged] (BEAM-4827) Add nexmark dashboards to nexmark page

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4827?focusedWorklogId=125233=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125233
 ]

ASF GitHub Bot logged work on BEAM-4827:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:23
Start Date: 19/Jul/18 20:23
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #500:  [BEAM-4827] Add 
Nexmark dashboards to nexmark page on the website
URL: https://github.com/apache/beam-site/pull/500#issuecomment-406402579
 
 
   LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125233)
Time Spent: 0.5h  (was: 20m)

> Add nexmark dashboards to nexmark page
> --
>
> Key: BEAM-4827
> URL: https://issues.apache.org/jira/browse/BEAM-4827
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> To make these dashboards accessible and visible, we should publish them on 
> the website as discussed on the ML.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=125221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125221
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:14
Start Date: 19/Jul/18 20:14
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5966: [BEAM-4761] Run 
Nexmark on Dataflow
URL: https://github.com/apache/beam/pull/5966#issuecomment-406400188
 
 
   This appears to be working now: 
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_Nexmark_Dataflow/3/
 I made one more change and bumped the run size up, as dataflow takes ~2 
minutes to spin up (which counts towards Nexmark run time). We need more than 2 
seconds of work to get any sort of test.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125221)
Time Spent: 2.5h  (was: 2h 20m)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4761) Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4761?focusedWorklogId=125222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125222
 ]

ASF GitHub Bot logged work on BEAM-4761:


Author: ASF GitHub Bot
Created on: 19/Jul/18 20:14
Start Date: 19/Jul/18 20:14
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5966: [BEAM-4761] Run 
Nexmark on Dataflow
URL: https://github.com/apache/beam/pull/5966#issuecomment-406400224
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125222)
Time Spent: 2h 40m  (was: 2.5h)

> Add postCommit scripts and perfkit dashboards for nexmark on Dataflow runner
> 
>
> Key: BEAM-4761
> URL: https://issues.apache.org/jira/browse/BEAM-4761
> Project: Beam
>  Issue Type: Test
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4830) Update Jenkins build to run ./gradlew check instead of just ./gradlew build

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549787#comment-16549787
 ] 

Luke Cwik commented on BEAM-4830:
-

Note that build already depends on check, is this really needed?

> Update Jenkins build to run ./gradlew check instead of just ./gradlew build
> ---
>
> Key: BEAM-4830
> URL: https://issues.apache.org/jira/browse/BEAM-4830
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Alex Amato
>Assignee: Luke Cwik
>Priority: Major
>
> The purpose of this is to catch errors developers see when they first start 
> contributing to beam. Let's ensure we run the same commands in the 
> [contributing guide|https://beam.apache.org/contribute/].
>  
> Note: check runs more than build, so we are not catching these problems in 
> the continuous Jenkins testing.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3032) Add RedshiftIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549783#comment-16549783
 ] 

Jean-Baptiste Onofré commented on BEAM-3032:


+1, I'm interested to help as well.

> Add RedshiftIO
> --
>
> Key: BEAM-3032
> URL: https://issues.apache.org/jira/browse/BEAM-3032
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
> Environment: AWS Redshift
>Reporter: Jacob Marble
>Assignee: Jacob Marble
>Priority: Minor
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> I would like to add a RedshiftIO Java extension to perform bulk read/write 
> to/from AWS Redshift via the UNLOAD and COPY Redshift SQL commands. This 
> requires S3, which is the subject of BEAM-2500.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3032) Add RedshiftIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549773#comment-16549773
 ] 

Ismaël Mejía commented on BEAM-3032:


We are interested in this code, and we can validate its worth in production so 
we will be glad to take it (and maintain it as we did with your excellent S3 
contribution). So please bring your code and I will take care (or find someone) 
who can help us to take it in.

> Add RedshiftIO
> --
>
> Key: BEAM-3032
> URL: https://issues.apache.org/jira/browse/BEAM-3032
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
> Environment: AWS Redshift
>Reporter: Jacob Marble
>Assignee: Jacob Marble
>Priority: Minor
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> I would like to add a RedshiftIO Java extension to perform bulk read/write 
> to/from AWS Redshift via the UNLOAD and COPY Redshift SQL commands. This 
> requires S3, which is the subject of BEAM-2500.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4814) Support for S3FileSystem to work behind a proxy server

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4814.

   Resolution: Fixed
Fix Version/s: 2.7.0

> Support for S3FileSystem to work behind a proxy server
> --
>
> Key: BEAM-4814
> URL: https://issues.apache.org/jira/browse/BEAM-4814
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: John Rudolf Lewis
>Assignee: John Rudolf Lewis
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> I want to run a Beam job in my Spark cluster that uses the S3FileSystem. My 
> Spark cluster is configured to require a proxy server with authentication in 
> order to make outbound connections. A small change is required to enable 
> ClientConfiguration to be added to the configuration to enable this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4814) Support for S3FileSystem to work behind a proxy server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4814?focusedWorklogId=125216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125216
 ]

ASF GitHub Bot logged work on BEAM-4814:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:56
Start Date: 19/Jul/18 19:56
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #5983: [BEAM-4814] Add 
client configuration to aws options
URL: https://github.com/apache/beam/pull/5983
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
index 48bb74342c1..f78e08968eb 100644
--- 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
+++ 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsModule.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.sdk.io.aws.options;
 
+import com.amazonaws.ClientConfiguration;
 import com.amazonaws.auth.AWSCredentialsProvider;
 import com.amazonaws.auth.AWSStaticCredentialsProvider;
 import com.amazonaws.auth.BasicAWSCredentials;
@@ -29,6 +30,9 @@
 import com.amazonaws.auth.profile.ProfileCredentialsProvider;
 import com.amazonaws.services.s3.model.SSEAwsKeyManagementParams;
 import com.amazonaws.services.s3.model.SSECustomerKey;
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility;
+import com.fasterxml.jackson.annotation.JsonProperty;
 import com.fasterxml.jackson.annotation.JsonTypeInfo;
 import com.fasterxml.jackson.core.JsonGenerator;
 import com.fasterxml.jackson.core.JsonParser;
@@ -68,6 +72,7 @@ public AwsModule() {
 setMixInAnnotation(AWSCredentialsProvider.class, 
AWSCredentialsProviderMixin.class);
 setMixInAnnotation(SSECustomerKey.class, SSECustomerKeyMixin.class);
 setMixInAnnotation(SSEAwsKeyManagementParams.class, 
SSEAwsKeyManagementParamsMixin.class);
+setMixInAnnotation(ClientConfiguration.class, 
ClientConfigurationMixin.class);
   }
 
   /** A mixin to add Jackson annotations to {@link AWSCredentialsProvider}. */
@@ -234,4 +239,23 @@ public SSEAwsKeyManagementParams deserialize(JsonParser 
parser, DeserializationC
   return new SSEAwsKeyManagementParams(awsKmsKeyId);
 }
   }
+
+  @JsonAutoDetect(
+fieldVisibility = Visibility.NONE,
+getterVisibility = Visibility.NONE,
+setterVisibility = Visibility.NONE
+  )
+  interface ClientConfigurationMixin {
+@JsonProperty
+String getProxyHost();
+
+@JsonProperty
+Integer getProxyPort();
+
+@JsonProperty
+String getProxyUsername();
+
+@JsonProperty
+String getProxyPassword();
+  }
 }
diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsOptions.java
 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsOptions.java
index 4f8969a11b4..65a6545f878 100644
--- 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsOptions.java
+++ 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/options/AwsOptions.java
@@ -18,6 +18,7 @@
 
 package org.apache.beam.sdk.io.aws.options;
 
+import com.amazonaws.ClientConfiguration;
 import com.amazonaws.auth.AWSCredentialsProvider;
 import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
 import org.apache.beam.sdk.options.Default;
@@ -75,4 +76,40 @@ public AWSCredentialsProvider create(PipelineOptions 
options) {
   return DefaultAWSCredentialsProviderChain.getInstance();
 }
   }
+
+  /**
+   * The client configuration instance that should be used to configure AWS 
service clients. Please
+   * note that the configuration deserialization only allows one to specify 
proxy settings.
+   *
+   * For example, to specify the proxy host, port, username and password, 
specify the following:
+   * 
+   * --clientConfiguration={
+   *   "proxyHost":"hostname",
+   *   "proxyPort":1234,
+   *   "proxyUsername":"username",
+   *   "proxyPassword":"password"
+   * }
+   * 
+   *
+   * @return
+   */
+  @Description(
+  "The client configuration instance that should be used to configure AWS 
service "
+  + "clients. Please note that the configuration deserialization only 
allows one to specify "
+  + "proxy settings. For example, to specify the proxy host, port, 
username and password, "
+  + "specify the following: 
--clientConfiguration={\"proxyHost\":\"hostname\",\"proxyPort\":1234,"
+  + 

[beam] branch master updated (e338528 -> f92ce36)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e338528  Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline
 add f1d4d41  Added ClientConfiguration to AwsOptions and S3FileSystem
 new f92ce36  Merge pull request #5983: [BEAM-4814] Add client 
configuration to aws options

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/io/aws/options/AwsModule.java  | 24 ++
 .../apache/beam/sdk/io/aws/options/AwsOptions.java | 37 ++
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  5 +++
 .../beam/sdk/io/aws/options/AwsModuleTest.java | 18 +++
 4 files changed, 84 insertions(+)

[beam] 01/01: Merge pull request #5983: [BEAM-4814] Add client configuration to aws options

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f92ce369bffae1fbedf07e5d0db508dd9627ba00
Merge: e338528 f1d4d41
Author: Ismaël Mejía 
AuthorDate: Thu Jul 19 21:56:48 2018 +0200

Merge pull request #5983: [BEAM-4814] Add client configuration to aws 
options

 .../apache/beam/sdk/io/aws/options/AwsModule.java  | 24 ++
 .../apache/beam/sdk/io/aws/options/AwsOptions.java | 37 ++
 .../apache/beam/sdk/io/aws/s3/S3FileSystem.java|  5 +++
 .../beam/sdk/io/aws/options/AwsModuleTest.java | 18 +++
 4 files changed, 84 insertions(+)

[jira] [Assigned] (BEAM-4469) Setting up Jekyll on Windows is difficult

[Rafael Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rafael Fernandez reassigned BEAM-4469:
--

Assignee: Rafael Fernandez

> Setting up Jekyll on Windows is difficult
> -
>
> Key: BEAM-4469
> URL: https://issues.apache.org/jira/browse/BEAM-4469
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, website
>Reporter: Scott Wegner
>Assignee: Rafael Fernandez
>Priority: Minor
>
> This bug is to capture 
> [conversation|https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit?pli=1=B50kLXQ]
>  on the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] doc. [~altay] 
> mentioned that setting up Jekyll is not easy, particularly on Windows. I'd 
> like to understand the issues more and see how we can improve them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125212
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:48
Start Date: 19/Jul/18 19:48
Worklog Time Spent: 10m 
  Work Description: jbonofre commented on issue #1729: [BEAM-1240] Create 
RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#issuecomment-406393148
 
 
   @jkff thanks for the update, I will address your comments !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125212)
Time Spent: 9h 20m  (was: 9h 10m)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4492) Update Python bigquery library to latest version

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549747#comment-16549747
 ] 

Mark Liu commented on BEAM-4492:


Following link shows places that use google-cloud-bigquery library:
https://github.com/apache/beam/search?l=Python=%22from+google.cloud+import+bigquery%22

> Update Python bigquery library to latest version
> 
>
> Key: BEAM-4492
> URL: https://issues.apache.org/jira/browse/BEAM-4492
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>
> Current google-cloud-bigquery is set to 0.25.0 in 
> https://github.com/apache/beam/blob/master/sdks/python/setup.py#L130
> However, the latest version is 1.2.0.
> According to comment in setup.py, this library is only used for testing, so 
> it should be easy to update. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4831) Fix broken go vet task in the gradle build

[Alex Amato (JIRA)]

Alex Amato created BEAM-4831:


 Summary: Fix broken go vet task in the gradle build
 Key: BEAM-4831
 URL: https://issues.apache.org/jira/browse/BEAM-4831
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Alex Amato
Assignee: Luke Cwik


Reproduce by running

{{./gradlew check}}

{{Today this is failing.}}

{{go vet seems to be trying to validate files in a vendor folder.}}

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4830) Update Jenkins build to run ./gradlew check instead of just ./gradlew build

[Alex Amato (JIRA)]

Alex Amato created BEAM-4830:


 Summary: Update Jenkins build to run ./gradlew check instead of 
just ./gradlew build
 Key: BEAM-4830
 URL: https://issues.apache.org/jira/browse/BEAM-4830
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Alex Amato
Assignee: Luke Cwik


The purpose of this is to catch errors developers see when they first start 
contributing to beam. Let's ensure we run the same commands in the 
[contributing guide|https://beam.apache.org/contribute/].

 

Note: check runs more than build, so we are not catching these problems in the 
continuous Jenkins testing.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125193
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837280
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,655 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.MessageProperties;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}) wrapped as {@link RabbitMqMessage}.
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect to a
+ * RabbitMQ broker. The following example illustrates various options for 
configuring the source,
+ * reading from the queue:
+ *
+ * {@code
+ * pipeline.apply(
+ *   
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * It's also possible to read from an exchange (providing the exchange type 
and routing key)
+ * instead of directly from a queue:
+ *
+ * {@code
+ * pipeline.apply(
+ *   
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE",
 "fanout", "QUEUE");
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue or
+ * exchange.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ 
URI.
+ *
+ * For instance, you can write to an exchange (providing the exchange type):
+ *
+ * {@code
+ * pipeline
+ *   .apply(...) // provide PCollection
+ *   
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE",
 "fanout"));
+ * }
+ *
+ * For instance, you can write to a queue:
+ *
+ * {@code
+ * pipeline
+ *   .apply(...) // provide PCollection
+ *   
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setQueueDeclare(false)
+.setMaxReadTime(null)
+.setMaxNumRecords(Long.MAX_VALUE)
+.setUseCorrelationId(false)
+.build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125185
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203835879
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,655 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.MessageProperties;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}) wrapped as {@link RabbitMqMessage}.
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect to a
+ * RabbitMQ broker. The following example illustrates various options for 
configuring the source,
+ * reading from the queue:
+ *
+ * {@code
+ * pipeline.apply(
+ *   
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
 
 Review comment:
   Missing ), and missing "PCollection messages = ..."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125185)
Time Spent: 8h 10m  (was: 8h)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125188
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837537
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
 ##
 @@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import com.rabbitmq.client.AMQP;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+/**
+ * It contains the message payload, and additional metadata like routing key 
or attributes. The main
+ * reason of this class is that AMQP.BasicProperties doesn't provide a 
serializable public API.
+ */
+public class RabbitMqMessage implements Serializable {
+
+  @Nullable private final String routingKey;
+  private final byte[] body;
+  private final String contentType;
+  private final String contentEncoding;
+  private final Map headers;
+  private final Integer deliveryMode;
+  private final Integer priority;
+  @Nullable private final String correlationId;
+  @Nullable private final String replyTo;
+  private final String expiration;
+  private final String messageId;
+  private final Date timestamp;
+  @Nullable private final String type;
+  @Nullable private final String userId;
+  @Nullable private final String appId;
+  @Nullable private final String clusterId;
+
+  public RabbitMqMessage(byte[] body) {
+this.body = body;
+routingKey = "";
+contentType = null;
+contentEncoding = null;
+headers = new HashMap<>();
+deliveryMode = 1;
+priority = 1;
+correlationId = null;
+replyTo = null;
+expiration = null;
+messageId = null;
+timestamp = new Date();
+type = null;
+userId = null;
+appId = null;
+clusterId = null;
+  }
+
+  public RabbitMqMessage(
+  String routingKey,
+  byte[] body,
+  String contentType,
+  String contentEncoding,
+  Map headers,
+  Integer deliveryMode,
+  Integer priority,
+  String correlationId,
+  String replyTo,
+  String expiration,
+  String messageId,
+  Date timestamp,
+  String type,
+  String userId,
+  String appId,
+  String clusterId) {
+this.routingKey = routingKey;
+this.body = body;
+this.contentType = contentType;
+this.contentEncoding = contentEncoding;
+this.headers = headers;
+this.deliveryMode = deliveryMode;
+this.priority = priority;
+this.correlationId = correlationId;
+this.replyTo = replyTo;
+this.expiration = expiration;
+this.messageId = messageId;
+this.timestamp = timestamp;
+this.type = type;
+this.userId = userId;
+this.appId = appId;
+this.clusterId = clusterId;
+  }
+
+  public String getRoutingKey() {
+return routingKey;
+  }
+
+  public byte[] getBody() {
+return body;
+  }
+
+  public String getContentType() {
+return contentType;
+  }
+
+  public String getContentEncoding() {
+return contentEncoding;
+  }
+
+  public Map getHeaders() {
+return headers;
+  }
+
+  public Integer getDeliveryMode() {
+return deliveryMode;
+  }
+
+  public Integer getPriority() {
+return priority;
+  }
+
+  public String getCorrelationId() {
+return correlationId;
+  }
+
+  public String getReplyTo() {
+return replyTo;
+  }
+
+  public String getExpiration() {
+return expiration;
+  }
+
+  public String getMessageId() {
+return messageId;
+  }
+
+  public Date getTimestamp() {
+return timestamp;
+  }
+
+  public String getType() {
+return type;
+  }
+
+  public String getUserId() {
+return userId;
+  }
+
+  public String getAppId() 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125197
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837119
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
 ##
 @@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import com.rabbitmq.client.AMQP;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+/**
+ * It contains the message payload, and additional metadata like routing key 
or attributes. The main
+ * reason of this class is that AMQP.BasicProperties doesn't provide a 
serializable public API.
+ */
+public class RabbitMqMessage implements Serializable {
+
+  @Nullable private final String routingKey;
+  private final byte[] body;
+  private final String contentType;
+  private final String contentEncoding;
+  private final Map headers;
+  private final Integer deliveryMode;
+  private final Integer priority;
+  @Nullable private final String correlationId;
+  @Nullable private final String replyTo;
+  private final String expiration;
+  private final String messageId;
+  private final Date timestamp;
+  @Nullable private final String type;
+  @Nullable private final String userId;
+  @Nullable private final String appId;
+  @Nullable private final String clusterId;
+
+  public RabbitMqMessage(byte[] body) {
+this.body = body;
+routingKey = "";
+contentType = null;
+contentEncoding = null;
+headers = new HashMap<>();
+deliveryMode = 1;
+priority = 1;
+correlationId = null;
+replyTo = null;
+expiration = null;
+messageId = null;
+timestamp = new Date();
+type = null;
+userId = null;
+appId = null;
+clusterId = null;
+  }
+
+  public RabbitMqMessage(
+  String routingKey,
+  byte[] body,
+  String contentType,
+  String contentEncoding,
+  Map headers,
+  Integer deliveryMode,
+  Integer priority,
+  String correlationId,
+  String replyTo,
+  String expiration,
+  String messageId,
+  Date timestamp,
+  String type,
+  String userId,
+  String appId,
+  String clusterId) {
+this.routingKey = routingKey;
+this.body = body;
+this.contentType = contentType;
+this.contentEncoding = contentEncoding;
+this.headers = headers;
+this.deliveryMode = deliveryMode;
+this.priority = priority;
+this.correlationId = correlationId;
+this.replyTo = replyTo;
+this.expiration = expiration;
+this.messageId = messageId;
+this.timestamp = timestamp;
+this.type = type;
+this.userId = userId;
+this.appId = appId;
+this.clusterId = clusterId;
+  }
+
+  public String getRoutingKey() {
+return routingKey;
+  }
+
+  public byte[] getBody() {
+return body;
+  }
+
+  public String getContentType() {
+return contentType;
+  }
+
+  public String getContentEncoding() {
+return contentEncoding;
+  }
+
+  public Map getHeaders() {
+return headers;
+  }
+
+  public Integer getDeliveryMode() {
+return deliveryMode;
+  }
+
+  public Integer getPriority() {
+return priority;
+  }
+
+  public String getCorrelationId() {
+return correlationId;
+  }
+
+  public String getReplyTo() {
+return replyTo;
+  }
+
+  public String getExpiration() {
+return expiration;
+  }
+
+  public String getMessageId() {
+return messageId;
+  }
+
+  public Date getTimestamp() {
+return timestamp;
+  }
+
+  public String getType() {
+return type;
+  }
+
+  public String getUserId() {
+return userId;
+  }
+
+  public String getAppId() 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125189
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837965
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
 ##
 @@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.Consumer;
+import com.rabbitmq.client.DefaultConsumer;
+import com.rabbitmq.client.Envelope;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.ServerSocket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.qpid.server.Broker;
+import org.apache.qpid.server.BrokerOptions;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Test of {@link RabbitMqIO}. */
+@RunWith(JUnit4.class)
+public class RabbitMqIOTest implements Serializable {
+  private static final Logger LOG = 
LoggerFactory.getLogger(RabbitMqIOTest.class);
+
+  private static int port;
+  @ClassRule public static TemporaryFolder temporaryFolder = new 
TemporaryFolder();
+
+  @Rule public transient TestPipeline p = TestPipeline.create();
+
+  private static transient Broker broker;
+
+  @BeforeClass
+  public static void startBroker() throws Exception {
+try (ServerSocket serverSocket = new ServerSocket(0)) {
+  port = serverSocket.getLocalPort();
+}
+
+System.setProperty("derby.stream.error.field", "MyApp.DEV_NULL");
+broker = new Broker();
+BrokerOptions options = new BrokerOptions();
+options.setConfigProperty(BrokerOptions.QPID_AMQP_PORT, 
String.valueOf(port));
+options.setConfigProperty(BrokerOptions.QPID_WORK_DIR, 
temporaryFolder.newFolder().toString());
+options.setConfigProperty(BrokerOptions.QPID_HOME_DIR, "src/test/qpid");
+broker.startup(options);
+  }
+
+  @AfterClass
+  public static void stopBroker() {
+broker.shutdown();
+  }
+
+  @Test
+  public void testReadQueue() throws Exception {
+final int maxNumRecords = 10;
+PCollection raw =
+p.apply(
+RabbitMqIO.read()
+.withUri("amqp://guest:guest@localhost:" + port)
+.withQueue("READ")
+.withMaxNumRecords(maxNumRecords));
+PCollection output = raw.apply(ParDo.of(new ConverterFn()));
+
+List records = generateRecords(maxNumRecords);
+PAssert.that(output).containsInAnyOrder(records);
+
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri("amqp://guest:guest@localhost:" + port);
+Connection connection = connectionFactory.newConnection();
+Channel channel = connection.createChannel();
+channel.queueDeclare("READ", false, false, false, null);
+for (byte[] record : records) {
+  channel.basicPublish("", "READ", null, record);
+}
+
+p.run();
+
+channel.close();
 
 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125191
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203836690
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,530 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ 
URI.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder().setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new 
AutoValue_RabbitMqIO_Write.Builder().setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  private static ConnectionFactory createConnectionFactory(String uri) throws 
URISyntaxException,
+  NoSuchAlgorithmException, KeyManagementException {
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri(uri);
+connectionFactory.setAutomaticRecoveryEnabled(true);
+connectionFactory.setConnectionTimeout(6);
+connectionFactory.setNetworkRecoveryInterval(5000);
+connectionFactory.setRequestedHeartbeat(60);
+connectionFactory.setTopologyRecoveryEnabled(true);
+connectionFactory.setRequestedChannelMax(0);
+connectionFactory.setRequestedFrameMax(0);
+return connectionFactory;
+  }
+
+  /**
+   

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125187
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203836361
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,530 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ 
URI.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder().setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new 
AutoValue_RabbitMqIO_Write.Builder().setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  private static ConnectionFactory createConnectionFactory(String uri) throws 
URISyntaxException,
+  NoSuchAlgorithmException, KeyManagementException {
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri(uri);
+connectionFactory.setAutomaticRecoveryEnabled(true);
+connectionFactory.setConnectionTimeout(6);
+connectionFactory.setNetworkRecoveryInterval(5000);
+connectionFactory.setRequestedHeartbeat(60);
+connectionFactory.setTopologyRecoveryEnabled(true);
+connectionFactory.setRequestedChannelMax(0);
+connectionFactory.setRequestedFrameMax(0);
+return connectionFactory;
+  }
+
+  /**
+   

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125196
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837888
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
 ##
 @@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.Consumer;
+import com.rabbitmq.client.DefaultConsumer;
+import com.rabbitmq.client.Envelope;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.ServerSocket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.qpid.server.Broker;
+import org.apache.qpid.server.BrokerOptions;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Test of {@link RabbitMqIO}. */
+@RunWith(JUnit4.class)
+public class RabbitMqIOTest implements Serializable {
+  private static final Logger LOG = 
LoggerFactory.getLogger(RabbitMqIOTest.class);
+
+  private static int port;
+  @ClassRule public static TemporaryFolder temporaryFolder = new 
TemporaryFolder();
+
+  @Rule public transient TestPipeline p = TestPipeline.create();
+
+  private static transient Broker broker;
+
+  @BeforeClass
+  public static void startBroker() throws Exception {
+try (ServerSocket serverSocket = new ServerSocket(0)) {
+  port = serverSocket.getLocalPort();
+}
+
+System.setProperty("derby.stream.error.field", "MyApp.DEV_NULL");
+broker = new Broker();
+BrokerOptions options = new BrokerOptions();
+options.setConfigProperty(BrokerOptions.QPID_AMQP_PORT, 
String.valueOf(port));
+options.setConfigProperty(BrokerOptions.QPID_WORK_DIR, 
temporaryFolder.newFolder().toString());
+options.setConfigProperty(BrokerOptions.QPID_HOME_DIR, "src/test/qpid");
+broker.startup(options);
+  }
+
+  @AfterClass
+  public static void stopBroker() {
+broker.shutdown();
+  }
+
+  @Test
+  public void testReadQueue() throws Exception {
 
 Review comment:
   Please make sure that tests cover all combinations of what to declare and 
what to not declare


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125196)
Time Spent: 9h  (was: 8h 50m)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125195=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125195
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203838246
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
 ##
 @@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.Consumer;
+import com.rabbitmq.client.DefaultConsumer;
+import com.rabbitmq.client.Envelope;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.ServerSocket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.qpid.server.Broker;
+import org.apache.qpid.server.BrokerOptions;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Test of {@link RabbitMqIO}. */
+@RunWith(JUnit4.class)
+public class RabbitMqIOTest implements Serializable {
+  private static final Logger LOG = 
LoggerFactory.getLogger(RabbitMqIOTest.class);
+
+  private static int port;
+  @ClassRule public static TemporaryFolder temporaryFolder = new 
TemporaryFolder();
+
+  @Rule public transient TestPipeline p = TestPipeline.create();
+
+  private static transient Broker broker;
+
+  @BeforeClass
+  public static void startBroker() throws Exception {
+try (ServerSocket serverSocket = new ServerSocket(0)) {
+  port = serverSocket.getLocalPort();
+}
+
+System.setProperty("derby.stream.error.field", "MyApp.DEV_NULL");
+broker = new Broker();
+BrokerOptions options = new BrokerOptions();
+options.setConfigProperty(BrokerOptions.QPID_AMQP_PORT, 
String.valueOf(port));
+options.setConfigProperty(BrokerOptions.QPID_WORK_DIR, 
temporaryFolder.newFolder().toString());
+options.setConfigProperty(BrokerOptions.QPID_HOME_DIR, "src/test/qpid");
+broker.startup(options);
+  }
+
+  @AfterClass
+  public static void stopBroker() {
+broker.shutdown();
+  }
+
+  @Test
+  public void testReadQueue() throws Exception {
+final int maxNumRecords = 10;
+PCollection raw =
+p.apply(
+RabbitMqIO.read()
+.withUri("amqp://guest:guest@localhost:" + port)
+.withQueue("READ")
+.withMaxNumRecords(maxNumRecords));
+PCollection output = raw.apply(ParDo.of(new ConverterFn()));
+
+List records = generateRecords(maxNumRecords);
+PAssert.that(output).containsInAnyOrder(records);
+
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri("amqp://guest:guest@localhost:" + port);
+Connection connection = connectionFactory.newConnection();
+Channel channel = connection.createChannel();
+channel.queueDeclare("READ", false, false, false, null);
+for (byte[] record : records) {
+  channel.basicPublish("", "READ", null, record);
+}
+
+p.run();
+
+channel.close();
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125184
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203835977
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,655 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.MessageProperties;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}) wrapped as {@link RabbitMqMessage}.
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect to a
+ * RabbitMQ broker. The following example illustrates various options for 
configuring the source,
+ * reading from the queue:
+ *
+ * {@code
+ * pipeline.apply(
+ *   
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * It's also possible to read from an exchange (providing the exchange type 
and routing key)
+ * instead of directly from a queue:
+ *
+ * {@code
+ * pipeline.apply(
+ *   
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withExchange("EXCHANGE",
 "fanout", "QUEUE");
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue or
 
 Review comment:
   It's actually RabbitMqMessage


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125184)
Time Spent: 8h 10m  (was: 8h)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125186
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203836628
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,530 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ 
URI.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder().setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new 
AutoValue_RabbitMqIO_Write.Builder().setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  private static ConnectionFactory createConnectionFactory(String uri) throws 
URISyntaxException,
+  NoSuchAlgorithmException, KeyManagementException {
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri(uri);
+connectionFactory.setAutomaticRecoveryEnabled(true);
+connectionFactory.setConnectionTimeout(6);
+connectionFactory.setNetworkRecoveryInterval(5000);
+connectionFactory.setRequestedHeartbeat(60);
+connectionFactory.setTopologyRecoveryEnabled(true);
+connectionFactory.setRequestedChannelMax(0);
+connectionFactory.setRequestedFrameMax(0);
+return connectionFactory;
+  }
+
+  /**
+   

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125190
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203838287
 
 

 ##
 File path: sdks/java/io/rabbitmq/src/test/qpid/etc/passwd
 ##
 @@ -0,0 +1,19 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+guest:guest
 
 Review comment:
   Ping?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 125190)

> Create RabbitMqIO
> -
>
> Key: BEAM-1240
> URL: https://issues.apache.org/jira/browse/BEAM-1240
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125194
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203838041
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIOTest.java
 ##
 @@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.Consumer;
+import com.rabbitmq.client.DefaultConsumer;
+import com.rabbitmq.client.Envelope;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.ServerSocket;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.qpid.server.Broker;
+import org.apache.qpid.server.BrokerOptions;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Test of {@link RabbitMqIO}. */
+@RunWith(JUnit4.class)
+public class RabbitMqIOTest implements Serializable {
+  private static final Logger LOG = 
LoggerFactory.getLogger(RabbitMqIOTest.class);
+
+  private static int port;
+  @ClassRule public static TemporaryFolder temporaryFolder = new 
TemporaryFolder();
+
+  @Rule public transient TestPipeline p = TestPipeline.create();
+
+  private static transient Broker broker;
+
+  @BeforeClass
+  public static void startBroker() throws Exception {
+try (ServerSocket serverSocket = new ServerSocket(0)) {
+  port = serverSocket.getLocalPort();
+}
+
+System.setProperty("derby.stream.error.field", "MyApp.DEV_NULL");
+broker = new Broker();
+BrokerOptions options = new BrokerOptions();
+options.setConfigProperty(BrokerOptions.QPID_AMQP_PORT, 
String.valueOf(port));
+options.setConfigProperty(BrokerOptions.QPID_WORK_DIR, 
temporaryFolder.newFolder().toString());
+options.setConfigProperty(BrokerOptions.QPID_HOME_DIR, "src/test/qpid");
+broker.startup(options);
+  }
+
+  @AfterClass
+  public static void stopBroker() {
+broker.shutdown();
+  }
+
+  @Test
+  public void testReadQueue() throws Exception {
+final int maxNumRecords = 10;
+PCollection raw =
+p.apply(
+RabbitMqIO.read()
+.withUri("amqp://guest:guest@localhost:" + port)
+.withQueue("READ")
+.withMaxNumRecords(maxNumRecords));
+PCollection output = raw.apply(ParDo.of(new ConverterFn()));
+
+List records = generateRecords(maxNumRecords);
+PAssert.that(output).containsInAnyOrder(records);
+
+ConnectionFactory connectionFactory = new ConnectionFactory();
+connectionFactory.setUri("amqp://guest:guest@localhost:" + port);
+Connection connection = connectionFactory.newConnection();
+Channel channel = connection.createChannel();
+channel.queueDeclare("READ", false, false, false, null);
+for (byte[] record : records) {
+  channel.basicPublish("", "READ", null, record);
+}
+
+p.run();
+
+channel.close();
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=125192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-125192
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 19/Jul/18 19:03
Start Date: 19/Jul/18 19:03
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r203837667
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqMessage.java
 ##
 @@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import com.rabbitmq.client.AMQP;
+import java.io.Serializable;
+import java.util.Arrays;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+/**
+ * It contains the message payload, and additional metadata like routing key 
or attributes. The main
+ * reason of this class is that AMQP.BasicProperties doesn't provide a 
serializable public API.
+ */
+public class RabbitMqMessage implements Serializable {
+
+  @Nullable private final String routingKey;
+  private final byte[] body;
+  private final String contentType;
+  private final String contentEncoding;
+  private final Map headers;
+  private final Integer deliveryMode;
+  private final Integer priority;
+  @Nullable private final String correlationId;
+  @Nullable private final String replyTo;
+  private final String expiration;
+  private final String messageId;
+  private final Date timestamp;
+  @Nullable private final String type;
+  @Nullable private final String userId;
+  @Nullable private final String appId;
+  @Nullable private final String clusterId;
+
+  public RabbitMqMessage(byte[] body) {
+this.body = body;
+routingKey = "";
+contentType = null;
+contentEncoding = null;
+headers = new HashMap<>();
+deliveryMode = 1;
+priority = 1;
+correlationId = null;
+replyTo = null;
+expiration = null;
+messageId = null;
+timestamp = new Date();
+type = null;
+userId = null;
+appId = null;
+clusterId = null;
+  }
+
+  public RabbitMqMessage(
+  String routingKey,
+  byte[] body,
+  String contentType,
+  String contentEncoding,
+  Map headers,
+  Integer deliveryMode,
+  Integer priority,
+  String correlationId,
+  String replyTo,
+  String expiration,
+  String messageId,
+  Date timestamp,
+  String type,
+  String userId,
+  String appId,
+  String clusterId) {
+this.routingKey = routingKey;
+this.body = body;
+this.contentType = contentType;
+this.contentEncoding = contentEncoding;
+this.headers = headers;
+this.deliveryMode = deliveryMode;
+this.priority = priority;
+this.correlationId = correlationId;
+this.replyTo = replyTo;
+this.expiration = expiration;
+this.messageId = messageId;
+this.timestamp = timestamp;
+this.type = type;
+this.userId = userId;
+this.appId = appId;
+this.clusterId = clusterId;
+  }
+
+  public String getRoutingKey() {
+return routingKey;
+  }
+
+  public byte[] getBody() {
+return body;
+  }
+
+  public String getContentType() {
+return contentType;
+  }
+
+  public String getContentEncoding() {
+return contentEncoding;
+  }
+
+  public Map getHeaders() {
+return headers;
+  }
+
+  public Integer getDeliveryMode() {
+return deliveryMode;
+  }
+
+  public Integer getPriority() {
+return priority;
+  }
+
+  public String getCorrelationId() {
+return correlationId;
+  }
+
+  public String getReplyTo() {
+return replyTo;
+  }
+
+  public String getExpiration() {
+return expiration;
+  }
+
+  public String getMessageId() {
+return messageId;
+  }
+
+  public Date getTimestamp() {
+return timestamp;
+  }
+
+  public String getType() {
+return type;
+  }
+
+  public String getUserId() {
+return userId;
+  }
+
+  public String getAppId() 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #449

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam2 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e33852830d863cfa9f9b3250e81afe29b8321d4b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e33852830d863cfa9f9b3250e81afe29b8321d4b
Commit message: "Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline"
 > git rev-list --no-walk 1fbc3b987f993a7213bd69819bdf5a5c54fc4314 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins8091110533067850984.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7717125748229345189.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7325959546513509886.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-mongodbio-it-449
namespace "beam-performancetests-mongodbio-it-449" created
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins6842095145454903280.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-mongodbio-it-449
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins481908923673703271.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins3250469563328707121.sh
+ rm -rf .env
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins5837425656246877851.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7371736748561728820.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins8249338997648713416.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins531211850693316144.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1078

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #752

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e33852830d863cfa9f9b3250e81afe29b8321d4b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e33852830d863cfa9f9b3250e81afe29b8321d4b
Commit message: "Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline"
 > git rev-list --no-walk 1fbc3b987f993a7213bd69819bdf5a5c54fc4314 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1815586450888369607.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8545699707963512782.sh
+ rm -rf .env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8412662107654149802.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7116992781802622791.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4113336085189813718.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6625231376595414956.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #765

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e33852830d863cfa9f9b3250e81afe29b8321d4b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e33852830d863cfa9f9b3250e81afe29b8321d4b
Commit message: "Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline"
 > git rev-list --no-walk 1fbc3b987f993a7213bd69819bdf5a5c54fc4314 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5430521271856530072.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins220225347310581709.sh
+ rm -rf .env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins2575051735818635743.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins7218047212082477913.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins912844381052611321.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins6355436901666008270.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #258

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e33852830d863cfa9f9b3250e81afe29b8321d4b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e33852830d863cfa9f9b3250e81afe29b8321d4b
Commit message: "Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline"
 > git rev-list --no-walk 1fbc3b987f993a7213bd69819bdf5a5c54fc4314 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5597310300395791182.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins7632728881600130757.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5959365271081381619.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins4400111743792525691.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5214600086597277256.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5128670134060606741.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.10)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (40.0.0)
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached 
https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
Collecting 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT_HDFS #153

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e33852830d863cfa9f9b3250e81afe29b8321d4b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e33852830d863cfa9f9b3250e81afe29b8321d4b
Commit message: "Merge pull request #5969 from 
amaliujia/rui_wang-add_unittest_for_beamsqlline"
 > git rev-list --no-walk 1fbc3b987f993a7213bd69819bdf5a5c54fc4314 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3795933192536137419.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7057324263125098461.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8206841584541608783.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-parquetioit-hdfs-153
namespace "beam-performancetests-parquetioit-hdfs-153" created
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6103206496858256020.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-parquetioit-hdfs-153
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7211273549699133587.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5919878642537197378.sh
+ rm -rf .env
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7755903010869596176.sh
+ virtualenv .env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins189402290423808.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.1)
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7110255995375017453.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6711785854638188388.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 

Build failed in Jenkins: beam_PreCommit_Java_Cron #123

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-4806] improve unit tests for SqlLineMain

[github] Add version bound to numpy.

[jxlewis] Updated documentation, added BEAM-4828 to In-Progress

[github] Fix indentation.

[github] SQS -> Amazon SQS

--
[...truncated 16.47 MB...]
INFO: 2018-07-19T18:19:01.567Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.615Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:364#1d275f544daf228c
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.658Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.706Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.750Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.793Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.828Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.877Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.917Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.965Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:01.991Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:02.032Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 19, 2018 6:19:07 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-19T18:19:02.061Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into