[jira] [Work logged] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?focusedWorklogId=153768=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153768
 ]

ASF GitHub Bot logged work on BEAM-5627:


Author: ASF GitHub Bot
Created on: 12/Oct/18 05:47
Start Date: 12/Oct/18 05:47
Worklog Time Spent: 10m 
  Work Description: HuangLED opened a new pull request #6671: [BEAM-5627] 
Fix sources test for py3.
URL: https://github.com/apache/beam/pull/6671
 
 
   This fixes one of the (many) cases with str/byte issue in py3. 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153768)
Time Spent: 2h 10m  (was: 2h)

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627

[jira] [Comment Edited] (BEAM-5551) Beam Dependency Update Request: org.elasticsearch:elasticsearch-hadoop

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647497#comment-16647497
 ] 

yifan zou edited comment on BEAM-5551 at 10/12/18 5:30 AM:
---

Could we close this issue if we are not able to update this library to the 
version 6? Or, do we plan to we upgrade it to the latest 5.*?


was (Author: yifanzou):
Could we close this issue if we are not able to update the library to the 
version 6?

> Beam Dependency Update Request: org.elasticsearch:elasticsearch-hadoop
> --
>
> Key: BEAM-5551
> URL: https://issues.apache.org/jira/browse/BEAM-5551
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2018-10-01 19:30:14.876535 
> -
> Please consider upgrading the dependency 
> org.elasticsearch:elasticsearch-hadoop. 
> The current version is 5.0.0. The latest version is 6.4.1 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:54.276690 
> -
> Please consider upgrading the dependency 
> org.elasticsearch:elasticsearch-hadoop. 
> The current version is 5.0.0. The latest version is 6.4.2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5551) Beam Dependency Update Request: org.elasticsearch:elasticsearch-hadoop

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647497#comment-16647497
 ] 

yifan zou commented on BEAM-5551:
-

Could we close this issue if we are not able to update the library to the 
version 6?

> Beam Dependency Update Request: org.elasticsearch:elasticsearch-hadoop
> --
>
> Key: BEAM-5551
> URL: https://issues.apache.org/jira/browse/BEAM-5551
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2018-10-01 19:30:14.876535 
> -
> Please consider upgrading the dependency 
> org.elasticsearch:elasticsearch-hadoop. 
> The current version is 5.0.0. The latest version is 6.4.1 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:54.276690 
> -
> Please consider upgrading the dependency 
> org.elasticsearch:elasticsearch-hadoop. 
> The current version is 5.0.0. The latest version is 6.4.2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5550) Beam Dependency Update Request: de.flapdoodle.embed:de.flapdoodle.embed.process

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5550.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: 
> de.flapdoodle.embed:de.flapdoodle.embed.process
> ---
>
> Key: BEAM-5550
> URL: https://issues.apache.org/jira/browse/BEAM-5550
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:12.660362 
> -
> Please consider upgrading the dependency 
> de.flapdoodle.embed:de.flapdoodle.embed.process. 
> The current version is 1.50.1. The latest version is 2.0.5 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:48.458620 
> -
> Please consider upgrading the dependency 
> de.flapdoodle.embed:de.flapdoodle.embed.process. 
> The current version is 1.50.1. The latest version is 2.0.5 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5548) Beam Dependency Update Request: com.typesafe:config

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5548.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: com.typesafe:config
> ---
>
> Key: BEAM-5548
> URL: https://issues.apache.org/jira/browse/BEAM-5548
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:09.147440 
> -
> Please consider upgrading the dependency com.typesafe:config. 
> The current version is 1.3.0. The latest version is 1.3.3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:42.280247 
> -
> Please consider upgrading the dependency com.typesafe:config. 
> The current version is 1.3.0. The latest version is 1.3.3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5546) Beam Dependency Update Request: commons-codec:commons-codec

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5546.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: commons-codec:commons-codec
> ---
>
> Key: BEAM-5546
> URL: https://issues.apache.org/jira/browse/BEAM-5546
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:05.437342 
> -
> Please consider upgrading the dependency commons-codec:commons-codec. 
> The current version is 1.9. The latest version is 1.11 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:29.372797 
> -
> Please consider upgrading the dependency commons-codec:commons-codec. 
> The current version is 1.9. The latest version is 1.11 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5545) Beam Dependency Update Request: commons-cli:commons-cli

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5545.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: commons-cli:commons-cli
> ---
>
> Key: BEAM-5545
> URL: https://issues.apache.org/jira/browse/BEAM-5545
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:03.552609 
> -
> Please consider upgrading the dependency commons-cli:commons-cli. 
> The current version is 1.2. The latest version is 1.4 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:24.012471 
> -
> Please consider upgrading the dependency commons-cli:commons-cli. 
> The current version is 1.2. The latest version is 1.4 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5547) Beam Dependency Update Request: org.apache.commons:commons-dbcp2

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5547.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: org.apache.commons:commons-dbcp2
> 
>
> Key: BEAM-5547
> URL: https://issues.apache.org/jira/browse/BEAM-5547
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:07.291455 
> -
> Please consider upgrading the dependency 
> org.apache.commons:commons-dbcp2. 
> The current version is 2.1.1. The latest version is 2.5.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:36.975443 
> -
> Please consider upgrading the dependency 
> org.apache.commons:commons-dbcp2. 
> The current version is 2.1.1. The latest version is 2.5.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5544) Beam Dependency Update Request: org.apache.cassandra:cassandra-all

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5544.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: org.apache.cassandra:cassandra-all
> --
>
> Key: BEAM-5544
> URL: https://issues.apache.org/jira/browse/BEAM-5544
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:01.676519 
> -
> Please consider upgrading the dependency 
> org.apache.cassandra:cassandra-all. 
> The current version is 3.9. The latest version is 3.11.3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:18.366388 
> -
> Please consider upgrading the dependency 
> org.apache.cassandra:cassandra-all. 
> The current version is 3.9. The latest version is 3.11.3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5542) Beam Dependency Update Request: biz.aQute:bndlib

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5542.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: biz.aQute:bndlib
> 
>
> Key: BEAM-5542
> URL: https://issues.apache.org/jira/browse/BEAM-5542
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:29:39.227364 
> -
> Please consider upgrading the dependency biz.aQute:bndlib. 
> The current version is 1.43.0. The latest version is 
> 2.0.0.20130123-133441 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:16:04.679328 
> -
> Please consider upgrading the dependency biz.aQute:bndlib. 
> The current version is 1.43.0. The latest version is 
> 2.0.0.20130123-133441 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5552) Beam Dependency Update Request: net.ltgt.gradle:gradle-apt-plugin

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5552.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: net.ltgt.gradle:gradle-apt-plugin
> -
>
> Key: BEAM-5552
> URL: https://issues.apache.org/jira/browse/BEAM-5552
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:23.446392 
> -
> Please consider upgrading the dependency 
> net.ltgt.gradle:gradle-apt-plugin. 
> The current version is 0.13. The latest version is 0.18 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:17:06.730239 
> -
> Please consider upgrading the dependency 
> net.ltgt.gradle:gradle-apt-plugin. 
> The current version is 0.13. The latest version is 0.18 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5554) Beam Dependency Update Request: gradle.plugin.com.palantir.gradle.docker:gradle-docker

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5554.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: 
> gradle.plugin.com.palantir.gradle.docker:gradle-docker
> --
>
> Key: BEAM-5554
> URL: https://issues.apache.org/jira/browse/BEAM-5554
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:32.134613 
> -
> Please consider upgrading the dependency 
> gradle.plugin.com.palantir.gradle.docker:gradle-docker. 
> The current version is 0.13.0. The latest version is 0.20.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:17:18.449825 
> -
> Please consider upgrading the dependency 
> gradle.plugin.com.palantir.gradle.docker:gradle-docker. 
> The current version is 0.13.0. The latest version is 0.20.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5559) Beam Dependency Update Request: com.google.guava:guava

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5559.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: com.google.guava:guava
> --
>
> Key: BEAM-5559
> URL: https://issues.apache.org/jira/browse/BEAM-5559
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:53.471497 
> -
> Please consider upgrading the dependency com.google.guava:guava. 
> The current version is 20.0. The latest version is 26.0-jre 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:18:05.174889 
> -
> Please consider upgrading the dependency com.google.guava:guava. 
> The current version is 20.0. The latest version is 26.0-jre 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5556) Beam Dependency Update Request: com.github.ben-manes:gradle-versions-plugin

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5556.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: com.github.ben-manes:gradle-versions-plugin
> ---
>
> Key: BEAM-5556
> URL: https://issues.apache.org/jira/browse/BEAM-5556
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:46.570372 
> -
> Please consider upgrading the dependency 
> com.github.ben-manes:gradle-versions-plugin. 
> The current version is 0.17.0. The latest version is 0.20.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:17:36.872262 
> -
> Please consider upgrading the dependency 
> com.github.ben-manes:gradle-versions-plugin. 
> The current version is 0.17.0. The latest version is 0.20.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5580) Beam Dependency Update Request: io.opencensus:opencensus-api

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5580.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: io.opencensus:opencensus-api
> 
>
> Key: BEAM-5580
> URL: https://issues.apache.org/jira/browse/BEAM-5580
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:59.099848 
> -
> Please consider upgrading the dependency 
> io.opencensus:opencensus-api. 
> The current version is 0.12.3. The latest version is 0.16.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:07.582558 
> -
> Please consider upgrading the dependency 
> io.opencensus:opencensus-api. 
> The current version is 0.12.3. The latest version is 0.16.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5581) Beam Dependency Update Request: io.opencensus:opencensus-contrib-grpc-metrics

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5581.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: io.opencensus:opencensus-contrib-grpc-metrics
> -
>
> Key: BEAM-5581
> URL: https://issues.apache.org/jira/browse/BEAM-5581
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:00.894091 
> -
> Please consider upgrading the dependency 
> io.opencensus:opencensus-contrib-grpc-metrics. 
> The current version is 0.12.3. The latest version is 0.16.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:13.127279 
> -
> Please consider upgrading the dependency 
> io.opencensus:opencensus-contrib-grpc-metrics. 
> The current version is 0.12.3. The latest version is 0.16.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5586) Beam Dependency Update Request: com.github.jengelman.gradle.plugins:shadow

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5586.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: com.github.jengelman.gradle.plugins:shadow
> --
>
> Key: BEAM-5586
> URL: https://issues.apache.org/jira/browse/BEAM-5586
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:24.176205 
> -
> Please consider upgrading the dependency 
> com.github.jengelman.gradle.plugins:shadow. 
> The current version is 2.0.4. The latest version is 4.0.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:48.837740 
> -
> Please consider upgrading the dependency 
> com.github.jengelman.gradle.plugins:shadow. 
> The current version is 2.0.4. The latest version is 4.0.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5583) Beam Dependency Update Request: com.carrotsearch.randomizedtesting:randomizedtesting-runner

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5583.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: 
> com.carrotsearch.randomizedtesting:randomizedtesting-runner
> ---
>
> Key: BEAM-5583
> URL: https://issues.apache.org/jira/browse/BEAM-5583
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:16.147096 
> -
> Please consider upgrading the dependency 
> com.carrotsearch.randomizedtesting:randomizedtesting-runner. 
> The current version is 2.5.2. The latest version is 2.7.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:35.518999 
> -
> Please consider upgrading the dependency 
> com.carrotsearch.randomizedtesting:randomizedtesting-runner. 
> The current version is 2.5.2. The latest version is 2.7.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5594) Beam Dependency Update Request: com.clearspring.analytics:stream

[Beam JIRA Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot closed BEAM-5594.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> Beam Dependency Update Request: com.clearspring.analytics:stream
> 
>
> Key: BEAM-5594
> URL: https://issues.apache.org/jira/browse/BEAM-5594
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:39.356598 
> -
> Please consider upgrading the dependency 
> com.clearspring.analytics:stream. 
> The current version is 2.9.5. The latest version is 2.9.6 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:21:28.362451 
> -
> Please consider upgrading the dependency 
> com.clearspring.analytics:stream. 
> The current version is 2.9.5. The latest version is 2.9.6 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5595) Beam Dependency Update Request: org.elasticsearch.client:transport

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647488#comment-16647488
 ] 

yifan zou commented on BEAM-5595:
-

I believe we are not going to update elasticsearch* libraries to version 6. If 
so, we could close this issue so that the dependency tool will stop updating 
this issue. 

> Beam Dependency Update Request: org.elasticsearch.client:transport
> --
>
> Key: BEAM-5595
> URL: https://issues.apache.org/jira/browse/BEAM-5595
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2018-10-01 19:32:41.617633 
> -
> Please consider upgrading the dependency 
> org.elasticsearch.client:transport. 
> The current version is 5.0.0. The latest version is 6.4.1 
> cc: [~echauchot], [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:21:34.320882 
> -
> Please consider upgrading the dependency 
> org.elasticsearch.client:transport. 
> The current version is 5.0.0. The latest version is 6.4.2 
> cc: [~echauchot], [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5708) Support caching of SDKHarness environments in flink

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5708?focusedWorklogId=153765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153765
 ]

ASF GitHub Bot logged work on BEAM-5708:


Author: ASF GitHub Bot
Created on: 12/Oct/18 05:14
Start Date: 12/Oct/18 05:14
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6638: 
[BEAM-5708] Cache environment in portable flink runner
URL: https://github.com/apache/beam/pull/6638#discussion_r224672532
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
 ##
 @@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
 WrappedContext wrapper = getCache().get(jobInfo.jobId());
 Preconditions.checkState(
 wrapper != null, "Releasing context for unknown job: " + 
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to 
the classloader not being
-// available anymore after the tasks have been removed from the execution 
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+  // Schedule task to clean the container later.
+  // Ensure that this class is loaded in the parent Flink classloader.
+  getExecutor()
+  .schedule(() -> release(wrapper), environmentCacheTTLMillis, 
TimeUnit.MILLISECONDS);
 
 Review comment:
   It isn't possible to defer this when the class is loaded by the user class 
loader. So we would probably need to compare the class loader that loaded the 
flink runtime with that of the beam class?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153765)
Time Spent: 1h  (was: 50m)

> Support caching of SDKHarness environments in flink
> ---
>
> Key: BEAM-5708
> URL: https://issues.apache.org/jira/browse/BEAM-5708
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Cache and reuse environment to improve performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5584) Beam Dependency Update Request: org.scala-lang:scala-library

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5584.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Close this issue. Not going to upgrade 2.13.0-M5 version.

> Beam Dependency Update Request: org.scala-lang:scala-library
> 
>
> Key: BEAM-5584
> URL: https://issues.apache.org/jira/browse/BEAM-5584
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:18.036529 
> -
> Please consider upgrading the dependency 
> org.scala-lang:scala-library. 
> The current version is 2.11.8. The latest version is 2.13.0-M5 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:41.075422 
> -
> Please consider upgrading the dependency 
> org.scala-lang:scala-library. 
> The current version is 2.11.8. The latest version is 2.13.0-M5 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153764
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 12/Oct/18 05:10
Start Date: 12/Oct/18 05:10
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6592: 
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224672134
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
 ##
 @@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
 WrappedContext wrapper = getCache().get(jobInfo.jobId());
 Preconditions.checkState(
 wrapper != null, "Releasing context for unknown job: " + 
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to 
the classloader not being
-// available anymore after the tasks have been removed from the execution 
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+  // Schedule task to clean the container later.
+  // Ensure that this class is loaded in the parent Flink classloader.
+  getExecutor()
+  .schedule(() -> release(wrapper), environmentCacheTTLMillis, 
TimeUnit.MILLISECONDS);
 
 Review comment:
   Yes, that's one of those situations where 2 separate commits in the same PR 
make sense. But since you have the other PR under review already, why not wrap 
that up first.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153764)
Time Spent: 32h 50m  (was: 32h 40m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 32h 50m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5588) Beam Dependency Update Request: org.slf4j:slf4j-jdk14

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5588.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Close this issue. Not going to upgrade to a beta version.

> Beam Dependency Update Request: org.slf4j:slf4j-jdk14
> -
>
> Key: BEAM-5588
> URL: https://issues.apache.org/jira/browse/BEAM-5588
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:27.843922 
> -
> Please consider upgrading the dependency org.slf4j:slf4j-jdk14. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: [~swegner], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:59.649783 
> -
> Please consider upgrading the dependency org.slf4j:slf4j-jdk14. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: [~swegner], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5587) Beam Dependency Update Request: org.slf4j:slf4j-api

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5587.
---
   Resolution: Fixed
Fix Version/s: Not applicable

Close this issue. Not going to upgrade to a beta version.

> Beam Dependency Update Request: org.slf4j:slf4j-api
> ---
>
> Key: BEAM-5587
> URL: https://issues.apache.org/jira/browse/BEAM-5587
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:32:26.011070 
> -
> Please consider upgrading the dependency org.slf4j:slf4j-api. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:54.196725 
> -
> Please consider upgrading the dependency org.slf4j:slf4j-api. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5578) Beam Dependency Update Request: io.netty:netty-all

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5578.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Close this issue. Not going to upgrade to an alpha version.

> Beam Dependency Update Request: io.netty:netty-all
> --
>
> Key: BEAM-5578
> URL: https://issues.apache.org/jira/browse/BEAM-5578
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:55.165644 
> -
> Please consider upgrading the dependency io.netty:netty-all. 
> The current version is 4.1.17.Final. The latest version is 
> 5.0.0.Alpha2 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:19:56.620627 
> -
> Please consider upgrading the dependency io.netty:netty-all. 
> The current version is 4.1.17.Final. The latest version is 
> 5.0.0.Alpha2 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5579) Beam Dependency Update Request: io.netty:netty-transport-native-epoll

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5579.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Close this issue. Not going to upgrade to an alpha version.

> Beam Dependency Update Request: io.netty:netty-transport-native-epoll
> -
>
> Key: BEAM-5579
> URL: https://issues.apache.org/jira/browse/BEAM-5579
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:57.170262 
> -
> Please consider upgrading the dependency 
> io.netty:netty-transport-native-epoll. 
> The current version is 4.1.25.Final. The latest version is 
> 5.0.0.Alpha2 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:20:02.153629 
> -
> Please consider upgrading the dependency 
> io.netty:netty-transport-native-epoll. 
> The current version is 4.1.25.Final. The latest version is 
> 5.0.0.Alpha2 
> cc: [~chamikara], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5565) Beam Dependency Update Request: org.apache.hbase:hbase-shaded-server

[yifan zou (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647471#comment-16647471
 ] 

yifan zou edited comment on BEAM-5565 at 10/12/18 5:04 AM:
---

Close this issue. Not going to upgrade to an alpha version.


was (Author: yifanzou):
Not going to upgrade to an alpha version.

> Beam Dependency Update Request: org.apache.hbase:hbase-shaded-server
> 
>
> Key: BEAM-5565
> URL: https://issues.apache.org/jira/browse/BEAM-5565
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:10.853495 
> -
> Please consider upgrading the dependency 
> org.apache.hbase:hbase-shaded-server. 
> The current version is 1.2.6. The latest version is 2.0.0-alpha2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:18:38.881126 
> -
> Please consider upgrading the dependency 
> org.apache.hbase:hbase-shaded-server. 
> The current version is 1.2.6. The latest version is 2.0.0-alpha2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5571) Beam Dependency Update Request: org.slf4j:jcl-over-slf4j

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5571.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Close this issue. Not going to upgrade to a beta version.

> Beam Dependency Update Request: org.slf4j:jcl-over-slf4j
> 
>
> Key: BEAM-5571
> URL: https://issues.apache.org/jira/browse/BEAM-5571
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:41.671404 
> -
> Please consider upgrading the dependency org.slf4j:jcl-over-slf4j. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:19:22.531634 
> -
> Please consider upgrading the dependency org.slf4j:jcl-over-slf4j. 
> The current version is 1.7.25. The latest version is 1.8.0-beta2 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5565) Beam Dependency Update Request: org.apache.hbase:hbase-shaded-server

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5565.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Not going to upgrade to an alpha version.

> Beam Dependency Update Request: org.apache.hbase:hbase-shaded-server
> 
>
> Key: BEAM-5565
> URL: https://issues.apache.org/jira/browse/BEAM-5565
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:31:10.853495 
> -
> Please consider upgrading the dependency 
> org.apache.hbase:hbase-shaded-server. 
> The current version is 1.2.6. The latest version is 2.0.0-alpha2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:18:38.881126 
> -
> Please consider upgrading the dependency 
> org.apache.hbase:hbase-shaded-server. 
> The current version is 1.2.6. The latest version is 2.0.0-alpha2 
> cc: [~timrobertson100], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5557) Beam Dependency Update Request: org.codehaus.groovy:groovy-all

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-5557.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

Not going to upgrade to an alpha version.

> Beam Dependency Update Request: org.codehaus.groovy:groovy-all
> --
>
> Key: BEAM-5557
> URL: https://issues.apache.org/jira/browse/BEAM-5557
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: Not applicable
>
>
>  - 2018-10-01 19:30:48.367955 
> -
> Please consider upgrading the dependency 
> org.codehaus.groovy:groovy-all. 
> The current version is 2.4.13. The latest version is 3.0.0-alpha-3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:17:48.104127 
> -
> Please consider upgrading the dependency 
> org.codehaus.groovy:groovy-all. 
> The current version is 2.4.13. The latest version is 3.0.0-alpha-3 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5724) Beam creates too many sdk_worker processes with --sdk-worker-parallelism=stage

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-5724:
---
Labels: portability-flink  (was: )

> Beam creates too many sdk_worker processes with --sdk-worker-parallelism=stage
> --
>
> Key: BEAM-5724
> URL: https://issues.apache.org/jira/browse/BEAM-5724
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Micah Wylde
>Priority: Major
>  Labels: portability-flink
>
> In the flink portable runner, we currently support two options for sdk worker 
> parallelism (how many python worker processes we run). The default is one per 
> taskmanager, and with --sdk-worker-parallelism=stage you get one per stage. 
> However, for complex pipelines with many beam operators that get fused into a 
> single flink task this can produce hundreds of worker processes per TM.
> Flink uses the notion of task slots to limit resource utilization on a box; I 
> think that beam should try to respect those limits as well. I think ideally 
> we'd produce a single python worker per task slot/flink operator chain.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153761
 ]

ASF GitHub Bot logged work on BEAM-5637:


Author: ASF GitHub Bot
Created on: 12/Oct/18 04:15
Start Date: 12/Oct/18 04:15
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python 
support for custom dataflow worker jar
URL: https://github.com/apache/beam/pull/6667#issuecomment-429199020
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153761)
Time Spent: 40m  (was: 0.5h)

> Python support for custom dataflow worker jar
> -
>
> Key: BEAM-5637
> URL: https://issues.apache.org/jira/browse/BEAM-5637
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> One of the slightly subtle aspects is that we would need to ignore one of the 
> staged jars for portable Python jobs. That requires a change to the Python 
> boot code: 
> https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153759=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153759
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 12/Oct/18 04:08
Start Date: 12/Oct/18 04:08
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6642: [BEAM-5701] port 
datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#issuecomment-429198259
 
 
   @jasonkuster No worry, mis-configurations in my local environment. The 
datastore_write_it_test passed in the postcommit. The error is no relevant to 
this test. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153759)
Time Spent: 4h  (was: 3h 50m)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1252

[Apache Jenkins Server]

See 

[beam] branch master updated (50e02f3 -> 0e146da)

[scott]

This is an automated email from the ASF dual-hosted git repository.

scott pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 50e02f3  Merge pull request #6662 from pabloem/refresh-api
 add 3d11be6  Pull URL out of logs for build description
 new 0e146da  Merge pull request #6669: [BEAM-5672] Pull URL out of logs 
for build description

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #6669: [BEAM-5672] Pull URL out of logs for build description

[scott]

This is an automated email from the ASF dual-hosted git repository.

scott pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0e146dab981dcb8c88ccbc6505ba48583a551ed0
Merge: 50e02f3 3d11be6
Author: Scott Wegner 
AuthorDate: Thu Oct 11 23:02:19 2018 -0400

Merge pull request #6669: [BEAM-5672] Pull URL out of logs for build 
description

 .test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

Jenkins build is back to normal : beam_PostCommit_Python_VR_Flink #341

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5672) Staged URL should be posted to GitHub PR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5672?focusedWorklogId=153756=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153756
 ]

ASF GitHub Bot logged work on BEAM-5672:


Author: ASF GitHub Bot
Created on: 12/Oct/18 03:03
Start Date: 12/Oct/18 03:03
Worklog Time Spent: 10m 
  Work Description: swegner closed pull request #6669: [BEAM-5672] Pull URL 
out of logs for build description
URL: https://github.com/apache/beam/pull/6669
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy 
b/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
index 5f3f4e3bba7..c5cf28f2efa 100644
--- a/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
+++ b/.test-infra/jenkins/job_PreCommit_Website_Stage_GCS.groovy
@@ -24,5 +24,9 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
 gradleTask: ':beam-website:stageWebsite',
 triggerPathPatterns: ['^website/.*$']
 )
-builder.build()
+builder.build {
+  publishers {
+buildDescription(/Website published to (http:\/\/.+\/index.html)/)
+  }
+}
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153756)
Time Spent: 0.5h  (was: 20m)

> Staged URL should be posted to GitHub PR
> 
>
> Key: BEAM-5672
> URL: https://issues.apache.org/jira/browse/BEAM-5672
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Website PRs are automatically tested and staged via Jenkins to a location on 
> GCS: https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/
> However, it's difficult to find the URL for the staged job; it's buried 
> inside the Jenkins job, inside the Gradle scan, inside the log.
> I believe there's an API for posting back a context message to the pull 
> request. If possible, we should push the URL so it's easily viewable from the 
> pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #340

[Apache Jenkins Server]

See 


Changes:

[pablo] Updating Dataflow v1b3 API

--
[...truncated 51.03 MB...]
[ToKeyedWorkItem (14/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (14/16) 
(48bbaeddc165fbad7eef618b7eda276c) [FINISHED]
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
39dcaad0269fde3acf7578fa1c2836a1.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
1c394862353a1f9908b6d4eaceba4f81.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
056228dfb155e9e65ad0be6431905dbe.
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (10/16) (272a1f2c5489f47c67e075ec26a03036) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (10/16) 
(272a1f2c5489f47c67e075ec26a03036).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
2b8c79cf165f2a27d899b9a5c888594d.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (6/16) (6bc503ee9990fc4d968fdbbc2fdb6da9) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (6/16) 
(6bc503ee9990fc4d968fdbbc2fdb6da9).
[ToKeyedWorkItem (10/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (10/16) 
(272a1f2c5489f47c67e075ec26a03036) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (2/16) 
(d530c28ce6552502047c9a4f73fececd) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (6/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (6/16) 
(6bc503ee9990fc4d968fdbbc2fdb6da9) [FINISHED]
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (13/16) (a7ee3d1db10f7217b8b9ae7ebc2b4af2) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (13/16) 
(a7ee3d1db10f7217b8b9ae7ebc2b4af2).
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (13/16) 
(a7ee3d1db10f7217b8b9ae7ebc2b4af2) [FINISHED]
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (16/16) (bbcebe3576dd9c860f05583a91046c9b) switched from 
RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
26227d6f826c4b6b3e9afbcae038dbe5.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (16/16) 
(bbcebe3576dd9c860f05583a91046c9b).
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
f697e4012084b9326bb5f382e9463b8a.
[ToKeyedWorkItem (16/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (16/16) 
(bbcebe3576dd9c860f05583a91046c9b) [FINISHED]
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (3/16) 
(39dcaad0269fde3acf7578fa1c2836a1) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-4] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (5/16) 
(1c394862353a1f9908b6d4eaceba4f81) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1669

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

[scott] Update release docs /current redirect for 2.7.0 release

[scott] [BEAM-5719] Use gcs rsync and validate that a valid baseUrl is

--
[...truncated 51.29 MB...]
at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Oct 12, 2018 2:34:19 AM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1108)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:78)
at 
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:207)
at 
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:55)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 

[beam] 01/01: Merge pull request #6662 from pabloem/refresh-api

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 50e02f33cd9794f855e0b39a9628e5596ca9da13
Merge: f9f3c38 32c080d
Author: Pablo 
AuthorDate: Thu Oct 11 19:28:57 2018 -0700

Merge pull request #6662 from pabloem/refresh-api

Updating Dataflow v1b3 API

 .../dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

[beam] branch master updated (f9f3c38 -> 50e02f3)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f9f3c38  Merge pull request #6658 from 
yifanzou/BEAM-5718/exclude_bigquery_io_it
 add 32c080d  Updating Dataflow v1b3 API
 new 50e02f3  Merge pull request #6662 from pabloem/refresh-api

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py | 9 ++---
 1 file changed, 6 insertions(+), 3 deletions(-)

Jenkins build is back to normal : beam_PerformanceTests_MongoDBIO_IT #785

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #1099

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #1088

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5615?focusedWorklogId=153755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153755
 ]

ASF GitHub Bot logged work on BEAM-5615:


Author: ASF GitHub Bot
Created on: 12/Oct/18 01:43
Start Date: 12/Oct/18 01:43
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #6570: [BEAM-5615] fix cmp 
is an invalid keyword for sort function in python 3
URL: https://github.com/apache/beam/pull/6570#issuecomment-429176130
 
 
   I took a closer look at `Top.Of()` API and I think my original analysis is 
correct - the API accepts both `compare` and `key` functions, and current 
behavior for this case is that we apply `key` function first, and the `compare` 
function afterwards. Disregarding `compare` function when `key` is supplied 
would be a breaking API change. Although `Top.Of()` docstring says: 
   ```
   compare should be an implementation of "a < b" taking at least two arguments
   (a and b). Additional arguments and side inputs specified in the apply call
   become additional arguments to the comparator.  Defaults to the natural
   ordering of the elements.
   
   The arguments 'key' and 'reverse' may instead be passed as keyword
   arguments, and have the same meaning as for Python's sort functions.
   ```
   "Instead" can be interpreted as "`compare` and `key` should be exclusive", 
however this is not enforced by the SDK, and the behavior of `Top.Of()` is 
consistent with behavior of `sorted()` built-in function on Python 2, except 
that `Top.Of()` requires that `compare` returns a boolean, while `sorted()` on 
`Python 2` requires that `cmp` returns -1, 0, 1. In Python 3, `sorted()` no 
longer accepts `cmp`.
   
   Things we can do:
   1) Fix the docstring, and explicitly allow `compare` and `key`. This is the 
less restrictive option for our users, although SDK code is becomes more 
convoluted.
   2) Gradually enforce the docstring - require `compare` and `key`  to be 
exclusive parameters. This would slightly simplify the SDK.
   3) Gradually deprecate `compare` parameter. Assume the natural ordering by 
default, and require a `key` function for custom order. This will also simplify 
SDK code and will be more consistent with `sorted()` signature on Python 3.
   
   3 may be more Pythonic option, 1 is the least restrictive for users, also it 
does not require API deprecation. 
   
   I propose we do a combination 1 and 3 by targeting 3 for Beam 3.0 (or first 
SDK that drops Python 2 support), while doing 1 in short term. 
   
   WDYT @aaltay @robertwb.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153755)
Time Spent: 3h 10m  (was: 3h)

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> 

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1352

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #6249

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5442) PortableRunner swallows custom options for Runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5442?focusedWorklogId=153751=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153751
 ]

ASF GitHub Bot logged work on BEAM-5442:


Author: ASF GitHub Bot
Created on: 12/Oct/18 01:20
Start Date: 12/Oct/18 01:20
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6600: [BEAM-5442] 
Store duplicate unknown options in a list argument
URL: https://github.com/apache/beam/pull/6600#issuecomment-429172685
 
 
   Is there an alternative we can do to this approach of inferring a list from 
duplicates / automatically populating these options into the parser?  It makes 
certain things brittle--for example, every time we try to access such  an 
option, we now need to branch on whether a possibly-multiply-passed option is 
passed once or more than once.
   
   Can we do this parsing at the runner side instead, where instead of 
inferring this behavior, we pass a list of unparsed options?  In this way, the 
runner side could parse these as they wish, and we don't need this "magic" 
behavior.
   
   CC: @robertwb @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153751)
Time Spent: 8h 20m  (was: 8h 10m)

> PortableRunner swallows custom options for Runner
> -
>
> Key: BEAM-5442
> URL: https://issues.apache.org/jira/browse/BEAM-5442
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Fix For: 2.8.0
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> The PortableRunner doesn't pass custom PipelineOptions to the executing 
> Runner.
> Example: {{--parallelism=4}} won't be forwarded to the FlinkRunner.
> (The option is just removed during proto translation without any warning)
> We should allow some form of customization through the options, even for the 
> PortableRunner. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1251

[Apache Jenkins Server]

See 


Changes:

[scott] Update release docs /current redirect for 2.7.0 release

[ehudm] Add individual emails on first build failure.

[scott] [BEAM-5719] Use gcs rsync and validate that a valid baseUrl is

--
[...truncated 18.94 MB...]
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
Oct 12, 2018 1:14:00 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
Oct 12, 2018 1:14:00 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-1012011355-6e6a44de/output/results/staging/
Oct 12, 2018 1:14:00 AM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <70577 bytes, hash 2xTMrYn-QRHxcVqHFaWRJg> to 

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #339

[Apache Jenkins Server]

See 


--
[...truncated 51.08 MB...]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
4644b93174ba61551d8bc68a11661810.
[ToKeyedWorkItem (9/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (9/16) (8117339f4d154f2c930fafe52859bc99) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (9/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (9/16) 
(8117339f4d154f2c930fafe52859bc99).
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (2/16) (d93729913b77b36eb975351d10037743) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (2/16) 
(d93729913b77b36eb975351d10037743).
[ToKeyedWorkItem (9/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (9/16) 
(8117339f4d154f2c930fafe52859bc99) [FINISHED]
[ToKeyedWorkItem (2/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (2/16) 
(d93729913b77b36eb975351d10037743) [FINISHED]
[ToKeyedWorkItem (1/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (1/16) (1e770a91a9bab7817ab09cab178f8328) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (1/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (1/16) 
(1e770a91a9bab7817ab09cab178f8328).
[ToKeyedWorkItem (13/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (13/16) 
(78d70156f46186e19b3b1d7d090d592b) [FINISHED]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (16/16) 
(b024344483f44c31e7ccd2a9804dff2b) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (1/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (1/16) 
(1e770a91a9bab7817ab09cab178f8328) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
ee42f9cab8db2583f54656bfcc2ffd39.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (4/16) (0af0a69d6be0b02070ed663f03e1e1c8) switched from RUNNING 
to FINISHED.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (4/16) 
(0af0a69d6be0b02070ed663f03e1e1c8).
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
379926c74f166333cf7c817460b6da94.
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
ToKeyedWorkItem (15/16) (5921ae151803795c3fa21099041eaa1e) switched from 
RUNNING to FINISHED.
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for ToKeyedWorkItem (15/16) 
(5921ae151803795c3fa21099041eaa1e).
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (14/16) 
(8aa4ab2205cad6e357e7bfea6c030799) switched from RUNNING to FINISHED.
[ToKeyedWorkItem (4/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (4/16) 
(0af0a69d6be0b02070ed663f03e1e1c8) [FINISHED]
[ToKeyedWorkItem (15/16)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task ToKeyedWorkItem (15/16) 
(5921ae151803795c3fa21099041eaa1e) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
8291adb1221d91b247366d16de840331.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
7e921d0057f0fab913fa806e9a9c610f.
[flink-akka.actor.default-dispatcher-5] INFO 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1351

[Apache Jenkins Server]

See 


Changes:

[yifanzou] [BEAM-5718] exclude BigQueryITReadIT

--
[...truncated 78.89 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 

[jira] [Work logged] (BEAM-1081) annotations should support custom messages and classes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1081?focusedWorklogId=153750=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153750
 ]

ASF GitHub Bot logged work on BEAM-1081:


Author: ASF GitHub Bot
Created on: 12/Oct/18 01:00
Start Date: 12/Oct/18 01:00
Worklog Time Spent: 10m 
  Work Description: jglezt opened a new pull request #6670: [BEAM-1081] 
Annotations custom message support and classes tests.
URL: https://github.com/apache/beam/pull/6670
 
 
   Added support for custom messages using the arg "custom_message".
   The annotation decorator already supports tagging classes, I only added a 
few tests to check it out.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153750)
Time Spent: 10m
Remaining Estimate: 0h

> annotations should support custom messages and classes
> --
>
> Key: BEAM-1081
> URL: 

Build failed in Jenkins: beam_PreCommit_Java_Cron #451

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

[scott] Update release docs /current redirect for 2.7.0 release

[valentyn] Skip tests that halt test suite execution on Python 3

[plovytsimon] [BEAM-5624] avro.schema parser for py3

[scott] [BEAM-5717] Only pass PR ID to Gradle if available

[scott] Trim PR ID property before checking for truthiness

[ehudm] Add individual emails on first build failure.

[kedin] Fix errorprone to version 2.3.1

[scott] [BEAM-5719] Use gcs rsync and validate that a valid baseUrl is

[yifanzou] [BEAM-5718] exclude BigQueryITReadIT

--
[...truncated 44.84 MB...]

> Task :beam-sdks-java-io-hadoop-common:buildDependents
Caching disabled for task ':beam-sdks-java-io-hadoop-common:buildDependents': 
Caching has not been enabled for the task
Task ':beam-sdks-java-io-hadoop-common:buildDependents' is not up-to-date 
because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-java-io-hadoop-common:buildDependents (Thread[Task worker for ':' 
Thread 5,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
5041dc99151d9d3ea85904a15452161d
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 5,5,main]) completed. Took 0.005 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:processTestResources' 
as it has no source files and no previous output files.
:beam-vendor-sdks-java-extensions-protobuf:processTestResources (Thread[Task 
worker for ':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:testClasses UP-TO-DATE
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:testClasses' as it 
has no actions.
:beam-vendor-sdks-java-extensions-protobuf:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:packageTests
Build cache key for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests' is 
2b50c5905885674a697a325a077c7fa7
Caching disabled for task 
':beam-vendor-sdks-java-extensions-protobuf:packageTests': Caching has not been 
enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:packageTests' is not 
up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:packageTests (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.002 secs.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:assemble
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:assemble' as it has 
no actions.
:beam-vendor-sdks-java-extensions-protobuf:assemble (Thread[Task worker for ':' 
Thread 5,5,main]) completed. Took 0.0 secs.
:beam-vendor-sdks-java-extensions-protobuf:checkstyleMain (Thread[Task worker 
for ':' Thread 5,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:checkstyleMain NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:checkstyleMain' as it 
has no source files and no previous output files.

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT #592

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT_HDFS #485

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #1091

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #768

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #767

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #780

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #1120

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #869

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #863

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #767

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Python_Cron #454

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

[scott] Update release docs /current redirect for 2.7.0 release

[valentyn] Skip tests that halt test suite execution on Python 3

[plovytsimon] [BEAM-5624] avro.schema parser for py3

[scott] [BEAM-5717] Only pass PR ID to Gradle if available

[scott] Trim PR ID property before checking for truthiness

[ehudm] Add individual emails on first build failure.

[kedin] Fix errorprone to version 2.3.1

[scott] [BEAM-5719] Use gcs rsync and validate that a valid baseUrl is

[yifanzou] [BEAM-5718] exclude BigQueryITReadIT

--
[...truncated 1.04 MB...]
test_create_write_read_compressed 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
:336:
 DeprecationWarning: Please use assertNotEqual instead.
  self.assertNotEquals(data, self._fake_hdfs.files[path].getvalue())
ok
test_delete_dir (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
ok
test_delete_error (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_delete_file (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_exists (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_directory 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_directory_trailing_slash 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
ok
test_match_file_empty 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file_error 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file_with_limits 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
:276:
 DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(len(files), 1)
ok
test_match_file_with_zero_limit 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_mkdirs (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_mkdirs_failed (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_open (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_open_bad_path (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_rename_directory 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_rename_file (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_rename_file_error 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... SKIP: This test 
still needs to be fixed on Python 3TODO: BEAM-5627
test_scheme (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_size (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_url_join (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_url_split (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
ok
test_delete_table_fails_dataset_not_exist 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_fails_service_error 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_fails_table_not_exist 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_succeeds (apache_beam.io.gcp.tests.utils_test.UtilsTest) ... 
SKIP: Bigquery dependencies are not installed.
test_big_query_legacy_sql 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
test_big_query_new_types 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
test_big_query_standard_sql 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
get_test_rows (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: 
GCP dependencies are not installed
test_read_from_query (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... 
SKIP: GCP dependencies are not installed
test_read_from_query_sql_format 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_query_unflatten_records 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_table (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... 
SKIP: GCP dependencies are not installed

Build failed in Jenkins: beam_PerformanceTests_JDBC #1201

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

[scott] Update release docs /current redirect for 2.7.0 release

[valentyn] Skip tests that halt test suite execution on Python 3

[plovytsimon] [BEAM-5624] avro.schema parser for py3

[scott] [BEAM-5717] Only pass PR ID to Gradle if available

[scott] Trim PR ID property before checking for truthiness

[ehudm] Add individual emails on first build failure.

[kedin] Fix errorprone to version 2.3.1

[scott] [BEAM-5719] Use gcs rsync and validate that a valid baseUrl is

[yifanzou] [BEAM-5718] exclude BigQueryITReadIT

--
[...truncated 265.35 KB...]
at org.postgresql.Driver.connect(Driver.java:254)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:94)
at 
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:79)
at 
org.apache.commons.dbcp2.DataSourceConnectionFactory.createConnection(DataSourceConnectionFactory.java:44)
at 
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at 
org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:868)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at 
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at 
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
at 
org.apache.beam.sdk.io.jdbc.JdbcIO$Write$WriteFn.startBundle(JdbcIO.java:791)
at 
org.apache.beam.sdk.io.jdbc.JdbcIO$Write$WriteFn$DoFnInvoker.invokeStartBundle(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.startBundle(SimpleDoFnRunner.java:226)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:301)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:311)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:309)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:77)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:621)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:609)
at 
org.apache.beam.sdk.io.common.TestRow$DeterministicallyConstructTestRowFn.processElement(TestRow.java:74)
at 
org.apache.beam.sdk.io.common.TestRow$DeterministicallyConstructTestRowFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:324)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:393)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:362)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:290)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 

Build failed in Jenkins: beam_PostCommit_Python_Verify #6248

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

[scott] Update release docs /current redirect for 2.7.0 release

--
[...truncated 1.29 MB...]
test_match_directory_trailing_slash 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
ok
test_match_file_empty 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file_error 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_match_file_with_limits 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
:276:
 DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(len(files), 1)
ok
test_match_file_with_zero_limit 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_mkdirs (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_mkdirs_failed (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_open (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_open_bad_path (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_rename_directory 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_rename_file (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) 
... ok
test_rename_file_error 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... SKIP: This test 
still needs to be fixed on Python 3TODO: BEAM-5627
test_scheme (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_size (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_url_join (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... ok
test_url_split (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest) ... 
ok
test_delete_table_fails_dataset_not_exist 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_fails_service_error 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_fails_table_not_exist 
(apache_beam.io.gcp.tests.utils_test.UtilsTest) ... SKIP: Bigquery dependencies 
are not installed.
test_delete_table_succeeds (apache_beam.io.gcp.tests.utils_test.UtilsTest) ... 
SKIP: Bigquery dependencies are not installed.
test_big_query_legacy_sql 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
test_big_query_new_types 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
test_big_query_standard_sql 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT) 
... SKIP: IT is skipped because --test-pipeline-options is not specified
get_test_rows (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: 
GCP dependencies are not installed
test_read_from_query (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... 
SKIP: GCP dependencies are not installed
test_read_from_query_sql_format 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_query_unflatten_records 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_table (apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... 
SKIP: GCP dependencies are not installed
test_read_from_table_and_job_complete_retry 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_table_and_multiple_pages 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_read_from_table_as_tablerows 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_table_schema_without_project 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_using_both_query_and_table_fails 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_using_neither_query_nor_table_fails 
(apache_beam.io.gcp.bigquery_test.TestBigQueryReader) ... SKIP: GCP 
dependencies are not installed
test_nested_schema_as_json (apache_beam.io.gcp.bigquery_test.TestBigQuerySink) 
... SKIP: GCP dependencies are not installed
test_parse_schema_descriptor 
(apache_beam.io.gcp.bigquery_test.TestBigQuerySink) ... SKIP: GCP dependencies 
are not installed
test_project_table_display_data 
(apache_beam.io.gcp.bigquery_test.TestBigQuerySink) ... SKIP: GCP dependencies 
are not installed

[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153745
 ]

ASF GitHub Bot logged work on BEAM-5637:


Author: ASF GitHub Bot
Created on: 12/Oct/18 00:03
Start Date: 12/Oct/18 00:03
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python 
support for custom dataflow worker jar
URL: https://github.com/apache/beam/pull/6667#issuecomment-429161109
 
 
   Run Python PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153745)
Time Spent: 0.5h  (was: 20m)

> Python support for custom dataflow worker jar
> -
>
> Key: BEAM-5637
> URL: https://issues.apache.org/jira/browse/BEAM-5637
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> One of the slightly subtle aspects is that we would need to ignore one of the 
> staged jars for portable Python jobs. That requires a change to the Python 
> boot code: 
> https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Website_Stage_GCS_Cron #30

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1350

[Apache Jenkins Server]

See 


Changes:

[ehudm] Fix dockerized-jenkins copy operation.

--
[...truncated 79.38 KB...]
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.4.3.zip
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.3.0.tar.gz

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Ruoyun Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647198#comment-16647198
 ] 

Ruoyun Huang commented on BEAM-5627:


Valentyn is right, there are a lot of them in the tests.

 

I will take care of 
[filesystems_test.py|https://github.com/apache/beam/pull/6610/files#diff-9bc1968a10d6ef5df1a1690027a774b0]
 and 
[io/sources_test.py|https://github.com/apache/beam/pull/6610/files#diff-aa300c32e00d2cbf44349027b526271f]
 (apparently they will affect internal test, will need extra efforts anyway). 

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5672) Staged URL should be posted to GitHub PR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5672?focusedWorklogId=153743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153743
 ]

ASF GitHub Bot logged work on BEAM-5672:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:49
Start Date: 11/Oct/18 23:49
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #6669: [BEAM-5672] Pull URL out 
of logs for build description
URL: https://github.com/apache/beam/pull/6669#issuecomment-429158619
 
 
   Run Website_Stage_GCS PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153743)
Time Spent: 20m  (was: 10m)

> Staged URL should be posted to GitHub PR
> 
>
> Key: BEAM-5672
> URL: https://issues.apache.org/jira/browse/BEAM-5672
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Website PRs are automatically tested and staged via Jenkins to a location on 
> GCS: https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/
> However, it's difficult to find the URL for the staged job; it's buried 
> inside the Jenkins job, inside the Gradle scan, inside the log.
> I believe there's an API for posting back a context message to the pull 
> request. If possible, we should push the URL so it's easily viewable from the 
> pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153740=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153740
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:42
Start Date: 11/Oct/18 23:42
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6592: 
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224598766
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Flink.groovy
 ##
 @@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
 
 Review comment:
   I checked, its highlighted in other groovy files as well. 
   It might have something to do with groovy files.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153740)
Time Spent: 32.5h  (was: 32h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 32.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153742
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:42
Start Date: 11/Oct/18 23:42
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6642: [BEAM-5701] port 
datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#issuecomment-429157520
 
 
   Run Python PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153742)
Time Spent: 3h 50m  (was: 3h 40m)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=153741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153741
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:42
Start Date: 11/Oct/18 23:42
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6592: 
[BEAM-4176] Enable Post Commit JAVA PVR tests for Flink
URL: https://github.com/apache/beam/pull/6592#discussion_r224601348
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
 ##
 @@ -104,9 +108,21 @@ private void scheduleRelease(JobInfo jobInfo) {
 WrappedContext wrapper = getCache().get(jobInfo.jobId());
 Preconditions.checkState(
 wrapper != null, "Releasing context for unknown job: " + 
jobInfo.jobId());
-// Do not release this asynchronously, as the releasing could fail due to 
the classloader not being
-// available anymore after the tasks have been removed from the execution 
engine.
-release(wrapper);
+
+PipelineOptions pipelineOptions =
+PipelineOptionsTranslation.fromProto(jobInfo.pipelineOptions());
+int environmentCacheTTLMillis =
+
pipelineOptions.as(PortablePipelineOptions.class).getEnvironmentCacheMillis();
+if (environmentCacheTTLMillis > 0) {
+  // Schedule task to clean the container later.
+  // Ensure that this class is loaded in the parent Flink classloader.
+  getExecutor()
+  .schedule(() -> release(wrapper), environmentCacheTTLMillis, 
TimeUnit.MILLISECONDS);
 
 Review comment:
   I had to incorporate those change to make this PR work. 
   There is no way to make PR depend upon other PR in git so had to resort to 
adding 2 separate commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153741)
Time Spent: 32h 40m  (was: 32.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: 81VxNWtFtke.png, Screen Shot 2018-08-14 at 4.18.31 
> PM.png, Screen Shot 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 32h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5672) Staged URL should be posted to GitHub PR

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5672?focusedWorklogId=153739=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153739
 ]

ASF GitHub Bot logged work on BEAM-5672:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:40
Start Date: 11/Oct/18 23:40
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #6669: [BEAM-5672] Pull URL 
out of logs for build description
URL: https://github.com/apache/beam/pull/6669#issuecomment-429157124
 
 
   R: @udim 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153739)
Time Spent: 10m
Remaining Estimate: 0h

> Staged URL should be posted to GitHub PR
> 
>
> Key: BEAM-5672
> URL: https://issues.apache.org/jira/browse/BEAM-5672
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Website PRs are automatically tested and staged via Jenkins to a location on 
> GCS: https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/
> However, it's difficult to find the URL for the staged job; it's buried 
> inside the Jenkins job, inside the Gradle scan, inside the log.
> I believe there's an API for posting back a context message to the pull 
> request. If possible, we should push the URL so it's easily viewable from the 
> pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5672) Staged URL should be posted to GitHub PR

[Scott Wegner (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647187#comment-16647187
 ] 

Scott Wegner commented on BEAM-5672:


The functionality I was thinking of is powered by the ghprb plugin via the 
[buildStatus|https://github.com/jenkinsci/ghprb-plugin#job-dsl-support] hook. 
However, this only supports a static string. It wouldn't be possible to set a 
status with a different URL per pull request.

However, we could possible add it to the github PR template?

> Staged URL should be posted to GitHub PR
> 
>
> Key: BEAM-5672
> URL: https://issues.apache.org/jira/browse/BEAM-5672
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Major
>
> Website PRs are automatically tested and staged via Jenkins to a location on 
> GCS: https://builds.apache.org/job/beam_PreCommit_Website_Stage_GCS_Commit/
> However, it's difficult to find the URL for the staged job; it's buried 
> inside the Jenkins job, inside the Gradle scan, inside the log.
> I believe there's an API for posting back a context message to the pull 
> request. If possible, we should push the URL so it's easily viewable from the 
> pull request.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5703) Migrate Python streaming and portable integration tests to use a staged dataflow worker jar

[Ruoyun Huang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647182#comment-16647182
 ] 

Ruoyun Huang commented on BEAM-5703:


Now I assume this is the central place of integration test : 
[https://github.com/apache/beam/blob/master/sdks/python/scripts/run_integration_test.sh]

Let me know if there are more others. 

> Migrate Python streaming and portable integration tests to use a staged 
> dataflow worker jar
> ---
>
> Key: BEAM-5703
> URL: https://issues.apache.org/jira/browse/BEAM-5703
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ruoyun Huang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153738
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:30
Start Date: 11/Oct/18 23:30
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6642: [BEAM-5701] port 
datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#issuecomment-429155493
 
 
   Wait, do not merge. The test failed on my local. Looking into that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153738)
Time Spent: 3h 40m  (was: 3.5h)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169
 ] 

Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:25 PM:
---

The scheduling of all tasks to the same slot is consistent, distribution over 
hosts isn't. With parallelism 4, different result (multiple hosts).

!image-2018-10-11-16-20-45-221.png|width=100!


was (Author: thw):
With parallelism 4, different result.

!image-2018-10-11-16-20-45-221.png|width=100%!

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5718) Exclude Bigquery IO Read IT

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5718?focusedWorklogId=153735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153735
 ]

ASF GitHub Bot logged work on BEAM-5718:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:25
Start Date: 11/Oct/18 23:25
Worklog Time Spent: 10m 
  Work Description: akedin closed pull request #6658: [BEAM-5718] exclude 
BigQueryITReadIT
URL: https://github.com/apache/beam/pull/6658
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/google-cloud-dataflow-java/build.gradle 
b/runners/google-cloud-dataflow-java/build.gradle
index 168ac9745fc..5e196e7060a 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -150,6 +150,7 @@ task googleCloudPlatformIntegrationTest(type: Test) {
   ])
 
   include '**/*IT.class'
+  exclude '**/BigQueryIOReadIT.class'
   exclude '**/PubsubReadIT.class'
   maxParallelForks 4
   classpath = configurations.googleCloudPlatformIntegrationTest


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153735)
Time Spent: 1.5h  (was: 1h 20m)

> Exclude Bigquery IO Read IT 
> 
>
> Key: BEAM-5718
> URL: https://issues.apache.org/jira/browse/BEAM-5718
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/] failing 
> consistently, appears to be a quota issue
>  
>  
>  
>  
> `java.lang.RuntimeException: java.io.IOException: Extract job 
> beam_job_fef64e64c9ae433997d0654f9951615b_testpipelinejenkins1011152113771f3474-extract
>  failed, status: { "errorResult" : { "location" : "extract_size_gb", 
> "message" : "Quota exceeded: Your project exceeded quota for export size. For 
> more information, see 
> [https://cloud.google.com/bigquery/troubleshooting-errors];, "reason" : 
> "quotaExceeded" }, "errors" : [ { "location" : "extract_size_gb", "message" : 
> "Quota exceeded: Your project exceeded quota for export size. For more 
> information, see [https://cloud.google.com/bigquery/troubleshooting-errors];, 
> "reason" : "quotaExceeded" } ], "state" : "DONE" }. at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.executeExtract(BigQuerySourceBase.java:191)`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647174#comment-16647174
 ] 

Udi Meiri commented on BEAM-5663:
-

My investigation from April 2018 is here: 
https://github.com/udim/beam/commit/3c08b21a9950e39bf71b150a087ae7bf7350f7c3

I haven't test with cython yet, but I just checked and Bazel has experimental 
support for pip: 
https://github.com/bazelbuild/rules_python/tree/master/examples/version


> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (0831dd0 -> f9f3c38)

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0831dd0  Merge pull request #6641 from udim/dockerized-jenkins-copy
 add 3ec025c  [BEAM-5718] exclude BigQueryITReadIT
 add f9f3c38  Merge pull request #6658 from 
yifanzou/BEAM-5718/exclude_bigquery_io_it

No new revisions were added by this update.

Summary of changes:
 runners/google-cloud-dataflow-java/build.gradle | 1 +
 1 file changed, 1 insertion(+)

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169
 ] 

Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:25 PM:
---

The scheduling of all tasks to the same slot is consistent, distribution over 
hosts isn't. With parallelism 4, different result (multiple hosts).

!image-2018-10-11-16-20-45-221.png|width=100%!


was (Author: thw):
The scheduling of all tasks to the same slot is consistent, distribution over 
hosts isn't. With parallelism 4, different result (multiple hosts).

!image-2018-10-11-16-20-45-221.png|width=100!

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169
 ] 

Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:23 PM:
---

With parallelism 4, different result.

!image-2018-10-11-16-20-45-221.png|width=100%!


was (Author: thw):
With parallelism 4, different result.

!image-2018-10-11-16-20-45-221.png|width=100!

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169
 ] 

Thomas Weise edited comment on BEAM-5713 at 10/11/18 11:23 PM:
---

With parallelism 4, different result.

!image-2018-10-11-16-20-45-221.png|width=100!


was (Author: thw):
With parallelism 4, *almost* same

!image-2018-10-11-16-20-45-221.png|width=100%!

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5637) Python support for custom dataflow worker jar

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5637?focusedWorklogId=153733=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153733
 ]

ASF GitHub Bot logged work on BEAM-5637:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:22
Start Date: 11/Oct/18 23:22
Worklog Time Spent: 10m 
  Work Description: HuangLED commented on issue #6667: [BEAM-5637] Python 
support for custom dataflow worker jar
URL: https://github.com/apache/beam/pull/6667#issuecomment-429154014
 
 
   R:  @herohde  @boyuanzz 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153733)
Time Spent: 20m  (was: 10m)

> Python support for custom dataflow worker jar
> -
>
> Key: BEAM-5637
> URL: https://issues.apache.org/jira/browse/BEAM-5637
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Henning Rohde
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> One of the slightly subtle aspects is that we would need to ignore one of the 
> staged jars for portable Python jobs. That requires a change to the Python 
> boot code: 
> https://github.com/apache/beam/blob/66d7c865b7267f388ee60752891a9141fad43774/sdks/python/container/boot.go#L104



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1668

[Apache Jenkins Server]

See 


Changes:

[ccy] [BEAM-3717] Fix problems with Beam downloads page

[valentyn] Skip tests that halt test suite execution on Python 3

[plovytsimon] [BEAM-5624] avro.schema parser for py3

[scott] [BEAM-5717] Only pass PR ID to Gradle if available

[scott] Trim PR ID property before checking for truthiness

[ehudm] Add individual emails on first build failure.

[kedin] Fix errorprone to version 2.3.1

--
[...truncated 50.04 MB...]
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
at 
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Oct 11, 2018 11:21:22 PM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1108)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 

[jira] [Commented] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647169#comment-16647169
 ] 

Thomas Weise commented on BEAM-5713:


With parallelism 4, *almost* same

!image-2018-10-11-16-20-45-221.png|width=100%!

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5713) Flink portable runner schedules all tasks of streaming job on same task manager

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-5713:
---
Attachment: image-2018-10-11-16-20-45-221.png

> Flink portable runner schedules all tasks of streaming job on same task 
> manager
> ---
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.8.0
>Reporter: Thomas Weise
>Assignee: Maximilian Michels
>Priority: Major
>  Labels: portability, portability-flink
> Attachments: Different SlotSharingGroup.png, With 
> RichParallelSourceFunction and parallelism 5.png, 
> image-2018-10-11-11-43-50-333.png, image-2018-10-11-16-20-45-221.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming 
> pipeline with parallelism of 8 will get all tasks scheduled on the same task 
> manager, causing the host to be fully booked and the remaining cluster idle.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153728=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153728
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 11/Oct/18 23:02
Start Date: 11/Oct/18 23:02
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6642: [BEAM-5701] port 
datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#issuecomment-429150143
 
 
   Renamed it to 'datastore_write_it_pipeline'


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153728)
Time Spent: 3.5h  (was: 3h 20m)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reopened BEAM-5627:
---

[~rakeshkumar] Feel free to close if this error is completely gone.

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647145#comment-16647145
 ] 

Valentyn Tymofieiev edited comment on BEAM-5627 at 10/11/18 10:57 PM:
--

[~rakeshkumar] Did you check if there are any other tests in IO module that 
still fail with this error?


was (Author: tvalentyn):
[~rakeshkumar] Did you check if there any other tests in IO module that fail 
with this error?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647145#comment-16647145
 ] 

Valentyn Tymofieiev commented on BEAM-5627:
---

[~rakeshkumar] Did you check if there any other tests in IO module that fail 
with this error?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rakesh Kumar resolved BEAM-5627.

   Resolution: Fixed
Fix Version/s: Not applicable

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Rakesh Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647144#comment-16647144
 ] 

Rakesh Kumar commented on BEAM-5627:


[~ruoyun] thanks for fixing this.

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153725
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 11/Oct/18 22:49
Start Date: 11/Oct/18 22:49
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on a change in pull request 
#6642: [BEAM-5701] port datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#discussion_r224627668
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/datastore_write_it_test.py
 ##
 @@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An integration test for datastore_write
+
+This test creates entities and writes them to Cloud Datastore. Subsequently,
+these entities are read from Cloud Datastore, compared to the expected value
+for the entity, and deleted.
+
+There is no output; instead, we use `assert_that` transform to verify the
+results in the pipeline.
+"""
+
+from __future__ import absolute_import
+
+import logging
+import random
+import unittest
+from datetime import datetime
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.io.gcp import datastore_write
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+
+class DatastoreWriteIT(unittest.TestCase):
+
+  NUM_ENTITIES = 1001
+  LIMIT = 500
+
+  def run_datastore_write(self, limit=None):
+test_pipeline = TestPipeline(is_integration_test=True)
+current_time = datetime.now().strftime("%m%d%H%M%S")
+seed = random.randint(0, 10)
+kind = 'testkind%s%d' % (current_time, seed)
+pipeline_verifiers = [PipelineStateMatcher()]
+extra_opts = {'kind': kind,
+  'num_entities': self.NUM_ENTITIES,
+  'on_success_matcher': all_of(*pipeline_verifiers)}
+if limit is not None:
+  extra_opts['limit'] = limit
+
+datastore_write.run(test_pipeline.get_full_options_as_args(
+**extra_opts))
+
+  @attr('IT')
+  def test_datastore_write_limit(self):
 
 Review comment:
   Nevermind.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153725)
Time Spent: 3h 20m  (was: 3h 10m)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5701) Port Python IT test to Github: datastore_write

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5701?focusedWorklogId=153724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153724
 ]

ASF GitHub Bot logged work on BEAM-5701:


Author: ASF GitHub Bot
Created on: 11/Oct/18 22:46
Start Date: 11/Oct/18 22:46
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on a change in pull request 
#6642: [BEAM-5701] port datastore_write Python integration tests to Beam
URL: https://github.com/apache/beam/pull/6642#discussion_r224627243
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/datastore_write_it_test.py
 ##
 @@ -0,0 +1,70 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An integration test for datastore_write
+
+This test creates entities and writes them to Cloud Datastore. Subsequently,
+these entities are read from Cloud Datastore, compared to the expected value
+for the entity, and deleted.
+
+There is no output; instead, we use `assert_that` transform to verify the
+results in the pipeline.
+"""
+
+from __future__ import absolute_import
+
+import logging
+import random
+import unittest
+from datetime import datetime
+
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
+
+from apache_beam.io.gcp import datastore_write
+from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
+from apache_beam.testing.test_pipeline import TestPipeline
+
+
+class DatastoreWriteIT(unittest.TestCase):
+
+  NUM_ENTITIES = 1001
+  LIMIT = 500
+
+  def run_datastore_write(self, limit=None):
+test_pipeline = TestPipeline(is_integration_test=True)
+current_time = datetime.now().strftime("%m%d%H%M%S")
+seed = random.randint(0, 10)
+kind = 'testkind%s%d' % (current_time, seed)
+pipeline_verifiers = [PipelineStateMatcher()]
+extra_opts = {'kind': kind,
+  'num_entities': self.NUM_ENTITIES,
+  'on_success_matcher': all_of(*pipeline_verifiers)}
+if limit is not None:
+  extra_opts['limit'] = limit
+
+datastore_write.run(test_pipeline.get_full_options_as_args(
+**extra_opts))
+
+  @attr('IT')
+  def test_datastore_write_limit(self):
 
 Review comment:
   Can we have two of these, a limit and non-limit test? Seems strange to allow 
parameterizing it and then not do so.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 153724)
Time Spent: 3h 10m  (was: 3h)

> Port Python IT test to Github: datastore_write
> --
>
> Key: BEAM-5701
> URL: https://issues.apache.org/jira/browse/BEAM-5701
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5712) Need an integration test for Datastore IO in Python

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5712?focusedWorklogId=153723=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153723
 ]

ASF GitHub Bot logged work on BEAM-5712:


Author: ASF GitHub Bot
Created on: 11/Oct/18 22:44
Start Date: 11/Oct/18 22:44
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6643: [BEAM-5712] Adding a 
Datastore Integration Test
URL: https://github.com/apache/beam/pull/6643
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/gcp/datastore_integration_test.py 
b/sdks/python/apache_beam/io/gcp/datastore_integration_test.py
new file mode 100644
index 000..7a72fdd3934
--- /dev/null
+++ b/sdks/python/apache_beam/io/gcp/datastore_integration_test.py
@@ -0,0 +1,201 @@
+
+"""Datastore integration test that tests read, write and delete transforms.
+
+This test creates entities and writes them to Cloud Datastore. Subsequently,
+these entities are read from Cloud Datastore, compared to the expected value
+for the entity, and deleted.
+
+There is no output; instead, we use `assert_that` transform to verify that
+results are as expected.
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import hashlib
+import logging
+import unittest
+import uuid
+
+from nose.plugins.attrib import attr
+
+import apache_beam as beam
+from apache_beam.io.gcp.datastore.v1.datastoreio import DeleteFromDatastore
+from apache_beam.io.gcp.datastore.v1.datastoreio import ReadFromDatastore
+from apache_beam.io.gcp.datastore.v1.datastoreio import WriteToDatastore
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+try:
+  from google.cloud.proto.datastore.v1 import datastore_pb2
+  from google.cloud.proto.datastore.v1 import entity_pb2
+  from google.cloud.proto.datastore.v1 import query_pb2
+
+  from googledatastore import helper as datastore_helper
+  from googledatastore import PropertyFilter
+except ImportError:
+  datastore_pb2 = None
+
+
+NUM_ENTITIES = 1001
+LIMIT_ENTITIES = 500
+
+
+@unittest.skipIf(datastore_pb2 is None, 'GCP dependencies are not installed')
+class DatastoreIT(unittest.TestCase):
+
+  def get_known_args(self,
+ entities=NUM_ENTITIES,
+ limit=LIMIT_ENTITIES,
+ kind=None):
+"""Main entry point."""
+
+parser = argparse.ArgumentParser()
+
+parser.add_argument('--kind',
+dest='kind',
+default='writereadtest',
+help='Datastore Kind')
+parser.add_argument('--num_entities',
+dest='num_entities',
+type=int,
+default=NUM_ENTITIES,
+required=True,
+help='Number of entities to write')
+parser.add_argument('--limit',
+dest='limit',
+default=LIMIT_ENTITIES,
+type=int,
+help='Limit of number of entities to write')
+
+known_args, _ = parser.parse_known_args(
+['--num_entities', str(entities),
+ '--limit', str(limit),
+ '--kind', str(kind)])
+return known_args
+
+  @attr('IT')
+  def test_datastore_functionality(self):
+self.known_args = self.get_known_args()
+self.ancestor = str(uuid.uuid4())
+self.query = make_ancestor_query(self.known_args.kind, None, self.ancestor)
+self._create_and_write_entities()
+self._read_with_limit()
+self._query_written_and_verify()
+self._delete_entities()
+self._verify_deleted()
+
+  def _create_and_write_entities(self):
+# Pipeline 1: Create and write the specified number of Entities to the
+# Cloud Datastore.
+
+# pylint: disable=expression-not-assigned
+with TestPipeline(is_integration_test=True) as p:
+  project = p.get_option('project')
+  logging.info('Writing %s entities to %s',
+   self.known_args.num_entities,
+   project)
+  (p
+   | 'Input' >> beam.Create(list(range(self.known_args.num_entities)))
+   | 'To String' >> beam.Map(str)
+   | 'To Entity' >> beam.Map(
+   EntityWrapper(self.known_args.kind,
+ None,
+ self.ancestor).make_entity)
+   | 'Write to Datastore' >> WriteToDatastore(project))
+
+  def _read_with_limit(self):
+# Optional Pipeline 2: If a read limit was provided, read it and confirm
+# that the expected entities were read.
+if self.known_args.limit is