[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=379218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379218
 ]

ASF GitHub Bot logged work on BEAM-7246:


Author: ASF GitHub Bot
Created on: 30/Jan/20 07:37
Start Date: 30/Jan/20 07:37
Worklog Time Spent: 10m 
  Work Description: mszb commented on issue #10712: [BEAM-7246] Added 
Google Spanner Write Transform
URL: https://github.com/apache/beam/pull/10712#issuecomment-580122678
 
 
   R: @chamikaramj 
   R: @aaltay 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379218)
Time Spent: 14h 10m  (was: 14h)

> Create a Spanner IO for Python
> --
>
> Key: BEAM-7246
> URL: https://issues.apache.org/jira/browse/BEAM-7246
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 14h 10m
>  Remaining Estimate: 0h
>
> Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only).
> Testing in this work item will be in the form of DirectRunner tests and 
> manual testing.
> Integration and performance tests are a separate work item (not included 
> here).
> See https://beam.apache.org/documentation/io/built-in/. The goal is to add 
> Google Clound Spanner to the Database column for the Python/Batch row.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=379215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379215
 ]

ASF GitHub Bot logged work on BEAM-8972:


Author: ASF GitHub Bot
Created on: 30/Jan/20 07:23
Start Date: 30/Jan/20 07:23
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add 
Jenkins job with Combine test for portable Java
URL: https://github.com/apache/beam/pull/10386#issuecomment-580118247
 
 
   Run Load Tests Java Combine Portable Flink Batch
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379215)
Time Spent: 6h 20m  (was: 6h 10m)

> Add a Jenkins job running Combine load test on Java with Flink in Portability 
> mode
> --
>
> Key: BEAM-8972
> URL: https://issues.apache.org/jira/browse/BEAM-8972
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8972) Add a Jenkins job running Combine load test on Java with Flink in Portability mode

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8972?focusedWorklogId=379188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379188
 ]

ASF GitHub Bot logged work on BEAM-8972:


Author: ASF GitHub Bot
Created on: 30/Jan/20 07:10
Start Date: 30/Jan/20 07:10
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #10386: [BEAM-8972] Add 
Jenkins job with Combine test for portable Java
URL: https://github.com/apache/beam/pull/10386#issuecomment-580114123
 
 
   run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379188)
Time Spent: 6h 10m  (was: 6h)

> Add a Jenkins job running Combine load test on Java with Flink in Portability 
> mode
> --
>
> Key: BEAM-8972
> URL: https://issues.apache.org/jira/browse/BEAM-8972
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379187
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 30/Jan/20 07:09
Start Date: 30/Jan/20 07:09
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372787438
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
 
 Review comment:
   I'm not sure if such variables exist, but I agree it would be better to use 
those instead of the paths. Maybe after Jenkins for Beam is up and running 
we'll be able to add the envs
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379187)
Time Spent: 17.5h  (was: 17h 20m)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17.5h
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9211) Spark portable jar test script is missing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9211?focusedWorklogId=379161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379161
 ]

ASF GitHub Bot logged work on BEAM-9211:


Author: ASF GitHub Bot
Created on: 30/Jan/20 05:27
Start Date: 30/Jan/20 05:27
Worklog Time Spent: 10m 
  Work Description: ibzib commented on issue #10723: [BEAM-9211] upload 
missing Spark portable jar test script
URL: https://github.com/apache/beam/pull/10723#issuecomment-580089288
 
 
   Run PortableJar_Spark PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379161)
Time Spent: 20m  (was: 10m)

> Spark portable jar test script is missing
> -
>
> Key: BEAM-9211
> URL: https://issues.apache.org/jira/browse/BEAM-9211
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Labels: portability-spark
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> beam_PostCommit_PortableJar_Spark has been failing since its creation because 
> I forgot to upload the test script it calls. Whoops.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9211) Spark portable jar test script is missing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9211?focusedWorklogId=379160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379160
 ]

ASF GitHub Bot logged work on BEAM-9211:


Author: ASF GitHub Bot
Created on: 30/Jan/20 05:26
Start Date: 30/Jan/20 05:26
Worklog Time Spent: 10m 
  Work Description: ibzib commented on pull request #10723: [BEAM-9211] 
upload missing Spark portable jar test script
URL: https://github.com/apache/beam/pull/10723
 
 
   I accidentally completely omitted this file from #10287. My bad.  
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-9213) FlinkRunner ignores --flink_submit_uber_jar when master unset

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9213?focusedWorklogId=379151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379151
 ]

ASF GitHub Bot logged work on BEAM-9213:


Author: ASF GitHub Bot
Created on: 30/Jan/20 05:00
Start Date: 30/Jan/20 05:00
Worklog Time Spent: 10m 
  Work Description: ibzib commented on pull request #10708: [BEAM-9213] 
throw error when flink_submit_uber_jar and not flink_master
URL: https://github.com/apache/beam/pull/10708#discussion_r372760207
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/flink_runner.py
 ##
 @@ -51,8 +51,11 @@ def default_job_server(self, options):
 flink_master = self.add_http_scheme(
 flink_options.flink_master)
 flink_options.flink_master = flink_master
-if (flink_options.flink_submit_uber_jar
-and flink_master not in MAGIC_HOST_NAMES):
+if flink_options.flink_submit_uber_jar:
+  if flink_master in MAGIC_HOST_NAMES:
+raise ValueError(
+'Cannot use flink_submit_uber_jar with flink_master %s'
+% flink_master)
 
 Review comment:
   > I think we may want to make flink_submit_uber_jar the default
   
   In the long term, I think we should make flink_submit_uber_jar the _only_ 
option in FlinkRunner. Having two substantially different behaviors under the 
same "runner" is confusing to users, and in retrospect 
"flink_submit_uber_jar=false" kind of seems like only an intermediate step. 
WDYT?
   
   (Incidentally, I've realized all this terminology itself is confusing. "Uber 
jar" seems to imply java, even though that is the "java-less" option.)
   
   However, in the short term, I'm not yet confident that flink_submit_uber_jar 
is ready. I've been seeing issues with it hanging in Beam 2.18: 
[BEAM-9225](https://issues.apache.org/jira/browse/BEAM-9225)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379151)
Time Spent: 50m  (was: 40m)

> FlinkRunner ignores --flink_submit_uber_jar when master unset
> -
>
> Key: BEAM-9213
> URL: https://issues.apache.org/jira/browse/BEAM-9213
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Instead, an error should be thrown to let the user know that 
> flink_submit_uber_jar is incompatible with auto/local master.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9225) Flink uber jar job server hangs

[Kyle Weaver (Jira)]

Kyle Weaver created BEAM-9225:
-

 Summary: Flink uber jar job server hangs
 Key: BEAM-9225
 URL: https://issues.apache.org/jira/browse/BEAM-9225
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Kyle Weaver
Assignee: Kyle Weaver


This was observed on Kubernetes. I suspect this behavior might also be the 
reason beam_PostCommit_PortableJar_Flink is timing out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9225) Flink uber jar job server hangs

[Kyle Weaver (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Weaver updated BEAM-9225:
--
Status: Open  (was: Triage Needed)

> Flink uber jar job server hangs
> ---
>
> Key: BEAM-9225
> URL: https://issues.apache.org/jira/browse/BEAM-9225
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Labels: portability-flink
>
> This was observed on Kubernetes. I suspect this behavior might also be the 
> reason beam_PostCommit_PortableJar_Flink is timing out.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379143
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 30/Jan/20 04:18
Start Date: 30/Jan/20 04:18
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-580075739
 
 
   @boyuanzz Thank you!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379143)
Time Spent: 5h 40m  (was: 5.5h)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379142
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 30/Jan/20 04:06
Start Date: 30/Jan/20 04:06
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-580073836
 
 
   Run Java_Examples_Dataflow PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379142)
Time Spent: 5.5h  (was: 5h 20m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379141
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 30/Jan/20 04:06
Start Date: 30/Jan/20 04:06
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-580073808
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379141)
Time Spent: 5h 20m  (was: 5h 10m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Updated] (BEAM-9224) flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job

[Tomo Suzuki (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomo Suzuki updated BEAM-9224:
--
Summary: flaky WordCountIT.testE2EWordCount in Jenkins 
Java_Examples_Dataflow Job  (was: flaky WordCount integration tests on dataflow 
runner)

> flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job
> 
>
> Key: BEAM-9224
> URL: https://issues.apache.org/jira/browse/BEAM-9224
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Tomo Suzuki
>Priority: Major
> Attachments: yY6GxCyAV3C.png
>
>
> Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
> {{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.
> Example: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console]
> {noformat}
> 14:39:58 > Task 
> :runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
> 14:39:58 
> 14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> 14:39:58 java.lang.RuntimeException at WordCountIT.java:69
> 14:39:58 
> 14:39:58 1 test completed, 1 failed
> {noformat}
> The trend is alarming:
> !yY6GxCyAV3C.png|width=675,height=338!
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/]
>  
> This similar to BEAM-3968 (still open), but this time the failure is on 
> {{testE2EWordCount}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9224) flaky WordCount integration tests on dataflow runner

[Tomo Suzuki (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomo Suzuki updated BEAM-9224:
--
Description: 
Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
{{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.

Example: 
[https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console]
{noformat}
14:39:58 > Task 
:runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
14:39:58 
14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
14:39:58 java.lang.RuntimeException at WordCountIT.java:69
14:39:58 
14:39:58 1 test completed, 1 failed
{noformat}
The trend is alarming:

!yY6GxCyAV3C.png|width=675,height=338!

[https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/]

 

This similar to BEAM-3968 (still open), but this time the failure is on 
{{testE2EWordCount}}.

  was:
Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
{{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.

Example: 
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console

{noformat}
14:39:58 > Task 
:runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
14:39:58 
14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
14:39:58 java.lang.RuntimeException at WordCountIT.java:69
14:39:58 
14:39:58 1 test completed, 1 failed
{noformat}





> flaky WordCount integration tests on dataflow runner
> 
>
> Key: BEAM-9224
> URL: https://issues.apache.org/jira/browse/BEAM-9224
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Tomo Suzuki
>Priority: Major
> Attachments: yY6GxCyAV3C.png
>
>
> Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
> {{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.
> Example: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console]
> {noformat}
> 14:39:58 > Task 
> :runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
> 14:39:58 
> 14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> 14:39:58 java.lang.RuntimeException at WordCountIT.java:69
> 14:39:58 
> 14:39:58 1 test completed, 1 failed
> {noformat}
> The trend is alarming:
> !yY6GxCyAV3C.png|width=675,height=338!
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/]
>  
> This similar to BEAM-3968 (still open), but this time the failure is on 
> {{testE2EWordCount}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9224) flaky WordCount integration tests on dataflow runner

[Tomo Suzuki (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomo Suzuki updated BEAM-9224:
--
Attachment: yY6GxCyAV3C.png

> flaky WordCount integration tests on dataflow runner
> 
>
> Key: BEAM-9224
> URL: https://issues.apache.org/jira/browse/BEAM-9224
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Tomo Suzuki
>Priority: Major
> Attachments: yY6GxCyAV3C.png
>
>
> Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
> {{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.
> Example: 
> https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console
> {noformat}
> 14:39:58 > Task 
> :runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
> 14:39:58 
> 14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> 14:39:58 java.lang.RuntimeException at WordCountIT.java:69
> 14:39:58 
> 14:39:58 1 test completed, 1 failed
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9224) flaky WordCount integration tests on dataflow runner

[Tomo Suzuki (Jira)]

Tomo Suzuki created BEAM-9224:
-

 Summary: flaky WordCount integration tests on dataflow runner
 Key: BEAM-9224
 URL: https://issues.apache.org/jira/browse/BEAM-9224
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Tomo Suzuki


Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
{{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.

Example: 
https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console

{noformat}
14:39:58 > Task 
:runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
14:39:58 
14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
14:39:58 java.lang.RuntimeException at WordCountIT.java:69
14:39:58 
14:39:58 1 test completed, 1 failed
{noformat}






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379139
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 30/Jan/20 03:47
Start Date: 30/Jan/20 03:47
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-580070779
 
 
   Java precommit failed:
   ```
   18:48:45 > Task :sdks:java:io:rabbitmq:test FAILED
   18:48:45 
   18:48:45 FAILURE: Build failed with an exception.
   18:48:45 
   18:48:45 * What went wrong:
   18:48:45 Execution failed for task ':sdks:java:io:rabbitmq:test'.
   18:48:45 > Process 'Gradle Test Executor 124' finished with non-zero exit 
value 143
   18:48:45   This problem might be caused by incorrect test process 
configuration.
   18:48:45   Please refer to the test execution section in the User Manual at 
https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
   ```
   
   Java_Examples_Dataflow failed:
   
   ```
   java.lang.RuntimeException: Dataflow job 
2020-01-29_13_55_09-15611797686391011440 terminated in state UNRECOGNIZED but 
did not return a failure reason.
   ```
   
   They both seem transient errors.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379139)
Time Spent: 5h 10m  (was: 5h)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  

[jira] [Work logged] (BEAM-9162) Upgrade Jackson to version 2.10.2

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9162?focusedWorklogId=379138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379138
 ]

ASF GitHub Bot logged work on BEAM-9162:


Author: ASF GitHub Bot
Created on: 30/Jan/20 03:43
Start Date: 30/Jan/20 03:43
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10643: [BEAM-9162] Upgrade 
Jackson to version 2.10.2
URL: https://github.com/apache/beam/pull/10643#issuecomment-580070010
 
 
   I wouldn't rely on the result of the linkage check for an artifact list. 
`LinkageCheck(artifactA,artifactB)` is not the same as `LinkageCheck(artifactA) 
+ LinkageCheck(artifactB)`, where I define `LinkageCheck(artifact...)` as a 
function that takes list of Maven artifacts and returns list of linkage errors 
for a Java project that has dependencies of the artifacts. This is because 
linkage errors in one artifact may be hidden by classes in another artifact. 
For example, the missing `javax.annotation.Nullable` problem 
([BEAM-8917](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-8917)) 
would be undetected if we run Linkage Check on the beam-sdks-java-core artifact 
and another artifact that happens to depend on jsr305 (which contains 
`javax.annotation.Nullable`).
   
   
   To ensure each Beam artifact is checked independently, I created a shell 
script that run "checkJavaLinkage" for each artifact:
   https://gist.github.com/suztomo/ec4834151a81f8e1d26aa0afb278818b
   What do you think about making this part of the Beam Jenkins jobs? 
([BEAM-9206](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-9206))
   
   
   > an issue with beam-sdks-java-extensions-sql-zetasql
   
   For this issue, today I made a release of Linkage Checker 1.1.3 and I just 
raised a PR https://github.com/apache/beam/pull/10721 . 
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379138)
Time Spent: 3h 10m  (was: 3h)

> Upgrade Jackson to version 2.10.2
> -
>
> Key: BEAM-9162
> URL: https://issues.apache.org/jira/browse/BEAM-9162
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Jackson has a new way to deal with [deserialization security 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10] in 
> 2.10.x so worth the upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9223) Upgrade Linkage Checker 1.1.3

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9223?focusedWorklogId=379137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379137
 ]

ASF GitHub Bot logged work on BEAM-9223:


Author: ASF GitHub Bot
Created on: 30/Jan/20 03:39
Start Date: 30/Jan/20 03:39
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10721: [BEAM-9223] Upgrade 
Linkage Checker 1.1.3
URL: https://github.com/apache/beam/pull/10721#issuecomment-580069481
 
 
   R: @iemejia @lukecwik 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379137)
Remaining Estimate: 0h
Time Spent: 10m

> Upgrade Linkage Checker 1.1.3
> -
>
> Key: BEAM-9223
> URL: https://issues.apache.org/jira/browse/BEAM-9223
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/CHANGELOG.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9223) Upgrade Linkage Checker 1.1.3

[Tomo Suzuki (Jira)]

Tomo Suzuki created BEAM-9223:
-

 Summary: Upgrade Linkage Checker 1.1.3
 Key: BEAM-9223
 URL: https://issues.apache.org/jira/browse/BEAM-9223
 Project: Beam
  Issue Type: Bug
  Components: build-system
Reporter: Tomo Suzuki
Assignee: Tomo Suzuki


https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/CHANGELOG.md



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379132
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:37
Start Date: 30/Jan/20 02:37
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10720: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10720#issuecomment-580057494
 
 
   Thanks Ankur! I'll pick it up if RC1 validation fails.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379132)
Time Spent: 1.5h  (was: 1h 20m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379131=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379131
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:34
Start Date: 30/Jan/20 02:34
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #10720: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10720#issuecomment-580057016
 
 
   R: @boyuanzz 
   cc: @ananvay @aaltay 
   
   Sorry for the trouble.
   Please cherrypick this PR if you rebuild RC.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379131)
Time Spent: 1h 20m  (was: 1h 10m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379130=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379130
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:33
Start Date: 30/Jan/20 02:33
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #10720: [BEAM-9220] 
Adding argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10720
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Created] (BEAM-9222) Add distributed read pattern

[Reza ardeshir rokni (Jira)]

Reza ardeshir rokni created BEAM-9222:
-

 Summary: Add distributed read pattern 
 Key: BEAM-9222
 URL: https://issues.apache.org/jira/browse/BEAM-9222
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Reza ardeshir rokni


A nice pattern that would be great to add to the Apache Beam patterns page:

[https://nl.devoteam.com/en/blog-post/querying-jdbc-database-parallel-google-dataflow-apache-beam/]
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=379122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379122
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:02
Start Date: 30/Jan/20 02:02
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-580049923
 
 
   Run XVR_Flink PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379122)
Time Spent: 21h 40m  (was: 21.5h)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 21h 40m
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=379124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379124
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:02
Start Date: 30/Jan/20 02:02
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-580050005
 
 
   Run Java_Examples_Dataflow PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379124)
Time Spent: 22h  (was: 21h 50m)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 22h
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=379123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379123
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Jan/20 02:02
Start Date: 30/Jan/20 02:02
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-580049972
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379123)
Time Spent: 21h 50m  (was: 21h 40m)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 21h 50m
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379121
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:35
Start Date: 30/Jan/20 01:35
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #10710: 
[BEAM-9215] Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379121)
Time Spent: 2h  (was: 1h 50m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379120
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:35
Start Date: 30/Jan/20 01:35
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-580044077
 
 
   "Run Java_Examples_Dataflow PreCommit" failures seems to be unrelated.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379120)
Time Spent: 1h 50m  (was: 1h 40m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8271) StateGetRequest/Response continuation_token should be string

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8271?focusedWorklogId=379117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379117
 ]

ASF GitHub Bot logged work on BEAM-8271:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:28
Start Date: 30/Jan/20 01:28
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10595: [BEAM-8271] Properly 
encode/decode StateGetRequest/Response continuation token
URL: https://github.com/apache/beam/pull/10595#issuecomment-580042571
 
 
   Ok, I’ll have another look at it.
   
   On Wed, Jan 29, 2020 at 5:06 PM Robert Bradshaw 
   wrote:
   
   > It's bytes because non-trivial runners may serialize arbitrary data into
   > this field used to continue the iterable. (Ids are strings because they're
   > just used to compare against, and also the type of proto map keys
   > constrains us here.) There shouldn't be any manipulation of this token
   > except for passing it back on the client side at least--just accepting it
   > and passing it back.
   >
   > It looks like Java treats this as a BytesString. I still think we should
   > do the same.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379117)
Time Spent: 50m  (was: 40m)

> StateGetRequest/Response continuation_token should be string
> 
>
> Key: BEAM-8271
> URL: https://issues.apache.org/jira/browse/BEAM-8271
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I've been working on adding typing to the python code and I came across a 
> discrepancy between regarding the type of the continuation token.  The .proto 
> defines it as bytes, but the code treats it as a string (i.e. unicode):
>  
> {code:java}
> // A request to get state.
> message StateGetRequest {
>   // (Optional) If specified, signals to the runner that the response
>   // should resume from the following continuation token.
>   //
>   // If unspecified, signals to the runner that the response should start
>   // from the beginning of the logical continuable stream.
>   bytes continuation_token = 1;
> }
> // A response to get state representing a logical byte stream which can be
> // continued using the state API.
> message StateGetResponse {
>   // (Optional) If specified, represents a token which can be used with the
>   // state API to get the next chunk of this logical byte stream. The end of
>   // the logical byte stream is signalled by this field being unset.
>   bytes continuation_token = 1;
>   // Represents a part of a logical byte stream. Elements within
>   // the logical byte stream are encoded in the nested context and
>   // concatenated together.
>   bytes data = 2;
> } 
> {code}
> From FnApiRunner.StateServicer:
> {code:python}
> def blocking_get(self, state_key, continuation_token=None):
>   with self._lock:
> full_state = self._state[self._to_key(state_key)]
> if self._use_continuation_tokens:
>   # The token is "nonce:index".
>   if not continuation_token:
> token_base = 'token_%x' % len(self._continuations)
> self._continuations[token_base] = tuple(full_state)
> return b'', '%s:0' % token_base
>   else:
> token_base, index = continuation_token.split(':')
> ix = int(index)
> full_state = self._continuations[token_base]
> if ix == len(full_state):
>   return b'', None
> else:
>   return full_state[ix], '%s:%d' % (token_base, ix + 1)
> else:
>   assert not continuation_token
>   return b''.join(full_state), None
> {code}
> This could be a problem in python3.  
> All other id values are string, whereas bytes is reserved for data, so I 
> think that the proto should be changed to string. 



--
This message was 

[jira] [Work logged] (BEAM-9213) FlinkRunner ignores --flink_submit_uber_jar when master unset

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9213?focusedWorklogId=379113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379113
 ]

ASF GitHub Bot logged work on BEAM-9213:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:11
Start Date: 30/Jan/20 01:11
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #10708: [BEAM-9213] 
throw error when flink_submit_uber_jar and not flink_master
URL: https://github.com/apache/beam/pull/10708#discussion_r372715663
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/flink_runner.py
 ##
 @@ -51,8 +51,11 @@ def default_job_server(self, options):
 flink_master = self.add_http_scheme(
 flink_options.flink_master)
 flink_options.flink_master = flink_master
-if (flink_options.flink_submit_uber_jar
-and flink_master not in MAGIC_HOST_NAMES):
+if flink_options.flink_submit_uber_jar:
+  if flink_master in MAGIC_HOST_NAMES:
+raise ValueError(
+'Cannot use flink_submit_uber_jar with flink_master %s'
+% flink_master)
 
 Review comment:
   On the other hand, I think we may want to make flink_submit_uber_jar the 
default, in which case this logic wouldn't make sense. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379113)
Time Spent: 40m  (was: 0.5h)

> FlinkRunner ignores --flink_submit_uber_jar when master unset
> -
>
> Key: BEAM-9213
> URL: https://issues.apache.org/jira/browse/BEAM-9213
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Instead, an error should be thrown to let the user know that 
> flink_submit_uber_jar is incompatible with auto/local master.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7746) Add type hints to python code

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=379111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379111
 ]

ASF GitHub Bot logged work on BEAM-7746:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:08
Start Date: 30/Jan/20 01:08
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #10591: [BEAM-7746] 
Silence a bunch of errors about "Cannot instantiate abstract class"
URL: https://github.com/apache/beam/pull/10591
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379111)
Time Spent: 59h 10m  (was: 59h)

> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 59h 10m
>  Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484 
> type hints so that I can clearly see what types are required, get completion 
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here:  [https://github.com/apache/beam/pull/9056]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8271) StateGetRequest/Response continuation_token should be string

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8271?focusedWorklogId=379108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379108
 ]

ASF GitHub Bot logged work on BEAM-8271:


Author: ASF GitHub Bot
Created on: 30/Jan/20 01:06
Start Date: 30/Jan/20 01:06
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #10595: [BEAM-8271] 
Properly encode/decode StateGetRequest/Response continuation token
URL: https://github.com/apache/beam/pull/10595#issuecomment-580037170
 
 
   It's bytes because non-trivial runners may serialize arbitrary data into 
this field used to continue the iterable. (Ids are strings because they're just 
used to compare against, and also the type of proto map keys constrains us 
here.) There shouldn't be any manipulation of this token except for passing it 
back on the client side at least--just accepting it and passing it back. 
   
   It looks like Java treats this as a BytesString. I still think we should do 
the same. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379108)
Time Spent: 40m  (was: 0.5h)

> StateGetRequest/Response continuation_token should be string
> 
>
> Key: BEAM-8271
> URL: https://issues.apache.org/jira/browse/BEAM-8271
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I've been working on adding typing to the python code and I came across a 
> discrepancy between regarding the type of the continuation token.  The .proto 
> defines it as bytes, but the code treats it as a string (i.e. unicode):
>  
> {code:java}
> // A request to get state.
> message StateGetRequest {
>   // (Optional) If specified, signals to the runner that the response
>   // should resume from the following continuation token.
>   //
>   // If unspecified, signals to the runner that the response should start
>   // from the beginning of the logical continuable stream.
>   bytes continuation_token = 1;
> }
> // A response to get state representing a logical byte stream which can be
> // continued using the state API.
> message StateGetResponse {
>   // (Optional) If specified, represents a token which can be used with the
>   // state API to get the next chunk of this logical byte stream. The end of
>   // the logical byte stream is signalled by this field being unset.
>   bytes continuation_token = 1;
>   // Represents a part of a logical byte stream. Elements within
>   // the logical byte stream are encoded in the nested context and
>   // concatenated together.
>   bytes data = 2;
> } 
> {code}
> From FnApiRunner.StateServicer:
> {code:python}
> def blocking_get(self, state_key, continuation_token=None):
>   with self._lock:
> full_state = self._state[self._to_key(state_key)]
> if self._use_continuation_tokens:
>   # The token is "nonce:index".
>   if not continuation_token:
> token_base = 'token_%x' % len(self._continuations)
> self._continuations[token_base] = tuple(full_state)
> return b'', '%s:0' % token_base
>   else:
> token_base, index = continuation_token.split(':')
> ix = int(index)
> full_state = self._continuations[token_base]
> if ix == len(full_state):
>   return b'', None
> else:
>   return full_state[ix], '%s:%d' % (token_base, ix + 1)
> else:
>   assert not continuation_token
>   return b''.join(full_state), None
> {code}
> This could be a problem in python3.  
> All other id values are string, whereas bytes is reserved for data, so I 
> think that the proto should be changed to string. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (BEAM-9220) Add use_runner_v2 argument for dataflow

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik closed BEAM-9220.
---
Fix Version/s: 2.20.0
   Resolution: Fixed

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379093
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 30/Jan/20 00:05
Start Date: 30/Jan/20 00:05
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10718: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718#issuecomment-580021967
 
 
   Turns out that we are trying to package up several new things together as a 
v2
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379093)
Time Spent: 50m  (was: 40m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379094
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 30/Jan/20 00:05
Start Date: 30/Jan/20 00:05
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10718: [BEAM-9220] 
Adding argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379094)
Time Spent: 1h  (was: 50m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9162) Upgrade Jackson to version 2.10.2

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9162?focusedWorklogId=379091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379091
 ]

ASF GitHub Bot logged work on BEAM-9162:


Author: ASF GitHub Bot
Created on: 30/Jan/20 00:03
Start Date: 30/Jan/20 00:03
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10643: [BEAM-9162] Upgrade 
Jackson to version 2.10.2
URL: https://github.com/apache/beam/pull/10643#issuecomment-580021293
 
 
   If you want to enumerate all modules then use:
   ```
   ./gradlew -Ppublishing 
-PjavaLinkageArtifactIds=beam-runners-apex,beam-runners-core-construction-java,beam-runners-core-java,beam-runners-direct-java,beam-runners-extensions-java-metrics,beam-runners-flink-1.7,beam-runners-flink-1.7-job-server,beam-runners-flink-1.8,beam-runners-flink-1.8-job-server,beam-runners-flink-1.9,beam-runners-flink-1.9-job-server,beam-runners-gearpump,beam-runners-google-cloud-dataflow-java,beam-runners-java-fn-execution,beam-runners-jet,beam-runners-local-java-core,beam-runners-portability-java,beam-runners-samza,beam-runners-samza-job-server,beam-runners-spark,beam-runners-spark-job-server,beam-sdks-java-bom,beam-sdks-java-core,beam-sdks-java-extensions-euphoria,beam-sdks-java-extensions-google-cloud-platform-core,beam-sdks-java-extensions-join-library,beam-sdks-java-extensions-json-jackson,beam-sdks-java-extensions-kryo,beam-sdks-java-extensions-protobuf,beam-sdks-java-extensions-sketching,beam-sdks-java-extensions-sorter,beam-sdks-java-extensions-sql,beam-sdks-java-extensions-sql-datacatalog,beam-sdks-java-extensions-sql-hcatalog,beam-sdks-java-extensions-sql-jdbc,beam-sdks-java-extensions-sql-perf-tests,beam-sdks-java-extensions-sql-zetasql,beam-sdks-java-extensions-zetasketch,beam-sdks-java-fn-execution,beam-sdks-java-harness,beam-sdks-java-io-amazon-web-services,beam-sdks-java-io-amazon-web-services2,beam-sdks-java-io-amqp,beam-sdks-java-io-cassandra,beam-sdks-java-io-clickhouse,beam-sdks-java-io-common,beam-sdks-java-io-elasticsearch,beam-sdks-java-io-google-cloud-platform,beam-sdks-java-io-hadoop-common,beam-sdks-java-io-hadoop-file-system,beam-sdks-java-io-hadoop-format,beam-sdks-java-io-hbase,beam-sdks-java-io-hcatalog,beam-sdks-java-io-jdbc,beam-sdks-java-io-jms,beam-sdks-java-io-kafka,beam-sdks-java-io-kinesis,beam-sdks-java-io-kudu,beam-sdks-java-io-mongodb,beam-sdks-java-io-mqtt,beam-sdks-java-io-parquet,beam-sdks-java-io-rabbitmq,beam-sdks-java-io-redis,beam-sdks-java-io-solr,beam-sdks-java-io-synthetic,beam-sdks-java-io-tika,beam-sdks-java-io-xml,beam-sdks-java-nexmark
 :checkJavaLinkage
   ```
   
   I believe that is the entire set of modules that aren't archetypes, model, 
or vendored dependencies.
   
   I think there is an issue with beam-sdks-java-extensions-sql-zetasql and 
beam-sdks-java-extensions-zetasketch which you might need to exclude from the 
list above.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379091)
Time Spent: 2h 50m  (was: 2h 40m)

> Upgrade Jackson to version 2.10.2
> -
>
> Key: BEAM-9162
> URL: https://issues.apache.org/jira/browse/BEAM-9162
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Jackson has a new way to deal with [deserialization security 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10] in 
> 2.10.x so worth the upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9162) Upgrade Jackson to version 2.10.2

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9162?focusedWorklogId=379092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379092
 ]

ASF GitHub Bot logged work on BEAM-9162:


Author: ASF GitHub Bot
Created on: 30/Jan/20 00:03
Start Date: 30/Jan/20 00:03
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10643: [BEAM-9162] Upgrade 
Jackson to version 2.10.2
URL: https://github.com/apache/beam/pull/10643#issuecomment-580021293
 
 
   If you want to check linkage for all relevant modules then use:
   ```
   ./gradlew -Ppublishing 
-PjavaLinkageArtifactIds=beam-runners-apex,beam-runners-core-construction-java,beam-runners-core-java,beam-runners-direct-java,beam-runners-extensions-java-metrics,beam-runners-flink-1.7,beam-runners-flink-1.7-job-server,beam-runners-flink-1.8,beam-runners-flink-1.8-job-server,beam-runners-flink-1.9,beam-runners-flink-1.9-job-server,beam-runners-gearpump,beam-runners-google-cloud-dataflow-java,beam-runners-java-fn-execution,beam-runners-jet,beam-runners-local-java-core,beam-runners-portability-java,beam-runners-samza,beam-runners-samza-job-server,beam-runners-spark,beam-runners-spark-job-server,beam-sdks-java-bom,beam-sdks-java-core,beam-sdks-java-extensions-euphoria,beam-sdks-java-extensions-google-cloud-platform-core,beam-sdks-java-extensions-join-library,beam-sdks-java-extensions-json-jackson,beam-sdks-java-extensions-kryo,beam-sdks-java-extensions-protobuf,beam-sdks-java-extensions-sketching,beam-sdks-java-extensions-sorter,beam-sdks-java-extensions-sql,beam-sdks-java-extensions-sql-datacatalog,beam-sdks-java-extensions-sql-hcatalog,beam-sdks-java-extensions-sql-jdbc,beam-sdks-java-extensions-sql-perf-tests,beam-sdks-java-extensions-sql-zetasql,beam-sdks-java-extensions-zetasketch,beam-sdks-java-fn-execution,beam-sdks-java-harness,beam-sdks-java-io-amazon-web-services,beam-sdks-java-io-amazon-web-services2,beam-sdks-java-io-amqp,beam-sdks-java-io-cassandra,beam-sdks-java-io-clickhouse,beam-sdks-java-io-common,beam-sdks-java-io-elasticsearch,beam-sdks-java-io-google-cloud-platform,beam-sdks-java-io-hadoop-common,beam-sdks-java-io-hadoop-file-system,beam-sdks-java-io-hadoop-format,beam-sdks-java-io-hbase,beam-sdks-java-io-hcatalog,beam-sdks-java-io-jdbc,beam-sdks-java-io-jms,beam-sdks-java-io-kafka,beam-sdks-java-io-kinesis,beam-sdks-java-io-kudu,beam-sdks-java-io-mongodb,beam-sdks-java-io-mqtt,beam-sdks-java-io-parquet,beam-sdks-java-io-rabbitmq,beam-sdks-java-io-redis,beam-sdks-java-io-solr,beam-sdks-java-io-synthetic,beam-sdks-java-io-tika,beam-sdks-java-io-xml,beam-sdks-java-nexmark
 :checkJavaLinkage
   ```
   
   I believe that is the entire set of modules that aren't archetypes, model, 
or vendored dependencies.
   
   I think there is an issue with beam-sdks-java-extensions-sql-zetasql and 
beam-sdks-java-extensions-zetasketch which you might need to exclude from the 
list above.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379092)
Time Spent: 3h  (was: 2h 50m)

> Upgrade Jackson to version 2.10.2
> -
>
> Key: BEAM-9162
> URL: https://issues.apache.org/jira/browse/BEAM-9162
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Jackson has a new way to deal with [deserialization security 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10] in 
> 2.10.x so worth the upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7847) Generate Python SDK docs using Python 3

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7847?focusedWorklogId=379084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379084
 ]

ASF GitHub Bot logged work on BEAM-7847:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:28
Start Date: 29/Jan/20 23:28
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #10141: [BEAM-7847] 
enabled to generate SDK docs with Python3
URL: https://github.com/apache/beam/pull/10141#issuecomment-580012030
 
 
   Thanks a lot, @lazylynx and @udim for landing this change.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379084)
Time Spent: 9h  (was: 8h 50m)

> Generate Python SDK docs using Python 3 
> 
>
> Key: BEAM-7847
> URL: https://issues.apache.org/jira/browse/BEAM-7847
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> Currently scripts/generate_pydoc.sh script fails on Python 3 with 
> "RuntimeError: empty_like method already has a docstring" errors:
> {noformat}
> pip install -e .[gcp,test]
> pip install Sphinx==1.6.5
> pip install sphinx_rtd_theme==0.2.4
> ./scripts/generate_pydoc.sh
> /home/valentyn/projects/beam/beam/beam/sdks/python/target/docs/source/apache_beam.testing.benchmarks.nexmark.queries.query0.rst:4:
>  WARNING: autodoc: failed to import module 
> 'apache_beam.testing.benchmarks.nexmark.queries.query0'; the following 
> exception was raised:
> Traceback (most recent call last):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/sphinx/ext/autodoc.py",
>  line 658, in import_object
> __import__(self.modname)
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py", 
> line 98, in 
> from apache_beam import io
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/__init__.py",
>  line 22, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/avroio.py",
>  line 61, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/filebasedsink.py",
>  line 34, in 
> from apache_beam.io import iobase
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/iobase.py",
>  line 50, in 
> from apache_beam.transforms import core
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/__init__.py",
>  line 29, in 
> from apache_beam.transforms.util import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 228, in 
> class _BatchSizeEstimator(object):
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 359, in _BatchSizeEstimator
> import numpy as np
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/__init__.py", 
> line 142, in 
> from . import core
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/__init__.py",
>  line 17, in 
> from . import multiarray
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/multiarray.py",
>  line 78, in 
> def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 203, in decorator
> docs_from_dispatcher=docs_from_dispatcher)(implementation)
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 159, in decorator
> add_docstring(implementation, dispatcher.__doc__)
> RuntimeError: empty_like method already has a docstring
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379083
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:23
Start Date: 29/Jan/20 23:23
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10718: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718#issuecomment-580010338
 
 
   Why?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379083)
Time Spent: 40m  (was: 0.5h)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379082
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:22
Start Date: 29/Jan/20 23:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372683343
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
 Review comment:
   Any reason why we aren't using the paths specified in 
https://cwiki.apache.org/confluence/display/INFRA/JDK+Installation+Matrix?
   
   If we aren't using an ASF jenkins machine VM image as our base then that is 
fine but can we define the constants somewhere and share them across the 
relevant locations?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379082)
Time Spent: 17h 20m  (was: 17h 10m)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17h 20m
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9221) Python precommit tests should catch errors when Python SDK dependency chain has conflicting dependencies.

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-9221:
-

 Summary: Python precommit tests should catch errors when Python 
SDK dependency chain has conflicting dependencies. 
 Key: BEAM-9221
 URL: https://issues.apache.org/jira/browse/BEAM-9221
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379080
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:20
Start Date: 29/Jan/20 23:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372683343
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
 Review comment:
   Any reason why we aren't using the paths specified in 
https://cwiki.apache.org/confluence/display/INFRA/JDK+Installation+Matrix?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379080)
Time Spent: 17h 10m  (was: 17h)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379078
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:20
Start Date: 29/Jan/20 23:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372684079
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
+  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
   publishers {
 archiveJunit('**/build/test-results/**/*.xml')
   }
 
   steps {
+gradle {
+  rootBuildScriptDir(commonJobProperties.checkoutDir)
+  tasks(':runners:google-cloud-dataflow-java:testJar')
+  
tasks(':runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar')
+  switches("-Dorg.gradle.java.home=${JAVA_8_HOME}")
+}
+
 gradle {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
   tasks(':runners:google-cloud-dataflow-java:validatesJava11Runner')
 
 Review comment:
   Why do we have a 
[validatesJava11Runner](https://github.com/apache/beam/blob/a39b69cc9ed5c930e59c863ccb66ba61351fe1e3/runners/google-cloud-dataflow-java/build.gradle#L322)
 task? (Shouldn't we just invoke the 
[validatesRunner](https://github.com/apache/beam/blob/a39b69cc9ed5c930e59c863ccb66ba61351fe1e3/runners/google-cloud-dataflow-java/build.gradle#L316)
 task?)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379078)
Time Spent: 17h  (was: 16h 50m)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379079
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:20
Start Date: 29/Jan/20 23:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372683270
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
 
 Review comment:
   Can we preferably use a Jenkins environment variable for the [Java home 
locations](https://issues.apache.org/jira/browse/BEAM-8919) here and in the 
[direct runner Java 11 
case](https://github.com/apache/beam/blob/19dcb5c10c270b367cb4f897090eae2f700b11a6/.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct_Java11.groovy#L30)?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379079)
Time Spent: 17h  (was: 16h 50m)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=379081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379081
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:20
Start Date: 29/Jan/20 23:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r372684505
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
+  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
   publishers {
 archiveJunit('**/build/test-results/**/*.xml')
   }
 
   steps {
+gradle {
+  rootBuildScriptDir(commonJobProperties.checkoutDir)
 
 Review comment:
   Can we override the 
[javaVersion](https://github.com/apache/beam/blob/003c89135754aac3f7e80f50523c8f7caa4ffcee/gradle.properties#L29)
 property instead of splitting these into two commands?
   
   This would lead us to using the Java 11 compiler with a source and class 
target of version 8 or do we want to explicitly guarantee that we are using the 
Java 8 compiler?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379081)
Time Spent: 17h 10m  (was: 17h)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9009) Python: Add per-unit test timeout

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9009?focusedWorklogId=379077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379077
 ]

ASF GitHub Bot logged work on BEAM-9009:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:14
Start Date: 29/Jan/20 23:14
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10437: [BEAM-9009] Add 
pytest-timeout plugin, set timeout
URL: https://github.com/apache/beam/pull/10437#issuecomment-580007162
 
 
   R: @ibzib 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379077)
Time Spent: 0.5h  (was: 20m)

> Python: Add per-unit test timeout
> -
>
> Key: BEAM-9009
> URL: https://issues.apache.org/jira/browse/BEAM-9009
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Initially proposing a generous timeout of 10m, in part due to Jenkins 
> slowness.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379076
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:11
Start Date: 29/Jan/20 23:11
Worklog Time Spent: 10m 
  Work Description: ananvay commented on issue #10718: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718#issuecomment-580006174
 
 
   Thanks Ankur! LGTM
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379076)
Time Spent: 0.5h  (was: 20m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379073
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:03
Start Date: 29/Jan/20 23:03
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #10718: [BEAM-9220] 
Adding argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=379074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379074
 ]

ASF GitHub Bot logged work on BEAM-9220:


Author: ASF GitHub Bot
Created on: 29/Jan/20 23:03
Start Date: 29/Jan/20 23:03
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #10718: [BEAM-9220] Adding 
argument use_runner_v2 for dataflow unified worker
URL: https://github.com/apache/beam/pull/10718#issuecomment-580003651
 
 
   R: @ananvay 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379074)
Time Spent: 20m  (was: 10m)

> Add use_runner_v2 argument for dataflow
> ---
>
> Key: BEAM-9220
> URL: https://issues.apache.org/jira/browse/BEAM-9220
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow
>Reporter: Ankur Goenka
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9220) Add use_runner_v2 argument for dataflow

[Ankur Goenka (Jira)]

Ankur Goenka created BEAM-9220:
--

 Summary: Add use_runner_v2 argument for dataflow
 Key: BEAM-9220
 URL: https://issues.apache.org/jira/browse/BEAM-9220
 Project: Beam
  Issue Type: New Feature
  Components: runner-dataflow
Reporter: Ankur Goenka






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7847) Generate Python SDK docs using Python 3

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7847?focusedWorklogId=379070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379070
 ]

ASF GitHub Bot logged work on BEAM-7847:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:57
Start Date: 29/Jan/20 22:57
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10141: [BEAM-7847] 
enabled to generate SDK docs with Python3
URL: https://github.com/apache/beam/pull/10141
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379070)
Time Spent: 8h 50m  (was: 8h 40m)

> Generate Python SDK docs using Python 3 
> 
>
> Key: BEAM-7847
> URL: https://issues.apache.org/jira/browse/BEAM-7847
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Major
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> Currently scripts/generate_pydoc.sh script fails on Python 3 with 
> "RuntimeError: empty_like method already has a docstring" errors:
> {noformat}
> pip install -e .[gcp,test]
> pip install Sphinx==1.6.5
> pip install sphinx_rtd_theme==0.2.4
> ./scripts/generate_pydoc.sh
> /home/valentyn/projects/beam/beam/beam/sdks/python/target/docs/source/apache_beam.testing.benchmarks.nexmark.queries.query0.rst:4:
>  WARNING: autodoc: failed to import module 
> 'apache_beam.testing.benchmarks.nexmark.queries.query0'; the following 
> exception was raised:
> Traceback (most recent call last):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/sphinx/ext/autodoc.py",
>  line 658, in import_object
> __import__(self.modname)
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py", 
> line 98, in 
> from apache_beam import io
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/__init__.py",
>  line 22, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/avroio.py",
>  line 61, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/filebasedsink.py",
>  line 34, in 
> from apache_beam.io import iobase
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/iobase.py",
>  line 50, in 
> from apache_beam.transforms import core
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/__init__.py",
>  line 29, in 
> from apache_beam.transforms.util import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 228, in 
> class _BatchSizeEstimator(object):
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 359, in _BatchSizeEstimator
> import numpy as np
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/__init__.py", 
> line 142, in 
> from . import core
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/__init__.py",
>  line 17, in 
> from . import multiarray
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/multiarray.py",
>  line 78, in 
> def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 203, in decorator
> docs_from_dispatcher=docs_from_dispatcher)(implementation)
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 159, in decorator
> add_docstring(implementation, dispatcher.__doc__)
> RuntimeError: empty_like method already has a docstring
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7847) Generate Python SDK docs using Python 3

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7847?focusedWorklogId=379069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379069
 ]

ASF GitHub Bot logged work on BEAM-7847:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:56
Start Date: 29/Jan/20 22:56
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10141: [BEAM-7847] enabled to 
generate SDK docs with Python3
URL: https://github.com/apache/beam/pull/10141#issuecomment-580001214
 
 
   Tests have passed on 691071a, this looks ready to merge.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379069)
Time Spent: 8h 40m  (was: 8.5h)

> Generate Python SDK docs using Python 3 
> 
>
> Key: BEAM-7847
> URL: https://issues.apache.org/jira/browse/BEAM-7847
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> Currently scripts/generate_pydoc.sh script fails on Python 3 with 
> "RuntimeError: empty_like method already has a docstring" errors:
> {noformat}
> pip install -e .[gcp,test]
> pip install Sphinx==1.6.5
> pip install sphinx_rtd_theme==0.2.4
> ./scripts/generate_pydoc.sh
> /home/valentyn/projects/beam/beam/beam/sdks/python/target/docs/source/apache_beam.testing.benchmarks.nexmark.queries.query0.rst:4:
>  WARNING: autodoc: failed to import module 
> 'apache_beam.testing.benchmarks.nexmark.queries.query0'; the following 
> exception was raised:
> Traceback (most recent call last):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/sphinx/ext/autodoc.py",
>  line 658, in import_object
> __import__(self.modname)
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/__init__.py", 
> line 98, in 
> from apache_beam import io
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/__init__.py",
>  line 22, in 
> from apache_beam.io.avroio import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/avroio.py",
>  line 61, in 
> from apache_beam.io import filebasedsink
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/filebasedsink.py",
>  line 34, in 
> from apache_beam.io import iobase
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/io/iobase.py",
>  line 50, in 
> from apache_beam.transforms import core
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/__init__.py",
>  line 29, in 
> from apache_beam.transforms.util import *
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 228, in 
> class _BatchSizeEstimator(object):
>   File 
> "/home/valentyn/projects/beam/beam/beam/sdks/python/apache_beam/transforms/util.py",
>  line 359, in _BatchSizeEstimator
> import numpy as np
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/__init__.py", 
> line 142, in 
> from . import core
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/__init__.py",
>  line 17, in 
> from . import multiarray
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/multiarray.py",
>  line 78, in 
> def empty_like(prototype, dtype=None, order=None, subok=None, shape=None):
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 203, in decorator
> docs_from_dispatcher=docs_from_dispatcher)(implementation)
>   File 
> "/home/valentyn/tmp/venv/py3/lib/python3.6/site-packages/numpy/core/overrides.py",
>  line 159, in decorator
> add_docstring(implementation, dispatcher.__doc__)
> RuntimeError: empty_like method already has a docstring
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8298) Implement state caching for side inputs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8298?focusedWorklogId=379068=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379068
 ]

ASF GitHub Bot logged work on BEAM-8298:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:56
Start Date: 29/Jan/20 22:56
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10705: [BEAM-8298] 
Implement side input caching.
URL: https://github.com/apache/beam/pull/10705#issuecomment-580001207
 
 
   Seeing test failures:
   ```
   13:11:10 AttributeError: '_IterableFromIterator' object has no attribute 
'extend'
   ```
   
   and linter problems:
   ```
   12:54:20 > Task :sdks:python:test-suites:tox:py37:lintPy37
   12:54:20 * Module apache_beam.runners.worker.sdk_worker_test
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:162:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   token=b'state_token1',
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:163:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   
user_state=beam_fn_api_pb2.ProcessBundleRequest.CacheToken.UserState())
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:165:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   token=b'state_token2',
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:166:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   
user_state=beam_fn_api_pb2.ProcessBundleRequest.CacheToken.UserState())
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:168:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   token=b'side1_token1',
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:169:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   
side_input=beam_fn_api_pb2.ProcessBundleRequest.CacheToken.SideInput(
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:172:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   token=b'side1_token2',
   12:54:20   ^ | (bad-continuation)
   12:54:20 apache_beam/runners/worker/sdk_worker_test.py:173:0: C0330: Wrong 
hanging indentation (add 2 spaces).
   12:54:20   
side_input=beam_fn_api_pb2.ProcessBundleRequest.CacheToken.SideInput(
   12:54:20   ^ | (bad-continuation)
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379068)
Time Spent: 2h 40m  (was: 2.5h)

> Implement state caching for side inputs
> ---
>
> Key: BEAM-8298
> URL: https://issues.apache.org/jira/browse/BEAM-8298
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-py-harness
>Reporter: Maximilian Michels
>Assignee: Jing Chen
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Caching is currently only implemented for user state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=379066=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379066
 ]

ASF GitHub Bot logged work on BEAM-8889:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:47
Start Date: 29/Jan/20 22:47
Worklog Time Spent: 10m 
  Work Description: veblush commented on pull request #10617: [BEAM-8889] 
adding gRPC connectivity to Beam/GCS connector
URL: https://github.com/apache/beam/pull/10617#discussion_r372673929
 
 

 ##
 File path: 
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
 ##
 @@ -427,23 +449,29 @@ public WritableByteChannel create(GcsPath path, String 
type) throws IOException
*/
   public WritableByteChannel create(GcsPath path, String type, Integer 
uploadBufferSizeBytes)
   throws IOException {
-GoogleCloudStorageWriteChannel channel =
-new GoogleCloudStorageWriteChannel(
-executorService,
-storageClient,
-new ClientRequestHelper<>(),
-path.getBucket(),
-path.getObject(),
-type,
-/* kmsKeyName= */ null,
-AsyncWriteChannelOptions.newBuilder().build(),
-new ObjectWriteConditions(),
-Collections.emptyMap());
-if (uploadBufferSizeBytes != null) {
-  channel.setUploadBufferSize(uploadBufferSizeBytes);
+if (uploadBufferSizeBytes == null) {
 
 Review comment:
   I think this can be address by passing `CreateObjectOptions` to 
`googleCloudStorage.create`. `CreateObjectOptions` has a way to configure 
`ContentType`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379066)
Remaining Estimate: 164.5h  (was: 164h 40m)
Time Spent: 3.5h  (was: 3h 20m)

> Make GcsUtil use GoogleCloudStorage
> ---
>
> Key: BEAM-8889
> URL: https://issues.apache.org/jira/browse/BEAM-8889
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.16.0
>Reporter: Esun Kim
>Assignee: VASU NORI
>Priority: Major
>  Labels: gcs
>   Original Estimate: 168h
>  Time Spent: 3.5h
>  Remaining Estimate: 164.5h
>
> [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java]
>  is a primary class to access Google Cloud Storage on Apache Beam. Current 
> implementation directly creates GoogleCloudStorageReadChannel and 
> GoogleCloudStorageWriteChannel by itself to read and write GCS data rather 
> than using 
> [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java]
>  which is an abstract class providing basic IO capability which eventually 
> creates channel objects. This request is about updating GcsUtil to use 
> GoogleCloudStorage to create read and write channel, which is expected 
> flexible because it can easily pick up the new change; e.g. new channel 
> implementation using new protocol without code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8042) Parsing of aggregate query fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8042?focusedWorklogId=379064=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379064
 ]

ASF GitHub Bot logged work on BEAM-8042:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:44
Start Date: 29/Jan/20 22:44
Worklog Time Spent: 10m 
  Work Description: kanterov commented on issue #10649: [BEAM-8042] 
[ZetaSQL] Fix aggregate column reference
URL: https://github.com/apache/beam/pull/10649#issuecomment-579996998
 
 
   @11moon11 no other comments, was asking just of curiosity 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379064)
Time Spent: 3h 20m  (was: 3h 10m)

> Parsing of aggregate query fails
> 
>
> Key: BEAM-8042
> URL: https://issues.apache.org/jira/browse/BEAM-8042
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kirill Kozlov
>Priority: Critical
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> {code}
>   @Rule
>   public TestPipeline pipeline = 
> TestPipeline.fromOptions(createPipelineOptions());
>   private static PipelineOptions createPipelineOptions() {
> BeamSqlPipelineOptions opts = 
> PipelineOptionsFactory.create().as(BeamSqlPipelineOptions.class);
> opts.setPlannerName(ZetaSQLQueryPlanner.class.getName());
> return opts;
>   }
>   @Test
>   public void testAggregate() {
> Schema inputSchema = Schema.builder()
> .addByteArrayField("id")
> .addInt64Field("has_f1")
> .addInt64Field("has_f2")
> .addInt64Field("has_f3")
> .addInt64Field("has_f4")
> .addInt64Field("has_f5")
> .addInt64Field("has_f6")
> .build();
> String sql = "SELECT \n" +
> "  id, \n" +
> "  COUNT(*) as count, \n" +
> "  SUM(has_f1) as f1_count, \n" +
> "  SUM(has_f2) as f2_count, \n" +
> "  SUM(has_f3) as f3_count, \n" +
> "  SUM(has_f4) as f4_count, \n" +
> "  SUM(has_f5) as f5_count, \n" +
> "  SUM(has_f6) as f6_count  \n" +
> "FROM PCOLLECTION \n" +
> "GROUP BY id";
> pipeline
> .apply(Create.empty(inputSchema))
> .apply(SqlTransform.query(sql));
> pipeline.run();
>   }
> {code}
> {code}
> Caused by: java.lang.RuntimeException: Error while applying rule 
> AggregateProjectMergeRule, args 
> [rel#553:LogicalAggregate.NONE(input=RelSubset#552,group={0},f1=COUNT(),f2=SUM($2),f3=SUM($3),f4=SUM($4),f5=SUM($5),f6=SUM($6),f7=SUM($7)),
>  
> rel#551:LogicalProject.NONE(input=RelSubset#550,key=$0,f1=$1,f2=$2,f3=$3,f4=$4,f5=$5,f6=$6)]
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:232)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:340)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:168)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:99)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:104)
>   at 
>   ... 39 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
>   at 
> org.apache.beam.repackaged.sql.com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:58)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.apply(AggregateProjectMergeRule.java:96)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:73)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
>   ... 48 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=379063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379063
 ]

ASF GitHub Bot logged work on BEAM-8280:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:35
Start Date: 29/Jan/20 22:35
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10717: [BEAM-8280] 
Enable type hint annotations
URL: https://github.com/apache/beam/pull/10717
 
 
   Changes default to use type annotations.
   
   Adds a couple of way to disable them:
   - `decorators.disable_type_annotations()` to disable globally.
   - `@decorators.no_annotations` decorator to disable on a specific
   function.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 

[jira] [Work logged] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9203?focusedWorklogId=379062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379062
 ]

ASF GitHub Bot logged work on BEAM-9203:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:33
Start Date: 29/Jan/20 22:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #10699: [BEAM-9203] 
Clarify exceptions in SQL modules
URL: https://github.com/apache/beam/pull/10699#issuecomment-579993357
 
 
   run sql postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379062)
Time Spent: 40m  (was: 0.5h)

> Programmatically determine if SQL exception is user error, unsupported, or bug
> --
>
> Key: BEAM-9203
> URL: https://issues.apache.org/jira/browse/BEAM-9203
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql, dsl-sql-zetasql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Right now there are a lot exceptions thrown by the Calcite SQL dialect and 
> ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are 
> user errors, or just the errors that are unsupported operations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9167) Reduce overhead of Go SDK side metrics

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9167?focusedWorklogId=379060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379060
 ]

ASF GitHub Bot logged work on BEAM-9167:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:33
Start Date: 29/Jan/20 22:33
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10716: [BEAM-9167] Metrics 
extraction refactoring.
URL: https://github.com/apache/beam/pull/10716#issuecomment-579993094
 
 
   Run Go PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379060)
Time Spent: 2h 20m  (was: 2h 10m)

> Reduce overhead of Go SDK side metrics
> --
>
> Key: BEAM-9167
> URL: https://issues.apache.org/jira/browse/BEAM-9167
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Locking overhead due to the global store and local caches of SDK counter data 
> can dominate certain workloads, which means we can do better.
> Instead of having a global store of metrics data to extract counters, we 
> should use per ptransform (or per bundle) counter sets, which would avoid 
> requiring locking per counter operation. The main detriment compared to the 
> current implementation is that a user would need to add their own locking if 
> they were to spawn multiple goroutines to process a Bundle's work in a DoFn.
> Given that self multithreaded DoFns aren't recommended/safe in Java,  largely 
> impossible in Python, and the other beam Go SDK provided constructs (like 
> Iterators and Emitters) are not thread safe, this is a small concern, 
> provided the documentation is clear on this.
> Removing the locking and switching to atomic ops reduces the overhead 
> significantly in example jobs and in the benchmarks.
> A second part of this change should be to move the exec package to manage 
> it's own per bundle state, rather than relying on a global datastore to 
> extract the per bundle,per ptransform values.
> Related: https://issues.apache.org/jira/browse/BEAM-6541 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9203) Programmatically determine if SQL exception is user error, unsupported, or bug

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9203?focusedWorklogId=379061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379061
 ]

ASF GitHub Bot logged work on BEAM-9203:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:33
Start Date: 29/Jan/20 22:33
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #10699: [BEAM-9203] 
Clarify exceptions in SQL modules
URL: https://github.com/apache/beam/pull/10699#issuecomment-579993273
 
 
   All fixed up and green.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379061)
Time Spent: 0.5h  (was: 20m)

> Programmatically determine if SQL exception is user error, unsupported, or bug
> --
>
> Key: BEAM-9203
> URL: https://issues.apache.org/jira/browse/BEAM-9203
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql, dsl-sql-zetasql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Right now there are a lot exceptions thrown by the Calcite SQL dialect and 
> ZetaSQL dialect of Beam SQL. It is hard to catch just the errors that are 
> user errors, or just the errors that are unsupported operations.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379058
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:29
Start Date: 29/Jan/20 22:29
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579991756
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379058)
Time Spent: 1h 40m  (was: 1.5h)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379057=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379057
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:28
Start Date: 29/Jan/20 22:28
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579991403
 
 
   Hmm, linter didn't like when I raised an exception from the finally block. 
So had to move it out.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379057)
Time Spent: 1.5h  (was: 1h 20m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379051
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:09
Start Date: 29/Jan/20 22:09
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579984038
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379051)
Time Spent: 1h 20m  (was: 1h 10m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379050
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:08
Start Date: 29/Jan/20 22:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579983716
 
 
   Thanks. Yeah, adding to finally is better. Updated.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379050)
Time Spent: 1h 10m  (was: 1h)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9167) Reduce overhead of Go SDK side metrics

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9167?focusedWorklogId=379048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379048
 ]

ASF GitHub Bot logged work on BEAM-9167:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:04
Start Date: 29/Jan/20 22:04
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10716: [BEAM-9167] Metrics 
extraction refactoring.
URL: https://github.com/apache/beam/pull/10716#issuecomment-579981995
 
 
   Run Go PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379048)
Time Spent: 2h 10m  (was: 2h)

> Reduce overhead of Go SDK side metrics
> --
>
> Key: BEAM-9167
> URL: https://issues.apache.org/jira/browse/BEAM-9167
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Locking overhead due to the global store and local caches of SDK counter data 
> can dominate certain workloads, which means we can do better.
> Instead of having a global store of metrics data to extract counters, we 
> should use per ptransform (or per bundle) counter sets, which would avoid 
> requiring locking per counter operation. The main detriment compared to the 
> current implementation is that a user would need to add their own locking if 
> they were to spawn multiple goroutines to process a Bundle's work in a DoFn.
> Given that self multithreaded DoFns aren't recommended/safe in Java,  largely 
> impossible in Python, and the other beam Go SDK provided constructs (like 
> Iterators and Emitters) are not thread safe, this is a small concern, 
> provided the documentation is clear on this.
> Removing the locking and switching to atomic ops reduces the overhead 
> significantly in example jobs and in the benchmarks.
> A second part of this change should be to move the exec package to manage 
> it's own per bundle state, rather than relying on a global datastore to 
> extract the per bundle,per ptransform values.
> Related: https://issues.apache.org/jira/browse/BEAM-6541 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9167) Reduce overhead of Go SDK side metrics

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9167?focusedWorklogId=379047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379047
 ]

ASF GitHub Bot logged work on BEAM-9167:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:02
Start Date: 29/Jan/20 22:02
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10716: [BEAM-9167] Metrics 
extraction refactoring.
URL: https://github.com/apache/beam/pull/10716#issuecomment-579981037
 
 
   R: @youngoli 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379047)
Time Spent: 2h  (was: 1h 50m)

> Reduce overhead of Go SDK side metrics
> --
>
> Key: BEAM-9167
> URL: https://issues.apache.org/jira/browse/BEAM-9167
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Locking overhead due to the global store and local caches of SDK counter data 
> can dominate certain workloads, which means we can do better.
> Instead of having a global store of metrics data to extract counters, we 
> should use per ptransform (or per bundle) counter sets, which would avoid 
> requiring locking per counter operation. The main detriment compared to the 
> current implementation is that a user would need to add their own locking if 
> they were to spawn multiple goroutines to process a Bundle's work in a DoFn.
> Given that self multithreaded DoFns aren't recommended/safe in Java,  largely 
> impossible in Python, and the other beam Go SDK provided constructs (like 
> Iterators and Emitters) are not thread safe, this is a small concern, 
> provided the documentation is clear on this.
> Removing the locking and switching to atomic ops reduces the overhead 
> significantly in example jobs and in the benchmarks.
> A second part of this change should be to move the exec package to manage 
> it's own per bundle state, rather than relying on a global datastore to 
> extract the per bundle,per ptransform values.
> Related: https://issues.apache.org/jira/browse/BEAM-6541 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=379045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379045
 ]

ASF GitHub Bot logged work on BEAM-8889:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:01
Start Date: 29/Jan/20 22:01
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #10617: 
[BEAM-8889] adding gRPC connectivity to Beam/GCS connector
URL: https://github.com/apache/beam/pull/10617#discussion_r372655202
 
 

 ##
 File path: 
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java
 ##
 @@ -427,23 +449,29 @@ public WritableByteChannel create(GcsPath path, String 
type) throws IOException
*/
   public WritableByteChannel create(GcsPath path, String type, Integer 
uploadBufferSizeBytes)
   throws IOException {
-GoogleCloudStorageWriteChannel channel =
-new GoogleCloudStorageWriteChannel(
-executorService,
-storageClient,
-new ClientRequestHelper<>(),
-path.getBucket(),
-path.getObject(),
-type,
-/* kmsKeyName= */ null,
-AsyncWriteChannelOptions.newBuilder().build(),
-new ObjectWriteConditions(),
-Collections.emptyMap());
-if (uploadBufferSizeBytes != null) {
-  channel.setUploadBufferSize(uploadBufferSizeBytes);
+if (uploadBufferSizeBytes == null) {
 
 Review comment:
   Was this addressed ? I see this being used in a number of places and 
probably users can set mime type themselves since this is a pipeline option. 
Can this be a breaking change for some users (or some applications of Beam) ?
   
   For example,
   
https://github.com/apache/beam/blob/35da90a94953597e9e5676e1e1e70f27d2a8f064/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryRowWriter.java#L58
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379045)
Remaining Estimate: 164h 40m  (was: 164h 50m)
Time Spent: 3h 20m  (was: 3h 10m)

> Make GcsUtil use GoogleCloudStorage
> ---
>
> Key: BEAM-8889
> URL: https://issues.apache.org/jira/browse/BEAM-8889
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.16.0
>Reporter: Esun Kim
>Assignee: VASU NORI
>Priority: Major
>  Labels: gcs
>   Original Estimate: 168h
>  Time Spent: 3h 20m
>  Remaining Estimate: 164h 40m
>
> [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java]
>  is a primary class to access Google Cloud Storage on Apache Beam. Current 
> implementation directly creates GoogleCloudStorageReadChannel and 
> GoogleCloudStorageWriteChannel by itself to read and write GCS data rather 
> than using 
> [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java]
>  which is an abstract class providing basic IO capability which eventually 
> creates channel objects. This request is about updating GcsUtil to use 
> GoogleCloudStorage to create read and write channel, which is expected 
> flexible because it can easily pick up the new change; e.g. new channel 
> implementation using new protocol without code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8618) Tear down unused DoFns periodically in Python SDK harness

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8618?focusedWorklogId=379044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379044
 ]

ASF GitHub Bot logged work on BEAM-8618:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:01
Start Date: 29/Jan/20 22:01
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10655: 
[BEAM-8618] Tear down unused DoFns periodically in Python SDK harness.
URL: https://github.com/apache/beam/pull/10655#discussion_r372655074
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
 ##
 @@ -280,6 +283,7 @@ def get(self, instruction_id, bundle_descriptor_id):
 try:
   # pop() is threadsafe
   processor = self.cached_bundle_processors[bundle_descriptor_id].pop()
+  self.last_access_time[bundle_descriptor_id] = time.time()
 except IndexError:
 
 Review comment:
   The strategy here is that the bundle processors which are unused after an 
amount of time will be shutdown. When a bundle processor is created in the 
exception block, there are no cached(unused) bundle processors. The bundle 
processors become unused only when they are added to the cached bundle 
processors list. What about rename last_access_time to 
`cached_bundle_processors_last_access_time` to make it more clear?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379044)
Time Spent: 2h 20m  (was: 2h 10m)

> Tear down unused DoFns periodically in Python SDK harness
> -
>
> Key: BEAM-8618
> URL: https://issues.apache.org/jira/browse/BEAM-8618
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Per the discussion in the ML, detail can be found [1],  the teardown of DoFns 
> should be supported in the portability framework. It happens at two places:
> 1) Upon the control service termination
> 2) Tear down the unused DoFns periodically
> The aim of this JIRA is to add support for tear down the unused DoFns 
> periodically in Python SDK harness.
> [1] 
> https://lists.apache.org/thread.html/0c4a4cf83cf2e35c3dfeb9d906e26cd82d3820968ba6f862f91739e4@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9167) Reduce overhead of Go SDK side metrics

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9167?focusedWorklogId=379046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379046
 ]

ASF GitHub Bot logged work on BEAM-9167:


Author: ASF GitHub Bot
Created on: 29/Jan/20 22:01
Start Date: 29/Jan/20 22:01
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #10716: [BEAM-9167] 
Metrics extraction refactoring.
URL: https://github.com/apache/beam/pull/10716
 
 
   Changes up how metrics are accessed. Instead of having them globally 
available, they're exclusively available on a per-bundle metrics store, which 
reduces the overhead of creating metrics proxies and updating. 
   
   In particular, this change moves the extraction of metrics to the FnAPI 
protos to the exec package, so the metrics package becomes proto agnostic. It's 
also forward looking, in that the metric extractors are typed with int64 to 
distinguish them for eventual inclusion of the float64 variants.
   
   Adds additional benchmarking around specific parts of the work of modifying 
metrics. Benchmark values, as always, are relative to machine, OS, and Go 
version.
   
   Two areas for subsequent work:
   Left as an open question are parts are how users can get access to the store 
(or provide an extractor) for metrics after a job. Missing for this is an 
equivalent "ingestor" for accepting metrics from an outside source for use by 
runners.
   
   For a more near term CL, is actually using the v2 metrics MonitoringInfo 
protos instead of the legacy style. This refactoring makes it easier to support 
both, without the metrics package needing to be tightly coupled to the protos.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379043=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379043
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:56
Start Date: 29/Jan/20 21:56
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579978589
 
 
   @lukecwik Thank you.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379043)
Time Spent: 5h  (was: 4h 50m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description 

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379041
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:51
Start Date: 29/Jan/20 21:51
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10710: [BEAM-9215] Fixes an 
issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579976768
 
 
   Nice catch. Thanks, Cham!
   
   `finally` might also work:
   ```
   prior.addSuppressed(e);
 } finally {
   // We should fail here regardless of whether above channel.close() 
call failed or not.
   throw prior;
 }
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379041)
Time Spent: 1h  (was: 50m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379032=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379032
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975350
 
 
   Run Java HadoopFormatIO Performance Test
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379032)
Time Spent: 4h 10m  (was: 4h)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379035=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379035
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975385
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379035)
Time Spent: 4h 20m  (was: 4h 10m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379038=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379038
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975521
 
 
   Run SQL Postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379038)
Time Spent: 4h 50m  (was: 4h 40m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Work logged] (BEAM-7516) Add a watermark manager for the fn_api_runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7516?focusedWorklogId=379034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379034
 ]

ASF GitHub Bot logged work on BEAM-7516:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #10704: 
[BEAM-7516][BEAM-8823] Refactoring FnApiRunner execution logic
URL: https://github.com/apache/beam/pull/10704#discussion_r372640130
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
 ##
 @@ -590,7 +428,8 @@ def run_stages(self,
 
 try:
   with self.maybe_profile():
-pcoll_buffers = collections.defaultdict(_ListBuffer)  # type: 
DefaultDict[bytes, PartitionableBuffer]
+pcoll_buffers = collections.defaultdict(
+fn_api_runner_execution._ListBuffer)  # type: DefaultDict[bytes, 
fn_api_runner_execution._ListBuffer]
 
 Review comment:
   PartitionableBuffer?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379034)
Time Spent: 7h  (was: 6h 50m)

> Add a watermark manager for the fn_api_runner
> -
>
> Key: BEAM-7516
> URL: https://issues.apache.org/jira/browse/BEAM-7516
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> To track watermarks for each stage



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7516) Add a watermark manager for the fn_api_runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7516?focusedWorklogId=379033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379033
 ]

ASF GitHub Bot logged work on BEAM-7516:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: robertwb commented on pull request #10704: 
[BEAM-7516][BEAM-8823] Refactoring FnApiRunner execution logic
URL: https://github.com/apache/beam/pull/10704#discussion_r372645012
 
 

 ##
 File path: 
sdks/python/apache_beam/runners/portability/fn_api_runner_execution.py
 ##
 @@ -0,0 +1,369 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Set of utilities for execution of a pipeline by the FnApiRunner."""
+
+from __future__ import absolute_import
+
+import collections
+import itertools
+from typing import TYPE_CHECKING
+from typing import Iterator
+from typing import List
+
+from typing_extensions import Protocol
+
+from apache_beam import coders
+from apache_beam.coders.coder_impl import create_InputStream
+from apache_beam.coders.coder_impl import create_OutputStream
+from apache_beam.portability import common_urns
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.runners.portability.fn_api_runner_transforms import 
only_element
+from apache_beam.runners.portability.fn_api_runner_transforms import 
split_buffer_id
+from apache_beam.runners.portability.fn_api_runner_transforms import 
unique_name
+from apache_beam.runners.worker import bundle_processor
+from apache_beam.transforms import trigger
+from apache_beam.transforms.window import GlobalWindow
+from apache_beam.transforms.window import GlobalWindows
+from apache_beam.utils import windowed_value
+
+if TYPE_CHECKING:
+  from apache_beam.transforms.window import BoundedWindow  # pylint: 
disable=ungrouped-imports
+
+
+class Buffer(Protocol):
+  def __iter__(self):
+# type: () -> Iterator[bytes]
+pass
+
+  def append(self, item):
+# type: (bytes) -> None
+pass
+
+
+class PartitionableBuffer(Buffer, Protocol):
+  def partition(self, n):
+# type: (int) -> List[List[bytes]]
+pass
+
+
+class _ListBuffer(List[bytes]):
+  """Used to support parititioning of a list."""
+  def partition(self, n):
+# type: (int) -> List[List[bytes]]
+return [self[k::n] for k in range(n)]
+
+
+class _GroupingBuffer(object):
+  """Used to accumulate groupded (shuffled) results."""
+  def __init__(
+  self,
+  pre_grouped_coder,  # type: coders.Coder
+  post_grouped_coder,  # type: coders.Coder
+  windowing
+  ):
+# type: (...) -> None
+self._key_coder = pre_grouped_coder.key_coder()
+self._pre_grouped_coder = pre_grouped_coder
+self._post_grouped_coder = post_grouped_coder
+self._table = collections.defaultdict(list)  # type: DefaultDict[bytes, 
List[Any]]
+self._windowing = windowing
+self._grouped_output = None  # type: Optional[List[List[bytes]]]
+
+  def append(self, elements_data):
+# type: (bytes) -> None
+if self._grouped_output:
+  raise RuntimeError('Grouping table append after read.')
+input_stream = create_InputStream(elements_data)
+coder_impl = self._pre_grouped_coder.get_impl()
+key_coder_impl = self._key_coder.get_impl()
+# TODO(robertwb): We could optimize this even more by using a
+# window-dropping coder for the data plane.
+is_trivial_windowing = self._windowing.is_default()
+while input_stream.size() > 0:
+  windowed_key_value = coder_impl.decode_from_stream(input_stream, True)
+  key, value = windowed_key_value.value
+  self._table[key_coder_impl.encode(key)].append(
+  value if is_trivial_windowing
+  else windowed_key_value.with_value(value))
+
+  def partition(self, n):
+# type: (int) -> List[List[bytes]]
+""" It is used to partition _GroupingBuffer to N parts. Once it is
+partitioned, it would not be re-partitioned with diff N. Re-partition
+is not supported now.
+"""
+if not self._grouped_output:
+  if self._windowing.is_default():
+globally_window = GlobalWindows.windowed_value(
+ 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379036=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379036
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975426
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379036)
Time Spent: 4.5h  (was: 4h 20m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379037=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379037
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:48
Start Date: 29/Jan/20 21:48
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975485
 
 
   Run Spark ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379037)
Time Spent: 4h 40m  (was: 4.5h)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379031
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:47
Start Date: 29/Jan/20 21:47
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975321
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379031)
Time Spent: 4h  (was: 3h 50m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description 

[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=379030=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379030
 ]

ASF GitHub Bot logged work on BEAM-8691:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:47
Start Date: 29/Jan/20 21:47
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10714: [BEAM-8691] 
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#issuecomment-579975237
 
 
   ping for tests
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379030)
Time Spent: 3h 50m  (was: 3h 40m)

> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:51.523448 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:43.901882 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:30.163557 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:37.979355 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:39.422837 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:06:11.312353 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:45.627449 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:23.823869 
> -
> Please consider upgrading the dependency 
> com.google.cloud.bigtable:bigtable-client-core. 
> The current version is 1.8.0. The latest version is 1.12.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The 

[jira] [Updated] (BEAM-8989) Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-8989:

Affects Version/s: 2.19.0

> Backwards incompatible change in ParDo.getSideInputs (caught by failure when 
> running Apache Nemo quickstart)
> 
>
> Key: BEAM-8989
> URL: https://issues.apache.org/jira/browse/BEAM-8989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0
>Reporter: Luke Cwik
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
>
> [PR/9275|https://github.com/apache/beam/pull/9275] changed 
> *ParDo.getSideInputs* from *List* to *Map PCollectionView>* which is backwards incompatible change and was released as 
> part of Beam 2.16.0 erroneously.
> Running the Apache Nemo Quickstart fails with:
>  
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Translator private 
> static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27Exception in thread 
> "main" java.lang.RuntimeException: Translator private static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27 at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:113)
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineVisitor.visitPrimitiveTransform(PipelineVisitor.java:46)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
>  at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:80) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:31) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at 
> org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at 
> org.apache.beam.examples.WordCount.main(WordCount.java:192)Caused by: 
> java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:109)
>  ... 14 moreCaused by: java.lang.NoSuchMethodError: 
> org.apache.beam.sdk.transforms.ParDo$MultiOutput.getSideInputs()Ljava/util/List;
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(PipelineTranslator.java:236)
>  ... 19 more{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8989) Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-8989:

Fix Version/s: (was: 2.19.0)
   2.20.0

> Backwards incompatible change in ParDo.getSideInputs (caught by failure when 
> running Apache Nemo quickstart)
> 
>
> Key: BEAM-8989
> URL: https://issues.apache.org/jira/browse/BEAM-8989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.16.0, 2.17.0, 2.18.0
>Reporter: Luke Cwik
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.20.0
>
>
> [PR/9275|https://github.com/apache/beam/pull/9275] changed 
> *ParDo.getSideInputs* from *List* to *Map PCollectionView>* which is backwards incompatible change and was released as 
> part of Beam 2.16.0 erroneously.
> Running the Apache Nemo Quickstart fails with:
>  
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Translator private 
> static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27Exception in thread 
> "main" java.lang.RuntimeException: Translator private static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27 at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:113)
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineVisitor.visitPrimitiveTransform(PipelineVisitor.java:46)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
>  at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:80) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:31) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at 
> org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at 
> org.apache.beam.examples.WordCount.main(WordCount.java:192)Caused by: 
> java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:109)
>  ... 14 moreCaused by: java.lang.NoSuchMethodError: 
> org.apache.beam.sdk.transforms.ParDo$MultiOutput.getSideInputs()Ljava/util/List;
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(PipelineTranslator.java:236)
>  ... 19 more{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8989) Backwards incompatible change in ParDo.getSideInputs (caught by failure when running Apache Nemo quickstart)

[Luke Cwik (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026262#comment-17026262
 ] 

Luke Cwik commented on BEAM-8989:
-

I'm not aware of any progress. Since RC1 is already built it makes sense to 
move it along and the issue owner should work to either get this fixed in the 
next possible release candidate (either 2.19.0 RC2 or 2.20.0 RC1).

> Backwards incompatible change in ParDo.getSideInputs (caught by failure when 
> running Apache Nemo quickstart)
> 
>
> Key: BEAM-8989
> URL: https://issues.apache.org/jira/browse/BEAM-8989
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.16.0, 2.17.0, 2.18.0
>Reporter: Luke Cwik
>Assignee: Reuven Lax
>Priority: Critical
> Fix For: 2.19.0
>
>
> [PR/9275|https://github.com/apache/beam/pull/9275] changed 
> *ParDo.getSideInputs* from *List* to *Map PCollectionView>* which is backwards incompatible change and was released as 
> part of Beam 2.16.0 erroneously.
> Running the Apache Nemo Quickstart fails with:
>  
> {code:java}
> Exception in thread "main" java.lang.RuntimeException: Translator private 
> static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27Exception in thread 
> "main" java.lang.RuntimeException: Translator private static void 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(org.apache.nemo.compiler.frontend.beam.PipelineTranslationContext,org.apache.beam.sdk.runners.TransformHierarchy$Node,org.apache.beam.sdk.transforms.ParDo$MultiOutput)
>  have failed to translate 
> org.apache.beam.examples.WordCount$ExtractWordsFn@600b9d27 at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:113)
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineVisitor.visitPrimitiveTransform(PipelineVisitor.java:46)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317)
>  at 
> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251)
>  at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:80) at 
> org.apache.nemo.compiler.frontend.beam.NemoRunner.run(NemoRunner.java:31) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) at 
> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) at 
> org.apache.beam.examples.WordCount.runWordCount(WordCount.java:185) at 
> org.apache.beam.examples.WordCount.main(WordCount.java:192)Caused by: 
> java.lang.reflect.InvocationTargetException at 
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.translatePrimitive(PipelineTranslator.java:109)
>  ... 14 moreCaused by: java.lang.NoSuchMethodError: 
> org.apache.beam.sdk.transforms.ParDo$MultiOutput.getSideInputs()Ljava/util/List;
>  at 
> org.apache.nemo.compiler.frontend.beam.PipelineTranslator.parDoMultiOutputTranslator(PipelineTranslator.java:236)
>  ... 19 more{code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9217) Update DoFn javadoc for schema type translation

[Jira]

 [ 
https://issues.apache.org/jira/browse/BEAM-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-9217:
---
Status: Open  (was: Triage Needed)

> Update DoFn javadoc for schema type translation
> ---
>
> Key: BEAM-9217
> URL: https://issues.apache.org/jira/browse/BEAM-9217
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
>
> see https://github.com/apache/beam/pull/10702#discussion_r372516471
> The javadoc for {{@Element}} currently states "the argument type must match 
> the input type of this DoFn", which is incorrect. We will translate between 
> mismatched types if they both have equivalent schemas.
> cc: [~reuvenlax]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9162) Upgrade Jackson to version 2.10.2

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9162?focusedWorklogId=379026=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379026
 ]

ASF GitHub Bot logged work on BEAM-9162:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:34
Start Date: 29/Jan/20 21:34
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10643: [BEAM-9162] Upgrade 
Jackson to version 2.10.2
URL: https://github.com/apache/beam/pull/10643#issuecomment-579970461
 
 
   @lukecwik do you think we can just get the gradle part that just runs the 
linkageChecker and outputs the errors for all modules in the meantime? That 
would allow me to run and do the manual comparison so we can get progress in 
this PR.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379026)
Time Spent: 2h 40m  (was: 2.5h)

> Upgrade Jackson to version 2.10.2
> -
>
> Key: BEAM-9162
> URL: https://issues.apache.org/jira/browse/BEAM-9162
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Jackson has a new way to deal with [deserialization security 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10] in 
> 2.10.x so worth the upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9219) Update Python dependencies page for 2.18.0

[David Wrede (Jira)]

David Wrede created BEAM-9219:
-

 Summary: Update Python dependencies page for 2.18.0
 Key: BEAM-9219
 URL: https://issues.apache.org/jira/browse/BEAM-9219
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: David Wrede






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=379018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379018
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:17
Start Date: 29/Jan/20 21:17
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579964044
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379018)
Time Spent: 50m  (was: 40m)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8042) Parsing of aggregate query fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8042?focusedWorklogId=379017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379017
 ]

ASF GitHub Bot logged work on BEAM-8042:


Author: ASF GitHub Bot
Created on: 29/Jan/20 21:14
Start Date: 29/Jan/20 21:14
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #10649: [BEAM-8042] 
[ZetaSQL] Fix aggregate column reference
URL: https://github.com/apache/beam/pull/10649#issuecomment-579931786
 
 
   @amaliujia @kanterov Do you have any other suggestions/comments about this?
   If not, should be OK to merge.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379017)
Time Spent: 3h 10m  (was: 3h)

> Parsing of aggregate query fails
> 
>
> Key: BEAM-8042
> URL: https://issues.apache.org/jira/browse/BEAM-8042
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kirill Kozlov
>Priority: Critical
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> {code}
>   @Rule
>   public TestPipeline pipeline = 
> TestPipeline.fromOptions(createPipelineOptions());
>   private static PipelineOptions createPipelineOptions() {
> BeamSqlPipelineOptions opts = 
> PipelineOptionsFactory.create().as(BeamSqlPipelineOptions.class);
> opts.setPlannerName(ZetaSQLQueryPlanner.class.getName());
> return opts;
>   }
>   @Test
>   public void testAggregate() {
> Schema inputSchema = Schema.builder()
> .addByteArrayField("id")
> .addInt64Field("has_f1")
> .addInt64Field("has_f2")
> .addInt64Field("has_f3")
> .addInt64Field("has_f4")
> .addInt64Field("has_f5")
> .addInt64Field("has_f6")
> .build();
> String sql = "SELECT \n" +
> "  id, \n" +
> "  COUNT(*) as count, \n" +
> "  SUM(has_f1) as f1_count, \n" +
> "  SUM(has_f2) as f2_count, \n" +
> "  SUM(has_f3) as f3_count, \n" +
> "  SUM(has_f4) as f4_count, \n" +
> "  SUM(has_f5) as f5_count, \n" +
> "  SUM(has_f6) as f6_count  \n" +
> "FROM PCOLLECTION \n" +
> "GROUP BY id";
> pipeline
> .apply(Create.empty(inputSchema))
> .apply(SqlTransform.query(sql));
> pipeline.run();
>   }
> {code}
> {code}
> Caused by: java.lang.RuntimeException: Error while applying rule 
> AggregateProjectMergeRule, args 
> [rel#553:LogicalAggregate.NONE(input=RelSubset#552,group={0},f1=COUNT(),f2=SUM($2),f3=SUM($3),f4=SUM($4),f5=SUM($5),f6=SUM($6),f7=SUM($7)),
>  
> rel#551:LogicalProject.NONE(input=RelSubset#550,key=$0,f1=$1,f2=$2,f3=$3,f4=$4,f5=$5,f6=$6)]
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:232)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:340)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:168)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:99)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:104)
>   at 
>   ... 39 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
>   at 
> org.apache.beam.repackaged.sql.com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:58)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.apply(AggregateProjectMergeRule.java:96)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:73)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
>   ... 48 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9188) Improving speed of splitting for Custom Sources

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9188?focusedWorklogId=379014=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379014
 ]

ASF GitHub Bot logged work on BEAM-9188:


Author: ASF GitHub Bot
Created on: 29/Jan/20 20:58
Start Date: 29/Jan/20 20:58
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #10701: [BEAM-9188] 
CassandraIO split performance improvement - cache size of the table
URL: https://github.com/apache/beam/pull/10701#issuecomment-579956274
 
 
   Retest it please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379014)
Time Spent: 2h 40m  (was: 2.5h)

> Improving speed of splitting for Custom Sources
> ---
>
> Key: BEAM-9188
> URL: https://issues.apache.org/jira/browse/BEAM-9188
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Radosław Stankiewicz
>Assignee: Radosław Stankiewicz
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> At this moment Custom Source in being split and serialized in sequence. If 
> there are many splits, it takes time to process all splits. 
>  
> Example: it takes 2s to calculate size and serialize CassandraSource due to 
> connection setup and teardown. With 100+ splits, it's a lot of time spent in 
> 1 worker. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8298) Implement state caching for side inputs

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8298?focusedWorklogId=379010=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379010
 ]

ASF GitHub Bot logged work on BEAM-8298:


Author: ASF GitHub Bot
Created on: 29/Jan/20 20:52
Start Date: 29/Jan/20 20:52
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #10705: [BEAM-8298] 
Implement side input caching.
URL: https://github.com/apache/beam/pull/10705#issuecomment-579953721
 
 
   Fixed statecache tests and added a unit test.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379010)
Time Spent: 2.5h  (was: 2h 20m)

> Implement state caching for side inputs
> ---
>
> Key: BEAM-8298
> URL: https://issues.apache.org/jira/browse/BEAM-8298
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core, sdk-py-harness
>Reporter: Maximilian Michels
>Assignee: Jing Chen
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Caching is currently only implemented for user state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=378999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378999
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 29/Jan/20 20:23
Start Date: 29/Jan/20 20:23
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#discussion_r372609060
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 ##
 @@ -1074,6 +1075,12 @@ public Timer withOutputTimestamp(Instant 
outputTimestamp) {
   return this;
 }
 
+@Override
+public Timer withOutputTimestampOffset(Duration outputTimestampOffset) {
+  this.outputTimestampOffset = outputTimestampOffset;
+  return this;
+}
 
 Review comment:
   I'm not sure that we need withOutputTimestampOffset - I think 
withOutputTimestamp is sufficient.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378999)
Time Spent: 16h 40m  (was: 16.5h)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=379000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379000
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 29/Jan/20 20:23
Start Date: 29/Jan/20 20:23
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#discussion_r372610731
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 ##
 @@ -1092,15 +1099,19 @@ private void verifyAbsoluteTimeDomain() {
  * 
  */
 private void setAndVerifyOutputTimestamp() {
-  // Output timestamp is currently not supported in processing time timers.
-  if (outputTimestamp != null && 
!TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
-throw new IllegalStateException("Cannot set outputTimestamp in 
processing time domain.");
-  }
+
   // Output timestamp is set to the delivery time if not initialized by an 
user.
-  if (outputTimestamp == null) {
+  if (outputTimestamp == null && 
TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
 outputTimestamp = target;
   }
 
+  if (TimeDomain.PROCESSING_TIME.equals(spec.getTimeDomain())) {
+outputTimestamp =
+outputTimestampOffset.equals(Duration.ZERO)
+? target
+: target.minus(offset.minus(outputTimestampOffset));
+  }
+
   if (TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
 
 Review comment:
   I think that we should verify that the output timestamp is > the timestamp 
of the input message (if in processElement) or the output timestamp of the 
firing timer (if in processTimer). The < check remains correct - even for 
processing-time timers.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379000)
Time Spent: 16h 50m  (was: 16h 40m)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the 

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=379001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-379001
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 29/Jan/20 20:23
Start Date: 29/Jan/20 20:23
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#discussion_r372611059
 
 

 ##
 File path: 
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 ##
 @@ -1092,15 +1099,19 @@ private void verifyAbsoluteTimeDomain() {
  * 
  */
 private void setAndVerifyOutputTimestamp() {
-  // Output timestamp is currently not supported in processing time timers.
-  if (outputTimestamp != null && 
!TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
-throw new IllegalStateException("Cannot set outputTimestamp in 
processing time domain.");
-  }
+
   // Output timestamp is set to the delivery time if not initialized by an 
user.
-  if (outputTimestamp == null) {
+  if (outputTimestamp == null && 
TimeDomain.EVENT_TIME.equals(spec.getTimeDomain())) {
 outputTimestamp = target;
   }
 
 Review comment:
   I think that if the timer is processing time, then then default 
outputTimestamp should be that of the input element (or the output time of the 
firing timer if in processTimer).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 379001)
Time Spent: 17h  (was: 16h 50m)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8042) Parsing of aggregate query fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8042?focusedWorklogId=378994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378994
 ]

ASF GitHub Bot logged work on BEAM-8042:


Author: ASF GitHub Bot
Created on: 29/Jan/20 19:55
Start Date: 29/Jan/20 19:55
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #10649: [BEAM-8042] 
[ZetaSQL] Fix aggregate column reference
URL: https://github.com/apache/beam/pull/10649#issuecomment-579931786
 
 
   @amaliujia @kanterov Do you have any other suggestions/comments about this?
   If not, should be OK to merge, I think.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378994)
Time Spent: 3h  (was: 2h 50m)

> Parsing of aggregate query fails
> 
>
> Key: BEAM-8042
> URL: https://issues.apache.org/jira/browse/BEAM-8042
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kirill Kozlov
>Priority: Critical
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> {code}
>   @Rule
>   public TestPipeline pipeline = 
> TestPipeline.fromOptions(createPipelineOptions());
>   private static PipelineOptions createPipelineOptions() {
> BeamSqlPipelineOptions opts = 
> PipelineOptionsFactory.create().as(BeamSqlPipelineOptions.class);
> opts.setPlannerName(ZetaSQLQueryPlanner.class.getName());
> return opts;
>   }
>   @Test
>   public void testAggregate() {
> Schema inputSchema = Schema.builder()
> .addByteArrayField("id")
> .addInt64Field("has_f1")
> .addInt64Field("has_f2")
> .addInt64Field("has_f3")
> .addInt64Field("has_f4")
> .addInt64Field("has_f5")
> .addInt64Field("has_f6")
> .build();
> String sql = "SELECT \n" +
> "  id, \n" +
> "  COUNT(*) as count, \n" +
> "  SUM(has_f1) as f1_count, \n" +
> "  SUM(has_f2) as f2_count, \n" +
> "  SUM(has_f3) as f3_count, \n" +
> "  SUM(has_f4) as f4_count, \n" +
> "  SUM(has_f5) as f5_count, \n" +
> "  SUM(has_f6) as f6_count  \n" +
> "FROM PCOLLECTION \n" +
> "GROUP BY id";
> pipeline
> .apply(Create.empty(inputSchema))
> .apply(SqlTransform.query(sql));
> pipeline.run();
>   }
> {code}
> {code}
> Caused by: java.lang.RuntimeException: Error while applying rule 
> AggregateProjectMergeRule, args 
> [rel#553:LogicalAggregate.NONE(input=RelSubset#552,group={0},f1=COUNT(),f2=SUM($2),f3=SUM($3),f4=SUM($4),f5=SUM($5),f6=SUM($6),f7=SUM($7)),
>  
> rel#551:LogicalProject.NONE(input=RelSubset#550,key=$0,f1=$1,f2=$2,f3=$3,f4=$4,f5=$5,f6=$6)]
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:232)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:340)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:168)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:99)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:104)
>   at 
>   ... 39 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
>   at 
> org.apache.beam.repackaged.sql.com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:58)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.apply(AggregateProjectMergeRule.java:96)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:73)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
>   ... 48 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-6284) [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with result UNKNOWN on succeeded job and checks passed

[Neeme Praks (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026171#comment-17026171
 ] 

Neeme Praks commented on BEAM-6284:
---

[https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Phrase/259/testReport/org.apache.beam.examples/WordCountIT/testE2EWordCount/]

{{java.lang.RuntimeException: Dataflow job 
2020-01-29_10_00_27-14511018726499248531 terminated in state UNRECOGNIZED but 
did not return a failure reason.}}

> [FLAKE][beam_PostCommit_Java_ValidatesRunner_Dataflow] TestRunner fails with 
> result UNKNOWN on succeeded job and checks passed
> --
>
> Key: BEAM-6284
> URL: https://issues.apache.org/jira/browse/BEAM-6284
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures, testing
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: currently-failing
> Fix For: 2.13.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> _Use this form to file an issue for test failure:_
>  * 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testWindowedSideInputFixedToGlobal/
> Initial investigation:
> According to logs all test-relevant checks have passed and it seem to be 
> testing framework failure.
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8042) Parsing of aggregate query fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8042?focusedWorklogId=378992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378992
 ]

ASF GitHub Bot logged work on BEAM-8042:


Author: ASF GitHub Bot
Created on: 29/Jan/20 19:53
Start Date: 29/Jan/20 19:53
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on pull request #10649: [BEAM-8042] 
[ZetaSQL] Fix aggregate column reference
URL: https://github.com/apache/beam/pull/10649#discussion_r372597466
 
 

 ##
 File path: 
sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
 ##
 @@ -1347,6 +1347,44 @@ public void testZetaSQLStructFieldAccessInTumble() {
 
pipeline.run().waitUntilFinish(Duration.standardMinutes(PIPELINE_EXECUTION_WAITTIME_MINUTES));
   }
 
+  @Test
+  public void testAggregateWithAndWithoutColumnRefs() {
+ZetaSQLQueryPlanner zetaSQLQueryPlanner = new ZetaSQLQueryPlanner(config);
+
+String sql =
+"SELECT \n"
++ "  id, \n"
++ "  SUM(has_f1) as f1_count, \n"
++ "  SUM(has_f2) as f2_count, \n"
++ "  SUM(has_f3) as f3_count, \n"
++ "  SUM(has_f4) as f4_count, \n"
++ "  SUM(has_f5) as f5_count, \n"
++ "  COUNT(*) as count, \n"
++ "  SUM(has_f6) as f6_count  \n"
++ "FROM (select 0 as id, 1 as has_f1, 2 as has_f2, 3 as has_f3, 4 
as has_f4, 5 as has_f5, 6 as has_f6)\n"
 
 Review comment:
   Leaving as is for now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378992)
Time Spent: 2h 50m  (was: 2h 40m)

> Parsing of aggregate query fails
> 
>
> Key: BEAM-8042
> URL: https://issues.apache.org/jira/browse/BEAM-8042
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kirill Kozlov
>Priority: Critical
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> {code}
>   @Rule
>   public TestPipeline pipeline = 
> TestPipeline.fromOptions(createPipelineOptions());
>   private static PipelineOptions createPipelineOptions() {
> BeamSqlPipelineOptions opts = 
> PipelineOptionsFactory.create().as(BeamSqlPipelineOptions.class);
> opts.setPlannerName(ZetaSQLQueryPlanner.class.getName());
> return opts;
>   }
>   @Test
>   public void testAggregate() {
> Schema inputSchema = Schema.builder()
> .addByteArrayField("id")
> .addInt64Field("has_f1")
> .addInt64Field("has_f2")
> .addInt64Field("has_f3")
> .addInt64Field("has_f4")
> .addInt64Field("has_f5")
> .addInt64Field("has_f6")
> .build();
> String sql = "SELECT \n" +
> "  id, \n" +
> "  COUNT(*) as count, \n" +
> "  SUM(has_f1) as f1_count, \n" +
> "  SUM(has_f2) as f2_count, \n" +
> "  SUM(has_f3) as f3_count, \n" +
> "  SUM(has_f4) as f4_count, \n" +
> "  SUM(has_f5) as f5_count, \n" +
> "  SUM(has_f6) as f6_count  \n" +
> "FROM PCOLLECTION \n" +
> "GROUP BY id";
> pipeline
> .apply(Create.empty(inputSchema))
> .apply(SqlTransform.query(sql));
> pipeline.run();
>   }
> {code}
> {code}
> Caused by: java.lang.RuntimeException: Error while applying rule 
> AggregateProjectMergeRule, args 
> [rel#553:LogicalAggregate.NONE(input=RelSubset#552,group={0},f1=COUNT(),f2=SUM($2),f3=SUM($3),f4=SUM($4),f5=SUM($5),f6=SUM($6),f7=SUM($7)),
>  
> rel#551:LogicalProject.NONE(input=RelSubset#550,key=$0,f1=$1,f2=$2,f3=$3,f4=$4,f5=$5,f6=$6)]
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:232)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:340)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:168)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:99)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)
>   at 
> 

[jira] [Created] (BEAM-9218) Python 3 Dataflow runner is evaluating a RuntimeValueProvider during template staging, causing 'not accessible' error

[Michael Charkin (Jira)]

Michael Charkin created BEAM-9218:
-

 Summary: Python 3 Dataflow runner is evaluating a 
RuntimeValueProvider during template staging, causing 'not accessible' error
 Key: BEAM-9218
 URL: https://issues.apache.org/jira/browse/BEAM-9218
 Project: Beam
  Issue Type: Bug
  Components: runner-dataflow
Reporter: Michael Charkin


beam can not stage cloud Dataflow template with python 3.7.

Looks like it is trying to access the RuntimeValueProvider during staging 
causing 'not accessible'

Template stages fine with python 2.7

 

Repo with code to reproduce the issue: 
[https://github.com/firemuzzy/dataflow-templates-bug-python3]

 

Sorry if this is not the correct places, I have exhausted all places for 
reporting this issue including the dataflow slack channel and Stack Overflow.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9132) State request handler is removed prematurely when closing ActiveBundle

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9132?focusedWorklogId=378991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378991
 ]

ASF GitHub Bot logged work on BEAM-9132:


Author: ASF GitHub Bot
Created on: 29/Jan/20 19:52
Start Date: 29/Jan/20 19:52
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #10694: [BEAM-9132] Avoid 
logging misleading error messages during pipeline failure
URL: https://github.com/apache/beam/pull/10694#discussion_r372585551
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java
 ##
 @@ -406,11 +407,14 @@ public void split(double fractionOfRemainder) {
 
 @Override
 public void close() throws Exception {
-  bundle.close();
-  currentClient.wrappedClient.unref();
-  if (loadBalanceBundles) {
-availableCaches.offer(currentCache);
-availableCachesSemaphore.release();
+  try {
+bundle.close();
+  } finally {
+currentClient.wrappedClient.unref();
+if (loadBalanceBundles) {
+  availableCaches.offer(currentCache);
+  availableCachesSemaphore.release();
+}
 
 Review comment:
   Looks like this was responsible for the cleanup failing. bundle.close() may 
throw leaving the environment still referenced. My tests do not yield any more 
errors like in the description.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378991)
Time Spent: 2.5h  (was: 2h 20m)

> State request handler is removed prematurely when closing ActiveBundle
> --
>
> Key: BEAM-9132
> URL: https://issues.apache.org/jira/browse/BEAM-9132
> Project: Beam
>  Issue Type: Bug
>  Components: java-fn-execution
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> We have observed these errors in a state-intense application: 
> {noformat}
> Error processing instruction 107. Original traceback is
> Traceback (most recent call last):
>   File "apache_beam/runners/common.py", line 780, in 
> apache_beam.runners.common.DoFnRunner.process
>   File "apache_beam/runners/common.py", line 587, in 
> apache_beam.runners.common.PerWindowInvoker.invoke_process
>   File "apache_beam/runners/common.py", line 659, in 
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
>   File "apache_beam/runners/common.py", line 880, in 
> apache_beam.runners.common._OutputProcessor.process_outputs
>   File "apache_beam/runners/common.py", line 895, in 
> apache_beam.runners.common._OutputProcessor.process_outputs
>   File "redacted.py", line 56, in process
> recent_events_map = load_recent_events_map(recent_events_state)
>   File "redacted.py", line 128, in _load_recent_events_map
> items_in_recent_events_bag = list(recent_events_state.read())
>   File "apache_beam/runners/worker/bundle_processor.py", line 335, in __iter__
> for elem in self.first:
>   File "apache_beam/runners/worker/bundle_processor.py", line 214, in __iter__
> self._state_key, self._coder_impl, is_cached=self._is_cached)
>   File "apache_beam/runners/worker/sdk_worker.py", line 692, in blocking_get
> self._materialize_iter(state_key, coder))
>   File "apache_beam/runners/worker/sdk_worker.py", line 723, in 
> _materialize_iter
> self._underlying.get_raw(state_key, continuation_token)
>   File "apache_beam/runners/worker/sdk_worker.py", line 603, in get_raw
> continuation_token=continuation_token)))
>   File "apache_beam/runners/worker/sdk_worker.py", line 637, in 
> _blocking_request
> raise RuntimeError(response.error)
> RuntimeError: Unknown process bundle instruction id '107'
> {noformat}
> Notice that the error is thrown on the Runner side. It seems to originate 
> from the {{ActiveBundle}} de-registering the state request handler too early 
> when the processing may still be going on in the SDK Harness.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9215) FileBesedSink may suppress exceptions during close

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9215?focusedWorklogId=378988=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378988
 ]

ASF GitHub Bot logged work on BEAM-9215:


Author: ASF GitHub Bot
Created on: 29/Jan/20 19:47
Start Date: 29/Jan/20 19:47
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10710: [BEAM-9215] 
Fixes an issue where FileBasedSink may suppress exceptions.
URL: https://github.com/apache/beam/pull/10710#issuecomment-579928771
 
 
   Run Java_Examples_Dataflow PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378988)
Time Spent: 40m  (was: 0.5h)

> FileBesedSink may suppress exceptions during close
> --
>
> Key: BEAM-9215
> URL: https://issues.apache.org/jira/browse/BEAM-9215
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Chamikara Madhusanka Jayalath
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We invoke closeChannelAndThrow() here when there's an exception when closing.
> [https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java#L961]
>  
> But we suppress the exception if second close() trivially succeeds for some 
> reason. We should raise the exception nevertheless allowing the runner to 
> retry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8042) Parsing of aggregate query fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8042?focusedWorklogId=378981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-378981
 ]

ASF GitHub Bot logged work on BEAM-8042:


Author: ASF GitHub Bot
Created on: 29/Jan/20 19:27
Start Date: 29/Jan/20 19:27
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10649: [BEAM-8042] 
[ZetaSQL] Fix aggregate column reference
URL: https://github.com/apache/beam/pull/10649#issuecomment-579919734
 
 
   Run JavaPortabilityApi PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 378981)
Time Spent: 2h 40m  (was: 2.5h)

> Parsing of aggregate query fails
> 
>
> Key: BEAM-8042
> URL: https://issues.apache.org/jira/browse/BEAM-8042
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql-zetasql
>Reporter: Rui Wang
>Assignee: Kirill Kozlov
>Priority: Critical
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> {code}
>   @Rule
>   public TestPipeline pipeline = 
> TestPipeline.fromOptions(createPipelineOptions());
>   private static PipelineOptions createPipelineOptions() {
> BeamSqlPipelineOptions opts = 
> PipelineOptionsFactory.create().as(BeamSqlPipelineOptions.class);
> opts.setPlannerName(ZetaSQLQueryPlanner.class.getName());
> return opts;
>   }
>   @Test
>   public void testAggregate() {
> Schema inputSchema = Schema.builder()
> .addByteArrayField("id")
> .addInt64Field("has_f1")
> .addInt64Field("has_f2")
> .addInt64Field("has_f3")
> .addInt64Field("has_f4")
> .addInt64Field("has_f5")
> .addInt64Field("has_f6")
> .build();
> String sql = "SELECT \n" +
> "  id, \n" +
> "  COUNT(*) as count, \n" +
> "  SUM(has_f1) as f1_count, \n" +
> "  SUM(has_f2) as f2_count, \n" +
> "  SUM(has_f3) as f3_count, \n" +
> "  SUM(has_f4) as f4_count, \n" +
> "  SUM(has_f5) as f5_count, \n" +
> "  SUM(has_f6) as f6_count  \n" +
> "FROM PCOLLECTION \n" +
> "GROUP BY id";
> pipeline
> .apply(Create.empty(inputSchema))
> .apply(SqlTransform.query(sql));
> pipeline.run();
>   }
> {code}
> {code}
> Caused by: java.lang.RuntimeException: Error while applying rule 
> AggregateProjectMergeRule, args 
> [rel#553:LogicalAggregate.NONE(input=RelSubset#552,group={0},f1=COUNT(),f2=SUM($2),f3=SUM($3),f4=SUM($4),f5=SUM($5),f6=SUM($6),f7=SUM($7)),
>  
> rel#551:LogicalProject.NONE(input=RelSubset#550,key=$0,f1=$1,f2=$2,f3=$3,f4=$4,f5=$5,f6=$6)]
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:232)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:637)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:340)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.transform(ZetaSQLPlannerImpl.java:168)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:99)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.parseQuery(ZetaSQLQueryPlanner.java:87)
>   at 
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.convertToBeamRel(ZetaSQLQueryPlanner.java:66)
>   at 
> org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:104)
>   at 
>   ... 39 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 7
>   at 
> org.apache.beam.repackaged.sql.com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:58)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.apply(AggregateProjectMergeRule.java:96)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.rel.rules.AggregateProjectMergeRule.onMatch(AggregateProjectMergeRule.java:73)
>   at 
> org.apache.beam.repackaged.sql.org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:205)
>   ... 48 more
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)