[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=382117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382117
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 05/Feb/20 07:25
Start Date: 05/Feb/20 07:25
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r375093447
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
+  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
   publishers {
 archiveJunit('**/build/test-results/**/*.xml')
   }
 
   steps {
+gradle {
+  rootBuildScriptDir(commonJobProperties.checkoutDir)
 
 Review comment:
   We want to check if the released version of Beam, which is compiled with 
Java 8, will work when a user with Java 11 environment imports it from Maven 
and uses it in their code.
   
   In other words: yes, we want to explicitly use the Java 8 compiler.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382117)
Time Spent: 19h 40m  (was: 19.5h)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=382115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382115
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 05/Feb/20 07:17
Start Date: 05/Feb/20 07:17
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10734: [BEAM-8979] 
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582273859
 
 
   Ok I pushed my fix.  If you run the tests again that would be great. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382115)
Time Spent: 6h 50m  (was: 6h 40m)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
>  line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32   File "/usr/lib/python3.7/distutils/cmd.py", line 313, in 
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
>  line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for 
> details): 1
> {code}
>  
> This is what I have tried so far to resolve this (without being successful):
>  * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter 
> to the _protoc_ call ingen_protos.py:131
>  * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9241) Fix inconsistent nullability mapping for Protobuf to Schema

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9241?focusedWorklogId=382111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382111
 ]

ASF GitHub Bot logged work on BEAM-9241:


Author: ASF GitHub Bot
Created on: 05/Feb/20 07:08
Start Date: 05/Feb/20 07:08
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10746: [BEAM-9241] Fix 
inconsistent proto nullability
URL: https://github.com/apache/beam/pull/10746#issuecomment-582271277
 
 
   @reuvenlax can me give a LGTM? Fixed your concern.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382111)
Time Spent: 50m  (was: 40m)

> Fix inconsistent nullability mapping for Protobuf to Schema
> ---
>
> Key: BEAM-9241
> URL: https://issues.apache.org/jira/browse/BEAM-9241
> Project: Beam
>  Issue Type: Bug
>  Components: extensions-java-protobuf
>Affects Versions: 2.18.0
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Fix the nullability issues with protobuf to schema mapping
>  * Proto3 primitive types should be *not* nullable.
>  * Proto2 required types should be *not* nullable.
>  * Proto2 optional should also be *not* nullable as having an optional value 
> doesn't mean it has not value. The spec states it has the optional value.
>  * Arrays should be *not* nullable, as proto arrays always have an empty 
> array when no value is set.
>  * Maps should be *not* nullable, as proto maps always have an empty map when 
> no value is set.
>  * Elements in an array should be *not* nullable, as nulls are not allowed in 
> an array.
>  * Names and Values should be *not* nullable, as nulls are not allowed.
>  * Rows are nullable, as messages are nullable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=382110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382110
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 05/Feb/20 07:08
Start Date: 05/Feb/20 07:08
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10734: [BEAM-8979] 
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582271275
 
 
   Here are the results of my investigation tonight.  I cloned mypy-protobuf 
and created a virtualenv with a path that exceeded the 512 character limit on 
darwin.
   
   I tried installing the scripts in several ways:
   
   | command | setup.py uses entry_points |  package format | produces 
#!/bin/sh |
   | - | - | - |- |
   | `pip install python/` | no |  wheel | no |
   | `cd python/ && python setup.py install` | no | egg | no |
   | `pip install python/` | yes |  wheel | yes |
   | `cd python/ && python setup.py install` | yes | egg | no |
   | `pip wheel -w . python/ && pip install 
mypy_protobuf-1.17-py2-none-any.whl` | yes |  wheel | yes |
   
   This confirms that you need to use pip to install (which we are), and 
entry_points in the project's setup.py file, for the fix to be applied to the 
installed script.  
   
   All tests were performed with the following:
   
   - python 2.7.16
   - pip 20.0.2
   
   My changes are here: 
https://github.com/chadrik/mypy-protobuf/tree/setup_entry_points 
   
   Next I'm going to try editing the build-requirements.txt to point at my 
github fork to see if we can get a working test.  If that works, then I'll make 
the upstream PR.
   
   
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382110)
Time Spent: 6h 40m  (was: 6.5h)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> 

[jira] [Commented] (BEAM-4457) Analyze FieldAccessDescriptors and drop fields that are never accessed

[Alex Van Boxel (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030397#comment-17030397
 ] 

Alex Van Boxel commented on BEAM-4457:
--

I remember the days when I wrote Apache Pig that it had a similar concept as 
well. Another part where this could benefit it the ToRow function where the row 
gets materialized in a RowWithStorage. Only the fields that are accessed should 
be materialized.

> Analyze FieldAccessDescriptors and drop fields that are never accessed
> --
>
> Key: BEAM-4457
> URL: https://issues.apache.org/jira/browse/BEAM-4457
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>
> We can walk backwards through the graph, analyzing which fields are accessed. 
> When we find paths where many fields are never accessed, we can insert a 
> projection transform to drop those fields preemptively. This can save a lot 
> of resources in the case where many fields in the input are never accessed.
> To do this, the FieldAccessDescriptor information must be added to the 
> portability protos. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=382063=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382063
 ]

ASF GitHub Bot logged work on BEAM-3453:


Author: ASF GitHub Bot
Created on: 05/Feb/20 04:46
Start Date: 05/Feb/20 04:46
Worklog Time Spent: 10m 
  Work Description: aaltay commented on pull request #10762: [BEAM-3453] 
Use default project when creating pubsub subscription
URL: https://github.com/apache/beam/pull/10762#discussion_r375057192
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py
 ##
 @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, 
short_topic_name,
 
 sub_client = pubsub.SubscriberClient()
 sub_name = sub_client.subscription_path(
-project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32)))
+sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 
32)))
 
 Review comment:
   I _think_ changing that would be a backward incompatible change and might 
break users. We can keep the existing behavior in place, and add a warning 
along the lines of this is wrong/deprecated/would change in the future.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382063)
Time Spent: 1h 10m  (was: 1h)

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-9251:

Status: Open  (was: Triage Needed)

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?focusedWorklogId=382062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382062
 ]

ASF GitHub Bot logged work on BEAM-9251:


Author: ASF GitHub Bot
Created on: 05/Feb/20 04:30
Start Date: 05/Feb/20 04:30
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10773: [BEAM-9251] Fix 
:sdks:java:io:kafka:updateOfflineRepository
URL: https://github.com/apache/beam/pull/10773
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382062)
Time Spent: 40m  (was: 0.5h)

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5504) PubsubAvroTable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=382061=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382061
 ]

ASF GitHub Bot logged work on BEAM-5504:


Author: ASF GitHub Bot
Created on: 05/Feb/20 04:19
Start Date: 05/Feb/20 04:19
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #10487: [BEAM-5504] 
Introduce PubsubAvroTable
URL: https://github.com/apache/beam/pull/10487#issuecomment-582233424
 
 
   @milantracy have you had a chance to work on that change?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382061)
Time Spent: 3h 10m  (was: 3h)

> PubsubAvroTable
> ---
>
> Key: BEAM-5504
> URL: https://issues.apache.org/jira/browse/BEAM-5504
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Jing Chen
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=382058=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382058
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 05/Feb/20 04:07
Start Date: 05/Feb/20 04:07
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#issuecomment-582231164
 
 
   run flink validatesrunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382058)
Time Spent: 22h 40m  (was: 22.5h)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 22h 40m
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2535) Allow explicit output time independent of firing specification for all timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2535?focusedWorklogId=382057=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382057
 ]

ASF GitHub Bot logged work on BEAM-2535:


Author: ASF GitHub Bot
Created on: 05/Feb/20 04:06
Start Date: 05/Feb/20 04:06
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #10627: [BEAM-2535] 
Support outputTimestamp and watermark holds in processing timers.
URL: https://github.com/apache/beam/pull/10627#issuecomment-582231125
 
 
   run dataflow validatesrunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382057)
Time Spent: 22.5h  (was: 22h 20m)

> Allow explicit output time independent of firing specification for all timers
> -
>
> Key: BEAM-2535
> URL: https://issues.apache.org/jira/browse/BEAM-2535
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 22.5h
>  Remaining Estimate: 0h
>
> Today, we have insufficient control over the event time timestamp of elements 
> output from a timer callback.
> 1. For an event time timer, it is the timestamp of the timer itself.
>  2. For a processing time timer, it is the current input watermark at the 
> time of processing.
> But for both of these, we may want to reserve the right to output a 
> particular time, aka set a "watermark hold".
> A naive implementation of a {{TimerWithWatermarkHold}} would work for making 
> sure output is not droppable, but does not fully explain window expiration 
> and late data/timer dropping.
> In the natural interpretation of a timer as a feedback loop on a transform, 
> timers should be viewed as another channel of input, with a watermark, and 
> items on that channel _all need event time timestamps even if they are 
> delivered according to a different time domain_.
> I propose that the specification for when a timer should fire should be 
> separated (with nice defaults) from the specification of the event time of 
> resulting outputs. These timestamps will determine a side channel with a new 
> "timer watermark" that constrains the output watermark.
>  - We still need to fire event time timers according to the input watermark, 
> so that event time timers fire.
>  - Late data dropping and window expiration will be in terms of the minimum 
> of the input watermark and the timer watermark. In this way, whenever a timer 
> is set, the window is not going to be garbage collected.
>  - We will need to make sure we have a way to "wake up" a window once it is 
> expired; this may be as simple as exhausting the timer channel as soon as the 
> input watermark indicates expiration of a window
> This is mostly aimed at end-user timers in a stateful+timely {{DoFn}}. It 
> seems reasonable to use timers as an implementation detail (e.g. in 
> runners-core utilities) without wanting any of this additional machinery. For 
> example, if there is no possibility of output from the timer callback.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?focusedWorklogId=382053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382053
 ]

ASF GitHub Bot logged work on BEAM-9251:


Author: ASF GitHub Bot
Created on: 05/Feb/20 03:40
Start Date: 05/Feb/20 03:40
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10773: [BEAM-9251] Fix 
:sdks:java:io:kafka:updateOfflineRepository
URL: https://github.com/apache/beam/pull/10773#issuecomment-582226414
 
 
   LGTM
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382053)
Time Spent: 0.5h  (was: 20m)

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-8865) FileIO's Javadoc is outdated: TypeDescriptors.KVs and unhandled IOException

[Tomo Suzuki (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomo Suzuki resolved BEAM-8865.
---
Fix Version/s: 2.19.0
   Resolution: Fixed

 GitHub Pull Request #10256 resolved

> FileIO's Javadoc is outdated: TypeDescriptors.KVs and unhandled IOException
> ---
>
> Key: BEAM-8865
> URL: https://issues.apache.org/jira/browse/BEAM-8865
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Tomo Suzuki
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.19.0
>
> Attachments: v1YuuuT2o2i.png
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The doc says
>  !v1YuuuT2o2i.png! 
> However, I had to modify the code to below:
> {noformat}
> p.apply("ReadLines",
> FileIO.match().filepattern("hdfs:///foo/bar/*.gz"))
> .apply(FileIO.readMatches().withCompression(GZIP))
> .apply(MapElements
> // uses imports from TypeDescriptors
> .into(kvs(strings(), strings()))
> .via((ReadableFile f) -> {
>   try {
> return KV.of(
> f.getMetadata().resourceId().toString(), 
> f.readFullyAsUTF8String());
>   } catch (IOException ex) {
> throw new RuntimeException("Failed to read the file", ex);
>   }
>  }))
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (BEAM-9224) flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job

[Tomo Suzuki (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomo Suzuki closed BEAM-9224.
-
Fix Version/s: Not applicable
   Resolution: Fixed

There was a problem in Dataflow backend. Once it was fixed, the check came back 
green.

> flaky WordCountIT.testE2EWordCount in Jenkins Java_Examples_Dataflow Job
> 
>
> Key: BEAM-9224
> URL: https://issues.apache.org/jira/browse/BEAM-9224
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Tomo Suzuki
>Priority: Major
> Fix For: Not applicable
>
> Attachments: yY6GxCyAV3C.png
>
>
> Java_Examples_Dataflow ("Run Java_Examples_Dataflow PreCommit") is flaky. 
> {{org.apache.beam.examples.WordCountIT.testE2EWordCount}} often fails.
> Example: 
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/6552/console]
> {noformat}
> 14:39:58 > Task 
> :runners:google-cloud-dataflow-java:examples-streaming:windmillPreCommit
> 14:39:58 
> 14:39:58 org.apache.beam.examples.WordCountIT > testE2EWordCount FAILED
> 14:39:58 java.lang.RuntimeException at WordCountIT.java:69
> 14:39:58 
> 14:39:58 1 test completed, 1 failed
> {noformat}
> The trend is alarming:
> !yY6GxCyAV3C.png|width=675,height=338!
> [https://builds.apache.org/job/beam_PreCommit_Java_Examples_Dataflow_Commit/]
>  
> This similar to BEAM-3968 (still open), but this time the failure is on 
> {{testE2EWordCount}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8376) Add FirestoreIO connector to Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8376?focusedWorklogId=382045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382045
 ]

ASF GitHub Bot logged work on BEAM-8376:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:52
Start Date: 05/Feb/20 02:52
Worklog Time Spent: 10m 
  Work Description: clement commented on issue #10187: [BEAM-8376] Initial 
version of firestore connector JavaSDK
URL: https://github.com/apache/beam/pull/10187#issuecomment-582216769
 
 
   Hi @djelekar, I work on the Firestore backend, and chiming in to second 
@fredzqm point. There are two interlocking issues when using atomic WriteBatch 
for large ingestion (throughput) jobs.
   
   First, under load and based on size, Firestore will split your dataset 
across multiple servers. When writing atomically to multiple documents, this 
increase the chance that the write will need to coordinate a 2-phase commit 
across multiple servers, which will increase the latency of the operation.
   
   Second, Firestore uses a pessimistic locking model under the hood. If the 
WriteBatch takes longer to execute (because of the issue above, or just because 
it is doing more work) it will be holding locks longer and disrupt unrelated 
read/write traffic on the document or index entries.
   
   I can see reasons why the experience looks better with WriteBatch, for 
example:
   - when using single writes, those should be asynchronous, and can (and 
should) be parallelized more aggressively
   - if the ingestion key range is not split, or not actively accessed by other 
processes, there will initially be no contention and good performance with 
WriteBatch, however there is a limit to how much throughput you will get from 
them once the ingestion runs longer and ramps up to more parallelism.
   
   Does that make sense? We are hoping to launch a dedicated feature for 
writing batches in a non-atomic fashion, but it is unclear at this point when 
this will be generally available, and as @fredzqm point out, single writes are 
the best option for now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382045)
Time Spent: 2h 40m  (was: 2.5h)

> Add FirestoreIO connector to Java SDK
> -
>
> Key: BEAM-8376
> URL: https://issues.apache.org/jira/browse/BEAM-8376
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Reporter: Stefan Djelekar
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Motivation:
> There is no Firestore connector for Java SDK at the moment.
> Having it will enhance the integrations with database options on the Google 
> Cloud Platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?focusedWorklogId=382041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382041
 ]

ASF GitHub Bot logged work on BEAM-9251:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:43
Start Date: 05/Feb/20 02:43
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10773: [BEAM-9251] Fix 
:sdks:java:io:kafka:updateOfflineRepository
URL: https://github.com/apache/beam/pull/10773
 
 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?focusedWorklogId=382042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382042
 ]

ASF GitHub Bot logged work on BEAM-9251:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:43
Start Date: 05/Feb/20 02:43
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10773: [BEAM-9251] Fix 
:sdks:java:io:kafka:updateOfflineRepository
URL: https://github.com/apache/beam/pull/10773#issuecomment-582215053
 
 
   R: @chamikaramj 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382042)
Time Spent: 20m  (was: 10m)

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-9251:
---

Assignee: Udi Meiri  (was: Alexey Romanenko)

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030314#comment-17030314
 ] 

Udi Meiri commented on BEAM-9251:
-

I believe I have a fix.

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Alexey Romanenko
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=382040=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382040
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:38
Start Date: 05/Feb/20 02:38
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10563: [BEAM-7310] Add support 
of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#issuecomment-582213871
 
 
   Opened https://issues.apache.org/jira/browse/BEAM-9251 for breakage of 
updateOfflineRepository
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382040)
Time Spent: 13h 10m  (was: 13h)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030311#comment-17030311
 ] 

Udi Meiri commented on BEAM-9251:
-

This dep was added in https://github.com/apache/beam/pull/10563

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Alexey Romanenko
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030307#comment-17030307
 ] 

Udi Meiri commented on BEAM-9251:
-

I also did an "rm -rfv ~/.m2/repository/" beforehand.

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Alexey Romanenko
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-9251:
---

Assignee: Alexey Romanenko

> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Assignee: Alexey Romanenko
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the following locations:
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
>- 
> https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
>  Required by:
>  project :sdks:java:io:kafka
> {code}
> I found the package here: 
> https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
> but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-9251:

Description: 
I resolved the 403 here: https://github.com/apache/beam/pull/10771
Now getting a not found error:
{code}
Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> Could not resolve all dependencies for configuration 
> ':sdks:java:io:kafka:detachedConfiguration14'.
   > Could not find io.confluent:kafka-avro-serializer:5.3.2.
 Searched in the following locations:
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
 Required by:
 project :sdks:java:io:kafka
{code}

I found the package here: 
https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
but I have no idea what the official location is.

  was:
I resolved the 403 here: https://github.com/apache/beam/pull/10771
Now getting a not found error:
{code}
Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> Could not resolve all dependencies for configuration 
> ':sdks:java:io:kafka:detachedConfiguration14'.
   > Could not find io.confluent:kafka-avro-serializer:5.3.2.
 Searched in the following locations:
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
file:/google/src/cloud/ehudm/update-beam-deps-71008f39339b412b3b1680b0c99ce3663846fd75/google3/third_party/maven/repository/central/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
file:/google/src/cloud/ehudm/update-beam-deps-71008f39339b412b3b1680b0c99ce3663846fd75/google3/third_party/maven/repository/central/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
 Required by:
 project :sdks:java:io:kafka
{code}

I found the package here: 
https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
but I have no idea what the official location is.


> :sdks:java:io:kafka:updateOfflineRepository fails
> -
>
> Key: BEAM-9251
> URL: https://issues.apache.org/jira/browse/BEAM-9251
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Udi Meiri
>Priority: Major
>
> I resolved the 403 here: https://github.com/apache/beam/pull/10771
> Now getting a not found error:
> {code}
> Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> > Could not resolve all dependencies for configuration 
> > ':sdks:java:io:kafka:detachedConfiguration14'.
>> Could not find io.confluent:kafka-avro-serializer:5.3.2.
>  Searched in the 

[jira] [Created] (BEAM-9251) :sdks:java:io:kafka:updateOfflineRepository fails

[Udi Meiri (Jira)]

Udi Meiri created BEAM-9251:
---

 Summary: :sdks:java:io:kafka:updateOfflineRepository fails
 Key: BEAM-9251
 URL: https://issues.apache.org/jira/browse/BEAM-9251
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Udi Meiri


I resolved the 403 here: https://github.com/apache/beam/pull/10771
Now getting a not found error:
{code}
Execution failed for task ':sdks:java:io:kafka:updateOfflineRepository'.
> Could not resolve all dependencies for configuration 
> ':sdks:java:io:kafka:detachedConfiguration14'.
   > Could not find io.confluent:kafka-avro-serializer:5.3.2.
 Searched in the following locations:
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
file:/usr/local/google/home/ehudm/.m2/repository/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.maven.apache.org/maven2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://jcenter.bintray.com/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://plugins.gradle.org/m2/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
https://repo.spring.io/plugins-release/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
   - 
file:/google/src/cloud/ehudm/update-beam-deps-71008f39339b412b3b1680b0c99ce3663846fd75/google3/third_party/maven/repository/central/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.pom
   - 
file:/google/src/cloud/ehudm/update-beam-deps-71008f39339b412b3b1680b0c99ce3663846fd75/google3/third_party/maven/repository/central/io/confluent/kafka-avro-serializer/5.3.2/kafka-avro-serializer-5.3.2.jar
 Required by:
 project :sdks:java:io:kafka
{code}

I found the package here: 
https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/5.3.2/
but I have no idea what the official location is.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=382028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382028
 ]

ASF GitHub Bot logged work on BEAM-9219:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:12
Start Date: 05/Feb/20 02:12
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10745: [BEAM-9219] 
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r375026371
 
 

 ##
 File path: website/src/documentation/sdks/python-dependencies.md
 ##
 @@ -26,460 +26,41 @@ behavior in the service. If you are using any of these 
packages in your code, be
 aware that some libraries are not forward-compatible and you may need to pin to
 the listed versions that will be in scope during execution.
 
-To see the compile and runtime dependencies for your Beam SDK version, 
expand
-the relevant section below.
+Dependencies for your Beam SDK version dependencies are listed in `setup.py` 
in the Beam repository. To view them, perform the following steps:
 
 Review comment:
   'dependencies' is repeated twice here as well. Let's shorten this to 
`Dependencies for your Beam SDK are listed in...`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382028)
Time Spent: 3h  (was: 2h 50m)

> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: David Wrede
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK 
> dependency pages more relevant and up-to-date while reducing the amount of 
> time it takes to provide that information. The current method of scraping and 
> copying dependencies into a table for every release is a non-trivial task 
> because of the semi-automated workflows done by the tech writers on the 
> website.
> In an effort to provide accurate dependency listings that are always in sync 
> with SDK releases, referring people to the appropriate places in the source 
> code (or through CLI commands) should provide people the information they are 
> looking for and not require the creation and maintenance of an automated 
> tooling solution to generate the dependency tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=382027=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-382027
 ]

ASF GitHub Bot logged work on BEAM-9219:


Author: ASF GitHub Bot
Created on: 05/Feb/20 02:12
Start Date: 05/Feb/20 02:12
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10745: [BEAM-9219] 
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r375025947
 
 

 ##
 File path: website/src/documentation/sdks/java-dependencies.md
 ##
 @@ -26,336 +26,24 @@ behavior in the service. If you are using any of these 
packages in your code, be
 aware that some libraries are not forward-compatible and you may need to pin to
 the listed versions that will be in scope during execution.
 
-To see the compile and runtime dependencies for your Beam SDK version, 
expand
-the relevant section below.
+Compile and runtime dependencies for your Beam SDK version dependencies are 
listed in `BeamModulePlugin.groovy` in the Beam repository. To view them, 
perform the following steps:
 
 Review comment:
   Looks like 'dependencies' is repeated twice.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 382027)
Time Spent: 3h  (was: 2h 50m)

> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: David Wrede
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK 
> dependency pages more relevant and up-to-date while reducing the amount of 
> time it takes to provide that information. The current method of scraping and 
> copying dependencies into a table for every release is a non-trivial task 
> because of the semi-automated workflows done by the tech writers on the 
> website.
> In an effort to provide accurate dependency listings that are always in sync 
> with SDK releases, referring people to the appropriate places in the source 
> code (or through CLI commands) should provide people the information they are 
> looking for and not require the creation and maintenance of an automated 
> tooling solution to generate the dependency tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9250) Improve beam release script based on 2.19.0 release experience

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9250?focusedWorklogId=381987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381987
 ]

ASF GitHub Bot logged work on BEAM-9250:


Author: ASF GitHub Bot
Created on: 05/Feb/20 01:11
Start Date: 05/Feb/20 01:11
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on pull request #10772: [BEAM-9250] 
Re-structure python release candidate target.
URL: https://github.com/apache/beam/pull/10772
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 

[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381985
 ]

ASF GitHub Bot logged work on BEAM-3453:


Author: ASF GitHub Bot
Created on: 05/Feb/20 01:09
Start Date: 05/Feb/20 01:09
Worklog Time Spent: 10m 
  Work Description: davidyan74 commented on pull request #10762: 
[BEAM-3453] Use default project when creating pubsub subscription
URL: https://github.com/apache/beam/pull/10762#discussion_r375011076
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py
 ##
 @@ -499,13 +499,21 @@ def __init__(self, evaluation_context, 
applied_ptransform,
 if self.source.id_label:
   raise NotImplementedError(
   'DirectRunner: id_label is not supported for PubSub reads')
+
+import google.auth
+
+_, sub_project = google.auth.default()
 
 Review comment:
   Done. I'm defaulting to the default project from auth if the project is not 
specified. Thanks!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381985)
Time Spent: 1h  (was: 50m)

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381969
 ]

ASF GitHub Bot logged work on BEAM-3453:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:40
Start Date: 05/Feb/20 00:40
Worklog Time Spent: 10m 
  Work Description: davidyan74 commented on pull request #10762: 
[BEAM-3453] Use default project when creating pubsub subscription
URL: https://github.com/apache/beam/pull/10762#discussion_r375003261
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py
 ##
 @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, 
short_topic_name,
 
 sub_client = pubsub.SubscriberClient()
 sub_name = sub_client.subscription_path(
-project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32)))
+sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 
32)))
 
 Review comment:
   Falling back to topic's project may make things less predictable, and 
failure to create the subscription could be transient and could be because of 
many reasons. The user should be sure which project it will create the 
subscription under. My take is that using the topic's project was a mistake 
from the beginning and it should be fixed. But let me know if you have other 
thoughts.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381969)
Time Spent: 50m  (was: 40m)

> Allow usage of public Google PubSub topics in Python DirectRunner
> -
>
> Key: BEAM-3453
> URL: https://issues.apache.org/jira/browse/BEAM-3453
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: Charles Chen
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently, the Beam Python DirectRunner does not allow the usage of data from 
> public Google Cloud PubSub topics.  We should allow this functionality so 
> that users can more easily test Beam Python's streaming functionality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2645) Implement DisplayData translation to/from protos

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2645?focusedWorklogId=381968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381968
 ]

ASF GitHub Bot logged work on BEAM-2645:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:38
Start Date: 05/Feb/20 00:38
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10770: [BEAM-2645] Define 
the display data model type
URL: https://github.com/apache/beam/pull/10770#issuecomment-582185886
 
 
   R: @robertwb @dpmills 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381968)
Time Spent: 20m  (was: 10m)

> Implement DisplayData translation to/from protos
> 
>
> Key: BEAM-2645
> URL: https://issues.apache.org/jira/browse/BEAM-2645
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9146) [Python] PTransform that integrates Video Intelligence functionality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9146?focusedWorklogId=381965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381965
 ]

ASF GitHub Bot logged work on BEAM-9146:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:22
Start Date: 05/Feb/20 00:22
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10764: [BEAM-9146] Integrate 
GCP Video Intelligence functionality for Python SDK
URL: https://github.com/apache/beam/pull/10764#issuecomment-582181978
 
 
   @EDjur - Sorry for the miscommunication. These types of decisions should 
have reflected on JIRA as well. I feel bad that you spent time on this and we 
are changing direction after your PR is out.
   
   I think what it means for this ticket is that:
   - If tfx_bsl has ability to call into services (which we need to add and TFX 
team agreed to support us with reviews)
   - We can add thin transforms for different things (e.g. AnnotateVideo) and 
each of these transform will have quite a bit of shared code and behavior.
   
   @kamilwu may have other thoughts, since he was also working on this. I would 
like to hear what is his opinion.
   
   For this PR, let's try to re-use as much as possible.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381965)
Time Spent: 50m  (was: 40m)

> [Python] PTransform that integrates Video Intelligence functionality
> 
>
> Key: BEAM-9146
> URL: https://issues.apache.org/jira/browse/BEAM-9146
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-py-gcp
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The goal is to create a PTransform that integrates Google Cloud Video 
> Intelligence functionality [1].
> The transform should be able to take both video GCS location or video data 
> bytes as an input.
> The transform should be put into _sdks/python/apache_beam/io/gcp/ai_ folder.
> [1] https://cloud.google.com/video-intelligence/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8399) Python HDFS implementation should support filenames of the format "hdfs://namenodehost/parent/child"

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8399?focusedWorklogId=381962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381962
 ]

ASF GitHub Bot logged work on BEAM-8399:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:16
Start Date: 05/Feb/20 00:16
Worklog Time Spent: 10m 
  Work Description: zhitaoli commented on issue #10223: [BEAM-8399] Add 
--hdfs_full_urls option (wip)
URL: https://github.com/apache/beam/pull/10223#issuecomment-582180187
 
 
   ping? is it possible to get this proper merged?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381962)
Time Spent: 1h 20m  (was: 1h 10m)

> Python HDFS implementation should support filenames of the format 
> "hdfs://namenodehost/parent/child"
> 
>
> Key: BEAM-8399
> URL: https://issues.apache.org/jira/browse/BEAM-8399
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Chamikara Madhusanka Jayalath
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> "hdfs://namenodehost/parent/child" and "/parent/child" seems to be the 
> correct filename formats for HDFS based on [1] but we currently support 
> format "hdfs://parent/child".
> To not break existing users, we have to either (1) somehow support both 
> versions by default (based on [2] seems like HDFS does not allow colons in 
> file path so this might be possible) (2) make  
> "hdfs://namenodehost/parent/child" optional for now and change it to default 
> after few versions.
> We should also make sure that Beam Java and Python HDFS file-system 
> implementations are consistent in this regard.
>  
> [1][https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html]
> [2] https://issues.apache.org/jira/browse/HDFS-13
>  
> cc: [~udim]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381960
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:12
Start Date: 05/Feb/20 00:12
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10766: [BEAM-4461] 
Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374995783
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
+@FieldAccess("selectFields")
+final FieldAccessDescriptor fieldAccess = 
FieldAccessDescriptor.withAllFields();
+
+public SelectDoFn(
+FieldAccessDescriptor fieldAccessDescriptor, Schema inputSchema, 
Schema outputSchema) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+  this.inputSchema = inputSchema;
+  this.outputSchema = outputSchema;
+}
+
+@ProcessElement
+public void process(@FieldAccess("selectFields") @Element Row row, 
OutputReceiver r) {
+  r.output(SelectHelpers.selectRow(row, fieldAccessDescriptor, 
inputSchema, outputSchema));
+}
+  }
+
+  public static class Fields extends PTransform, 
PCollection> {
+private FieldAccessDescriptor fieldAccessDescriptor;
+
+public Fields(FieldAccessDescriptor fieldAccessDescriptor) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+}
+
+@Override
+public PCollection expand(PCollection input) {
+  Schema inputSchema = input.getSchema();
+  FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
+  Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, 
resolved);
+  return input
+  .apply(ParDo.of(new SelectDoFn<>(resolved, inputSchema, 
outputSchema)))
+  .setRowSchema(outputSchema);
+}
   }
 
-  @Override
-  public PCollection expand(PCollection input) {
-Schema inputSchema = input.getSchema();
-FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
-Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, resolved);
-
-return input
-.apply(
-ParDo.of(
-new DoFn() {
-  // TODO: This should be the same as resolved so that Beam 
knows which fields
-  // are being accessed. Currently Beam only supports wildcard 
descriptors.
-  // Once BEAM-4457 is fixed, fix this.
-  @FieldAccess("selectFields")
-  final FieldAccessDescriptor fieldAccessDescriptor =
-  FieldAccessDescriptor.withAllFields();
-
-  @ProcessElement
-  public void process(
-

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381959
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 05/Feb/20 00:11
Start Date: 05/Feb/20 00:11
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10766: [BEAM-4461] 
Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374995503
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
 
 Review comment:
   This is an old comment. I don't know that we've fully decided what to do 
about BEAM-4457
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381959)
Time Spent: 29h 20m  (was: 29h 10m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 20m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=381955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381955
 ]

ASF GitHub Bot logged work on BEAM-8889:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:57
Start Date: 04/Feb/20 23:57
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10769: [BEAM-8889] Upgrades 
gcsio to 2.0.0
URL: https://github.com/apache/beam/pull/10769#issuecomment-582174552
 
 
   Would you try running this to see if any new linkage error is added or not?
   https://gist.github.com/suztomo/ec4834151a81f8e1d26aa0afb278818b
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381955)
Remaining Estimate: 162h 40m  (was: 162h 50m)
Time Spent: 5h 20m  (was: 5h 10m)

> Make GcsUtil use GoogleCloudStorage
> ---
>
> Key: BEAM-8889
> URL: https://issues.apache.org/jira/browse/BEAM-8889
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.16.0
>Reporter: Esun Kim
>Assignee: VASU NORI
>Priority: Major
>  Labels: gcs
>   Original Estimate: 168h
>  Time Spent: 5h 20m
>  Remaining Estimate: 162h 40m
>
> [GcsUtil|https://github.com/apache/beam/blob/master/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java]
>  is a primary class to access Google Cloud Storage on Apache Beam. Current 
> implementation directly creates GoogleCloudStorageReadChannel and 
> GoogleCloudStorageWriteChannel by itself to read and write GCS data rather 
> than using 
> [GoogleCloudStorage|https://github.com/GoogleCloudPlatform/bigdata-interop/blob/master/gcsio/src/main/java/com/google/cloud/hadoop/gcsio/GoogleCloudStorage.java]
>  which is an abstract class providing basic IO capability which eventually 
> creates channel objects. This request is about updating GcsUtil to use 
> GoogleCloudStorage to create read and write channel, which is expected 
> flexible because it can easily pick up the new change; e.g. new channel 
> implementation using new protocol without code change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-2645) Implement DisplayData translation to/from protos

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2645?focusedWorklogId=381954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381954
 ]

ASF GitHub Bot logged work on BEAM-2645:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:55
Start Date: 04/Feb/20 23:55
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10770: [BEAM-2645] 
Define the display data model type
URL: https://github.com/apache/beam/pull/10770
 
 
   Use a URN + value system with an optional label allowing for 'dynamic' types 
to be specified and labelled with a set of well known types.
   Fix-up existing Java usage.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381953
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:53
Start Date: 04/Feb/20 23:53
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582173546
 
 
   @aaltay Thank you. Now Java precommit is green.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381953)
Time Spent: 2h 50m  (was: 2h 40m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 

[jira] [Work logged] (BEAM-8889) Make GcsUtil use GoogleCloudStorage

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8889?focusedWorklogId=381950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381950
 ]

ASF GitHub Bot logged work on BEAM-8889:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:36
Start Date: 04/Feb/20 23:36
Worklog Time Spent: 10m 
  Work Description: veblush commented on pull request #10769: [BEAM-8889] 
Upgrades gcsio to 2.0.0
URL: https://github.com/apache/beam/pull/10769
 
 
   This is part of 
[BEAM-8889](https://issues.apache.org/jira/projects/BEAM/issues/BEAM-8889) to 
get new GCS connector. Since new GCS connector is based on gcsio 2.0.0, beam 
dependency should be upgraded to catch up the recent changes. Since it doesn't 
have breaking changes, it doesn't expect to have any code migration related 
with this.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @chamikaramj`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 

[jira] [Assigned] (BEAM-3545) Fn API metrics in Go SDK harness

[Robert Burke (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke reassigned BEAM-3545:
--

Assignee: Robert Burke

> Fn API metrics in Go SDK harness
> 
>
> Key: BEAM-3545
> URL: https://issues.apache.org/jira/browse/BEAM-3545
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Kenneth Knowles
>Assignee: Robert Burke
>Priority: Major
>  Labels: portability
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9250) Improve beam release script based on 2.19.0 release experience

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9250?focusedWorklogId=381948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381948
 ]

ASF GitHub Bot logged work on BEAM-9250:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:22
Start Date: 04/Feb/20 23:22
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on pull request #10768: [BEAM-9250] 
Create BQ dataset before staring virtualenv.
URL: https://github.com/apache/beam/pull/10768
 
 
   Major changes include:
   * Creating bq datasets before activating virtualenv. `bq mk` fails on py3 
env for me.
   * Clean up all created datasets before proceeding to next test.
   * Wait for processing any key to proceed to next py versioned tests.
   
   R: @markflyhigh 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 

[jira] [Created] (BEAM-9250) Improve beam release script based on 2.19.0 release experience

[Boyuan Zhang (Jira)]

Boyuan Zhang created BEAM-9250:
--

 Summary: Improve beam release script based on 2.19.0 release 
experience
 Key: BEAM-9250
 URL: https://issues.apache.org/jira/browse/BEAM-9250
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Boyuan Zhang
Assignee: Boyuan Zhang
 Fix For: 2.20.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381939
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374973771
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 ##
 @@ -1,171 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.schemas.transforms;
-
-import com.google.auto.value.AutoValue;
-import java.util.List;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.annotations.Experimental.Kind;
-import org.apache.beam.sdk.schemas.Schema;
-import org.apache.beam.sdk.schemas.Schema.Field;
-import org.apache.beam.sdk.transforms.DoFn;
-import org.apache.beam.sdk.transforms.PTransform;
-import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.SerializableFunction;
-import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.Row;
-import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
-
-/**
- * A {@link PTransform} to unnest nested rows.
- *
- * For example, consider a Row with the following nestedschema:
- *
- * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
- *
- * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
- *
- * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
- *
- * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
- * location.longitude: DOUBLE
- *
- * By default nested names are concatenated to generated the unnested name, 
however {@link
- * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
- *
- * Note that currently array and map values are not unnested.
- */
-@Experimental(Kind.SCHEMAS)
-public class Unnest {
 
 Review comment:
   I assume there's no concern with removing Unnest outright (rather than 
deprecating for a period and pointing to `Select.flattenedSchema`) since this 
is all Experimental?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381939)
Time Spent: 29h 10m  (was: 29h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381937
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374967790
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
+@FieldAccess("selectFields")
+final FieldAccessDescriptor fieldAccess = 
FieldAccessDescriptor.withAllFields();
+
+public SelectDoFn(
+FieldAccessDescriptor fieldAccessDescriptor, Schema inputSchema, 
Schema outputSchema) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+  this.inputSchema = inputSchema;
+  this.outputSchema = outputSchema;
+}
+
+@ProcessElement
+public void process(@FieldAccess("selectFields") @Element Row row, 
OutputReceiver r) {
+  r.output(SelectHelpers.selectRow(row, fieldAccessDescriptor, 
inputSchema, outputSchema));
+}
+  }
+
+  public static class Fields extends PTransform, 
PCollection> {
+private FieldAccessDescriptor fieldAccessDescriptor;
+
+public Fields(FieldAccessDescriptor fieldAccessDescriptor) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+}
+
+@Override
+public PCollection expand(PCollection input) {
+  Schema inputSchema = input.getSchema();
+  FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
+  Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, 
resolved);
+  return input
+  .apply(ParDo.of(new SelectDoFn<>(resolved, inputSchema, 
outputSchema)))
+  .setRowSchema(outputSchema);
+}
   }
 
-  @Override
-  public PCollection expand(PCollection input) {
-Schema inputSchema = input.getSchema();
-FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
-Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, resolved);
-
-return input
-.apply(
-ParDo.of(
-new DoFn() {
-  // TODO: This should be the same as resolved so that Beam 
knows which fields
-  // are being accessed. Currently Beam only supports wildcard 
descriptors.
-  // Once BEAM-4457 is fixed, fix this.
-  @FieldAccess("selectFields")
-  final FieldAccessDescriptor fieldAccessDescriptor =
-  FieldAccessDescriptor.withAllFields();
-
-  @ProcessElement
-  public void process(
- 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381942
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374974662
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/FieldAccessDescriptor.java
 ##
 @@ -300,6 +340,16 @@ public FieldAccessDescriptor withNestedField(
 return withNestedField(field, fieldAccess);
   }
 
+  public FieldAccessDescriptor withNestedFieldAs(
+  String nestedFieldName, String nestedFieldRename, FieldAccessDescriptor 
fieldAccess) {
+FieldDescriptor field =
+FieldDescriptor.builder()
+.setFieldName(nestedFieldName)
+.setFieldRename(nestedFieldRename)
+.build();
+return withNestedField(field, fieldAccess);
+  }
+
 
 Review comment:
   Consider adding a test for this in `FieldAccessDescriptorTest`, and adding a 
docstring
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381942)
Time Spent: 29h 10m  (was: 29h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381941
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374973078
 
 

 ##
 File path: 
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/utils/SchemaTestUtils.java
 ##
 @@ -90,4 +91,71 @@ public void describeTo(Description description) {
   description.appendText(expected.toString());
 }
   }
+
+  public static class RowFieldMatcher extends BaseMatcher {
 
 Review comment:
   This looks nifty, but maybe made it into this PR by accident? I don't see it 
used anywhere
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381941)
Time Spent: 29h 10m  (was: 29h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381938
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374970128
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
 ##
 @@ -344,4 +352,46 @@ private static void selectIntoRowWithQualifiers(
 throw new RuntimeException("Unexpected type " + qualifier.getKind());
 }
   }
+
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with _ characters.
+   */
+  public static final SerializableFunction, String> 
CONCAT_FIELD_NAMES =
+  l -> {
+return String.join("_", l);
+  };
+  /**
+   * This policy keeps the raw nested field name. If two differently-nested 
fields have the same
+   * name, unnesting will fail with this policy.
+   */
 
 Review comment:
   Similar bike-shed: This mentions unnesting which isn't a thing after this 
change, maybe change to flattening?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381938)
Time Spent: 29h  (was: 28h 50m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381934
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374966208
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
 
 Review comment:
   Does BEAM-4457 unblock this? it doesn't mention anything about wildcard 
descriptors.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381934)
Time Spent: 28.5h  (was: 28h 20m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 28.5h
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381943=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381943
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374969408
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
 
 Review comment:
   This should probably indicate it will concat field names with underscores
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381943)
Time Spent: 29h 10m  (was: 29h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381935
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374967493
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
+@FieldAccess("selectFields")
+final FieldAccessDescriptor fieldAccess = 
FieldAccessDescriptor.withAllFields();
+
+public SelectDoFn(
+FieldAccessDescriptor fieldAccessDescriptor, Schema inputSchema, 
Schema outputSchema) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+  this.inputSchema = inputSchema;
+  this.outputSchema = outputSchema;
+}
+
+@ProcessElement
+public void process(@FieldAccess("selectFields") @Element Row row, 
OutputReceiver r) {
+  r.output(SelectHelpers.selectRow(row, fieldAccessDescriptor, 
inputSchema, outputSchema));
+}
+  }
+
+  public static class Fields extends PTransform, 
PCollection> {
+private FieldAccessDescriptor fieldAccessDescriptor;
+
+public Fields(FieldAccessDescriptor fieldAccessDescriptor) {
+  this.fieldAccessDescriptor = fieldAccessDescriptor;
+}
+
+@Override
+public PCollection expand(PCollection input) {
+  Schema inputSchema = input.getSchema();
+  FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
+  Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, 
resolved);
+  return input
+  .apply(ParDo.of(new SelectDoFn<>(resolved, inputSchema, 
outputSchema)))
+  .setRowSchema(outputSchema);
+}
   }
 
-  @Override
-  public PCollection expand(PCollection input) {
-Schema inputSchema = input.getSchema();
-FieldAccessDescriptor resolved = 
fieldAccessDescriptor.resolve(inputSchema);
-Schema outputSchema = SelectHelpers.getOutputSchema(inputSchema, resolved);
-
-return input
-.apply(
-ParDo.of(
-new DoFn() {
-  // TODO: This should be the same as resolved so that Beam 
knows which fields
-  // are being accessed. Currently Beam only supports wildcard 
descriptors.
-  // Once BEAM-4457 is fixed, fix this.
-  @FieldAccess("selectFields")
-  final FieldAccessDescriptor fieldAccessDescriptor =
-  FieldAccessDescriptor.withAllFields();
-
-  @ProcessElement
-  public void process(
- 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381936
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374966003
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Select.java
 ##
 @@ -69,55 +75,117 @@
  * }
  */
 @Experimental(Kind.SCHEMAS)
-public class Select extends PTransform, PCollection> {
-  private final FieldAccessDescriptor fieldAccessDescriptor;
-
-  private Select(FieldAccessDescriptor fieldAccessDescriptor) {
-this.fieldAccessDescriptor = fieldAccessDescriptor;
+public class Select {
+  public static  Select.Fields create() {
+return new Fields<>(FieldAccessDescriptor.create());
   }
 
   /** Select a set of top-level field ids from the row. */
-  public static  Select fieldIds(Integer... ids) {
-return new Select<>(FieldAccessDescriptor.withFieldIds(ids));
+  public static  Fields fieldIds(Integer... ids) {
+return new Fields<>(FieldAccessDescriptor.withFieldIds(ids));
   }
 
   /** Select a set of top-level field names from the row. */
-  public static  Select fieldNames(String... names) {
-return new Select<>(FieldAccessDescriptor.withFieldNames(names));
+  public static  Fields fieldNames(String... names) {
+return new Fields<>(FieldAccessDescriptor.withFieldNames(names));
   }
 
   /**
* Select a set of fields described in a {@link FieldAccessDescriptor}.
*
* This allows for nested fields to be selected as well.
*/
-  public static  Select fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
-return new Select<>(fieldAccessDescriptor);
+  public static  Fields fieldAccess(FieldAccessDescriptor 
fieldAccessDescriptor) {
+return new Fields<>(fieldAccessDescriptor);
+  }
+
+  /**
+   * Selects every leaf-level field. This results in a a nested schema being 
flattened into a single
+   * top-level schema.
+   */
+  public static  Flattened flattenedSchema() {
+return new Flattened<>();
+  }
+
+  private static class SelectDoFn extends DoFn {
+private FieldAccessDescriptor fieldAccessDescriptor;
+private Schema inputSchema;
+private Schema outputSchema;
+
+// TODO: This should be the same as resolved so that Beam knows which 
fields
+// are being accessed. Currently Beam only supports wildcard descriptors.
+// Once BEAM-4457 is fixed, fix this.
+@FieldAccess("selectFields")
+final FieldAccessDescriptor fieldAccess = 
FieldAccessDescriptor.withAllFields();
 
 Review comment:
   It took me a minute to understand what's going on with "selectFields" here 
and in `process`. I had no idea that the value on `FieldAccess` could be an id 
referencing a member `FieldAccessDescriptor` until I [dug into the 
code](https://github.com/apache/beam/blob/2ad1f68/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java#L479-L494).
 Let's document that as part of 
[BEAM-9217](https://issues.apache.org/jira/browse/BEAM-9217) as well.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381936)
Time Spent: 28h 50m  (was: 28h 40m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 28h 50m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381940
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 23:06
Start Date: 04/Feb/20 23:06
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10766: 
[BEAM-4461] Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766#discussion_r374969691
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/SelectHelpers.java
 ##
 @@ -344,4 +352,46 @@ private static void selectIntoRowWithQualifiers(
 throw new RuntimeException("Unexpected type " + qualifier.getKind());
 }
   }
+
+  /**
+   * This is the default naming policy for naming fields. Every field name in 
the path to a given
+   * field is concated with _ characters.
+   */
 
 Review comment:
   bike-shed: Nothing in this file actually makes this a default, seems to be a 
hold-over from when the function was in `Unnest`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381940)
Time Spent: 29h 10m  (was: 29h)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 29h 10m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-9167) Reduce overhead of Go SDK side metrics

[Robert Burke (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-9167.

Fix Version/s: Not applicable
   Resolution: Fixed

SDK side performance of user metrics is now reduced significantly if the proxy 
object is used. There's other metrics related work (eg. framework metrics 
around PCollections and ParDos, programmatic extraction, using the updated 
Monitoring infos), but they are tracked by other JIRAs.

> Reduce overhead of Go SDK side metrics
> --
>
> Key: BEAM-9167
> URL: https://issues.apache.org/jira/browse/BEAM-9167
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Locking overhead due to the global store and local caches of SDK counter data 
> can dominate certain workloads, which means we can do better.
> Instead of having a global store of metrics data to extract counters, we 
> should use per ptransform (or per bundle) counter sets, which would avoid 
> requiring locking per counter operation. The main detriment compared to the 
> current implementation is that a user would need to add their own locking if 
> they were to spawn multiple goroutines to process a Bundle's work in a DoFn.
> Given that self multithreaded DoFns aren't recommended/safe in Java,  largely 
> impossible in Python, and the other beam Go SDK provided constructs (like 
> Iterators and Emitters) are not thread safe, this is a small concern, 
> provided the documentation is clear on this.
> Removing the locking and switching to atomic ops reduces the overhead 
> significantly in example jobs and in the benchmarks.
> A second part of this change should be to move the exec package to manage 
> it's own per bundle state, rather than relying on a global datastore to 
> extract the per bundle,per ptransform values.
> Related: https://issues.apache.org/jira/browse/BEAM-6541 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-7726) [Go SDK] State Backed Iterables

[Robert Burke (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-7726.

Resolution: Fixed

The Go SDK now supports using State Backed iterables if the runner triggers it.

> [Go SDK] State Backed Iterables
> ---
>
> Key: BEAM-7726
> URL: https://issues.apache.org/jira/browse/BEAM-7726
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> The Go SDK should support the State backed iterables protocol per the proto.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L644]
>  
> Primary case is for iterables after CoGBKs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-7726) [Go SDK] State Backed Iterables

[Robert Burke (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004337#comment-17004337
 ] 

Robert Burke edited comment on BEAM-7726 at 2/4/20 10:46 PM:
-

The data channel is correctly multiplexing bundles. There's no other way to do 
the multiple streams thing in the current protocol and GRPC without the runner 
having multiple endpoints, or the process doing so (eg. Multiple SDK Harnesses 
per worker, which is how python handles it).

I think I have a resolution for state backed iterables blocking the 
datachannel, which will work for any runners that support datasource split 
requests. If the data channel is eventually split down to a the current value 
and no more, we can close the reader, which will cause the channel to be 
unblocked. Any buffered data will be drained. Care needs to be taken to avoid 
deadlocking or dataloss or race conditions, but there should only be lock 
contention  when the Split thread is closing the reader.

Edit (2020/02/04): I wasn't able to confirm that this actually worked better, 
and even though there was no material locking overhead, the additional 
complexity to that part of the code isn't worth  questionable benefits. Tabling 
for now. 


was (Author: lostluck):
The data channel is correctly multiplexing bundles. There's no other way to do 
the multiple streams thing in the current protocol and GRPC without the runner 
having multiple endpoints, or the process doing so (eg. Multiple SDK Harnesses 
per worker, which is how python handles it).

I think I have a resolution for state backed iterables blocking the 
datachannel, which will work for any runners that support datasource split 
requests. If the data channel is eventually split down to a the current value 
and no more, we can close the reader, which will cause the channel to be 
unblocked. Any buffered data will be drained. Care needs to be taken to avoid 
deadlocking or dataloss or race conditions, but there should only be lock 
contention  when the Split thread is closing the reader.

Edit: I wasn't able to confirm that this actually worked better, and even 
though there was no material locking overhead, the additional complexity to 
that part of the code isn't worth  questionable benefits. Tabling for now. 

> [Go SDK] State Backed Iterables
> ---
>
> Key: BEAM-7726
> URL: https://issues.apache.org/jira/browse/BEAM-7726
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> The Go SDK should support the State backed iterables protocol per the proto.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L644]
>  
> Primary case is for iterables after CoGBKs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-7726) [Go SDK] State Backed Iterables

[Robert Burke (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004337#comment-17004337
 ] 

Robert Burke edited comment on BEAM-7726 at 2/4/20 10:45 PM:
-

The data channel is correctly multiplexing bundles. There's no other way to do 
the multiple streams thing in the current protocol and GRPC without the runner 
having multiple endpoints, or the process doing so (eg. Multiple SDK Harnesses 
per worker, which is how python handles it).

I think I have a resolution for state backed iterables blocking the 
datachannel, which will work for any runners that support datasource split 
requests. If the data channel is eventually split down to a the current value 
and no more, we can close the reader, which will cause the channel to be 
unblocked. Any buffered data will be drained. Care needs to be taken to avoid 
deadlocking or dataloss or race conditions, but there should only be lock 
contention  when the Split thread is closing the reader.

Edit: I wasn't able to confirm that this actually worked better, and even 
though there was no material locking overhead, the additional complexity to 
that part of the code isn't worth  questionable benefits. Tabling for now. 


was (Author: lostluck):
The data channel is correctly multiplexing bundles. There's no other way to do 
the multiple streams thing in the current protocol and GRPC without the runner 
having multiple endpoints, or the process doing so (eg. Multiple SDK Harnesses 
per worker, which is how python handles it).

I think I have a resolution for state backed iterables blocking the 
datachannel, which will work for any runners that support datasource split 
requests. If the data channel is eventually split down to a the current value 
and no more, we can close the reader, which will cause the channel to be 
unblocked. Any buffered data will be drained. Care needs to be taken to avoid 
deadlocking or dataloss or race conditions, but there should only be lock 
contention  when the Split thread is closing the reader.

 

> [Go SDK] State Backed Iterables
> ---
>
> Key: BEAM-7726
> URL: https://issues.apache.org/jira/browse/BEAM-7726
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> The Go SDK should support the State backed iterables protocol per the proto.
> [https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L644]
>  
> Primary case is for iterables after CoGBKs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381920
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 22:30
Start Date: 04/Feb/20 22:30
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582147693
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381920)
Time Spent: 2h 40m  (was: 2.5h)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 12:09:44.298346 
> 

[jira] [Commented] (BEAM-4457) Analyze FieldAccessDescriptors and drop fields that are never accessed

[Kirill Kozlov (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030165#comment-17030165
 ] 

Kirill Kozlov commented on BEAM-4457:
-

Definitely a cool idea! Yes, right now push-down heavily relies on the SQL 
optimizer rules, but this can help simplify them.

> Analyze FieldAccessDescriptors and drop fields that are never accessed
> --
>
> Key: BEAM-4457
> URL: https://issues.apache.org/jira/browse/BEAM-4457
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>
> We can walk backwards through the graph, analyzing which fields are accessed. 
> When we find paths where many fields are never accessed, we can insert a 
> projection transform to drop those fields preemptively. This can save a lot 
> of resources in the case where many fields in the input are never accessed.
> To do this, the FieldAccessDescriptor information must be added to the 
> portability protos. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-4457) Analyze FieldAccessDescriptors and drop fields that are never accessed

[Andrew Pilloud (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030164#comment-17030164
 ] 

Andrew Pilloud commented on BEAM-4457:
--

SQL doesn't (currently) use FieldAccessDescriptors, so it doesn't directly 
apply, but we have a similar desire to be able to run a pass of the optimizer 
before sending the graph to the runner.

> Analyze FieldAccessDescriptors and drop fields that are never accessed
> --
>
> Key: BEAM-4457
> URL: https://issues.apache.org/jira/browse/BEAM-4457
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>
> We can walk backwards through the graph, analyzing which fields are accessed. 
> When we find paths where many fields are never accessed, we can insert a 
> projection transform to drop those fields preemptively. This can save a lot 
> of resources in the case where many fields in the input are never accessed.
> To do this, the FieldAccessDescriptor information must be added to the 
> portability protos. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-4457) Analyze FieldAccessDescriptors and drop fields that are never accessed

[Brian Hulette (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030161#comment-17030161
 ] 

Brian Hulette commented on BEAM-4457:
-

This is a cool idea! Ideally it would integrate with [~kirillkozlov]'s 
projection push-down work as well (CC [~apilloud]), but I guess that's a long 
way off since push-down relies on the SQL optimizer.

Would the runner be responsible for doing this?

> Analyze FieldAccessDescriptors and drop fields that are never accessed
> --
>
> Key: BEAM-4457
> URL: https://issues.apache.org/jira/browse/BEAM-4457
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-gcp
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>
> We can walk backwards through the graph, analyzing which fields are accessed. 
> When we find paths where many fields are never accessed, we can insert a 
> projection transform to drop those fields preemptively. This can save a lot 
> of resources in the case where many fields in the input are never accessed.
> To do this, the FieldAccessDescriptor information must be added to the 
> portability protos. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381903
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 04/Feb/20 22:08
Start Date: 04/Feb/20 22:08
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10755: [BEAM-4150] Remove 
fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755#issuecomment-582139562
 
 
   Whoops. Dead comment I thought I deleted after answering my own question
   after expanding how much code was revealed.
   
   On Tue, Feb 4, 2020, 1:20 PM Lukasz Cwik  wrote:
   
   > Tthey always have one.
   >
   > On Tue, Feb 4, 2020 at 11:04 AM Robert Burke 
   > wrote:
   >
   > > *@lostluck* commented on this pull request.
   > > --
   > >
   > > In sdks/go/pkg/beam/core/runtime/exec/translate.go
   > > :
   > >
   > > > @@ -64,6 +64,13 @@ func UnmarshalPlan(desc
   > *fnpb.ProcessBundleDescriptor) (*Plan, error) {
   > > }
   > >
   > > u := {UID: b.idgen.New()}
   > > + u.Coder, err = b.coders.Coder(cid) // Expected to be windowed coder
   > > + if err != nil {
   > > + return nil, err
   > > + }
   > > + if !coder.IsW(u.Coder) {
   > > + return nil, errors.Errorf("unwindowed coder %v on DataSource %v: %v",
   > cid, id, u.Coder)
   > > + }
   > >
   > > for key, pid := range transform.GetOutputs() {
   > >
   > > Do GRPC datasources always only have a single output?
   > >
   > > I note that we aren't using the pid, in the new code.
   > >
   > > —
   > > You are receiving this because you authored the thread.
   > > Reply to this email directly, view it on GitHub
   > > <
   > 
https://github.com/apache/beam/pull/10755?email_source=notifications_token=ACM4V3D4ENJGCCU7DCMAVXLRBG333A5CNFSM4KPKV7XKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCUG4ZCI#pullrequestreview-353225865
   > >,
   > > or unsubscribe
   > > <
   > 
https://github.com/notifications/unsubscribe-auth/ACM4V3AXN6VNEWJK3UURZZ3RBG333ANCNFSM4KPKV7XA
   > >
   > > .
   > >
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381903)
Time Spent: 5h  (was: 4h 50m)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381892
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:45
Start Date: 04/Feb/20 21:45
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582130282
 
 
   Another transient error from Java Precommit check:
   
   ```
   16:06:50 > Task :runners:gearpump:test
   16:06:50 
   16:06:50 org.apache.beam.runners.gearpump.translators.io.ValueSoureTest > 
testValueSource FAILED
   16:06:50 org.jboss.netty.channel.ChannelException
   16:06:50 Caused by: java.net.BindException
   ```
   
   @aaltay Thanks!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381892)
Time Spent: 2.5h  (was: 2h 20m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> 

[jira] [Work logged] (BEAM-8382) Add polling interval to KinesisIO.Read

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8382?focusedWorklogId=381893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381893
 ]

ASF GitHub Bot logged work on BEAM-8382:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:45
Start Date: 04/Feb/20 21:45
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #9765: [WIP][BEAM-8382] 
Add rate limit policy to KinesisIO.Read
URL: https://github.com/apache/beam/pull/9765#issuecomment-582130339
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381893)
Time Spent: 11h 50m  (was: 11h 40m)

> Add polling interval to KinesisIO.Read
> --
>
> Key: BEAM-8382
> URL: https://issues.apache.org/jira/browse/BEAM-8382
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> With the current implementation we are observing Kinesis throttling due to 
> ReadProvisionedThroughputExceeded on the order of hundreds of times per 
> second, regardless of the actual Kinesis throughput. This is because the 
> ShardReadersPool readLoop() method is polling getRecords() as fast as 
> possible.
> From the KDS documentation:
> {quote}Each shard can support up to five read transactions per second.
> {quote}
> and
> {quote}For best results, sleep for at least 1 second (1,000 milliseconds) 
> between calls to getRecords to avoid exceeding the limit on getRecords 
> frequency.
> {quote}
> [https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html]
> [https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-sdk.html]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9236) Mark missing Schema based classes and methods as Experimental

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9236?focusedWorklogId=381891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381891
 ]

ASF GitHub Bot logged work on BEAM-9236:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:45
Start Date: 04/Feb/20 21:45
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10763: [BEAM-9236] Mark 
missing Schema based classes and methods as Experimental (2)
URL: https://github.com/apache/beam/pull/10763#issuecomment-582130241
 
 
   Thanks @alexvanboxel !
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381891)
Time Spent: 3h 40m  (was: 3.5h)

> Mark missing Schema based classes and methods as Experimental
> -
>
> Key: BEAM-9236
> URL: https://issues.apache.org/jira/browse/BEAM-9236
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-9236) Mark missing Schema based classes and methods as Experimental

[Jira]

 [ 
https://issues.apache.org/jira/browse/BEAM-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-9236.

Resolution: Fixed

> Mark missing Schema based classes and methods as Experimental
> -
>
> Key: BEAM-9236
> URL: https://issues.apache.org/jira/browse/BEAM-9236
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9236) Mark missing Schema based classes and methods as Experimental

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9236?focusedWorklogId=381890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381890
 ]

ASF GitHub Bot logged work on BEAM-9236:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:44
Start Date: 04/Feb/20 21:44
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10763: [BEAM-9236] 
Mark missing Schema based classes and methods as Experimental (2)
URL: https://github.com/apache/beam/pull/10763
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381890)
Time Spent: 3.5h  (was: 3h 20m)

> Mark missing Schema based classes and methods as Experimental
> -
>
> Key: BEAM-9236
> URL: https://issues.apache.org/jira/browse/BEAM-9236
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-9242) Processing Stuck messages are reported by Dataflow as errors

[Pablo Estrada (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Estrada resolved BEAM-9242.
-
Fix Version/s: 2.20.0
   Resolution: Fixed

> Processing Stuck messages are reported by Dataflow as errors
> 
>
> Key: BEAM-9242
> URL: https://issues.apache.org/jira/browse/BEAM-9242
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> These messages are only warning-level as they don't directly indicate 
> pipeline stuckness. Their being reported as errors misleads the users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381873
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:20
Start Date: 04/Feb/20 21:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10755: [BEAM-4150] Remove 
fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755#issuecomment-582120561
 
 
   Tthey always have one.
   
   On Tue, Feb 4, 2020 at 11:04 AM Robert Burke 
   wrote:
   
   > *@lostluck* commented on this pull request.
   > --
   >
   > In sdks/go/pkg/beam/core/runtime/exec/translate.go
   > :
   >
   > > @@ -64,6 +64,13 @@ func UnmarshalPlan(desc 
*fnpb.ProcessBundleDescriptor) (*Plan, error) {
   >}
   >
   >u := {UID: b.idgen.New()}
   > +  u.Coder, err = b.coders.Coder(cid) // Expected to be windowed 
coder
   > +  if err != nil {
   > +  return nil, err
   > +  }
   > +  if !coder.IsW(u.Coder) {
   > +  return nil, errors.Errorf("unwindowed coder %v on 
DataSource %v: %v", cid, id, u.Coder)
   > +  }
   >
   >for key, pid := range transform.GetOutputs() {
   >
   > Do GRPC datasources always only have a single output?
   >
   > I note that we aren't using the pid, in the new code.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381873)
Time Spent: 4h 50m  (was: 4h 40m)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3221) Model pipeline representation improvements

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3221?focusedWorklogId=381871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381871
 ]

ASF GitHub Bot logged work on BEAM-3221:


Author: ASF GitHub Bot
Created on: 04/Feb/20 21:15
Start Date: 04/Feb/20 21:15
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10754: [BEAM-3221] 
Expand/clarify timestamp comments within beam_runner_api.proto
URL: https://github.com/apache/beam/pull/10754
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381871)
Time Spent: 1h 20m  (was: 1h 10m)

> Model pipeline representation improvements
> --
>
> Key: BEAM-3221
> URL: https://issues.apache.org/jira/browse/BEAM-3221
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Collections of various (breaking) tweaks to the Runner API, notably the 
> pipeline representation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381848
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 20:37
Start Date: 04/Feb/20 20:37
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582103691
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381848)
Time Spent: 2h 20m  (was: 2h 10m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 12:09:44.298346 
> 

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=381846=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381846
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 04/Feb/20 20:29
Start Date: 04/Feb/20 20:29
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #10766: [BEAM-4461] 
Add Selected.flattenedSchema
URL: https://github.com/apache/beam/pull/10766
 
 
   R: @TheNeuralBit 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381846)
Time Spent: 28h 20m  (was: 28h 10m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 28h 20m
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381845
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 20:27
Start Date: 04/Feb/20 20:27
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582099408
 
 
   Java Precommit failed:
   
   ```
   14:41:02 Execution failed for task ':sdks:java:io:rabbitmq:test'.
   14:41:02 > Process 'Gradle Test Executor 113' finished with non-zero exit 
value 143
   14:41:02   This problem might be caused by incorrect test process 
configuration.
   ```
   
   A usual transient error.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381845)
Time Spent: 2h 10m  (was: 2h)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current 

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381841
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 04/Feb/20 20:20
Start Date: 04/Feb/20 20:20
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10734: [BEAM-8979] 
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582096914
 
 
   I'll fork mypy-protobuf and see if I can modify setup.py fix the issue, and
   if it works I'll make a PR for it there. I can see that they're not using
   the entry-points feature of setuptools, so that might just fix it.
   
   
   
   On Tue, Feb 4, 2020 at 11:59 AM Udi Meiri  wrote:
   
   > Perhaps if we put in a wrapper script for mypy-protobuf...
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381841)
Time Spent: 6.5h  (was: 6h 20m)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
>  line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32   File "/usr/lib/python3.7/distutils/cmd.py", line 313, in 
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File "setup.py", line 220, in run

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381839
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:59
Start Date: 04/Feb/20 19:59
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce 
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582088284
 
 
   Perhaps if we put in a wrapper script for mypy-protobuf...
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381839)
Time Spent: 6h 20m  (was: 6h 10m)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
>  line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32   File "/usr/lib/python3.7/distutils/cmd.py", line 313, in 
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
>  line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for 
> details): 1
> {code}
>  
> This is what I have tried so far to resolve this (without being successful):
>  * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter 
> to the _protoc_ call ingen_protos.py:131
>  * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9009) Python: Add per-unit test timeout

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9009?focusedWorklogId=381835=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381835
 ]

ASF GitHub Bot logged work on BEAM-9009:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:54
Start Date: 04/Feb/20 19:54
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10437: [BEAM-9009] Add 
pytest-timeout plugin, set timeout
URL: https://github.com/apache/beam/pull/10437
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381835)
Time Spent: 40m  (was: 0.5h)

> Python: Add per-unit test timeout
> -
>
> Key: BEAM-9009
> URL: https://issues.apache.org/jira/browse/BEAM-9009
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Initially proposing a generous timeout of 10m, in part due to Jenkins 
> slowness.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-4150) Standardize use of PCollection coder proto attribute

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4150.
-
Fix Version/s: (was: Not applicable)
   2.20.0
 Assignee: Luke Cwik
   Resolution: Fixed

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.20.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3221) Model pipeline representation improvements

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3221?focusedWorklogId=381834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381834
 ]

ASF GitHub Bot logged work on BEAM-3221:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:43
Start Date: 04/Feb/20 19:43
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10754: [BEAM-3221] 
Expand/clarify timestamp comments within beam_runner_api.proto
URL: https://github.com/apache/beam/pull/10754#issuecomment-582081882
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381834)
Time Spent: 1h 10m  (was: 1h)

> Model pipeline representation improvements
> --
>
> Key: BEAM-3221
> URL: https://issues.apache.org/jira/browse/BEAM-3221
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Collections of various (breaking) tweaks to the Runner API, notably the 
> pipeline representation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381832=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381832
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:40
Start Date: 04/Feb/20 19:40
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10755: [BEAM-4150] 
Remove fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381832)
Time Spent: 4h 40m  (was: 4.5h)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381831
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:31
Start Date: 04/Feb/20 19:31
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce 
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582076522
 
 
   As for shortened test names, please go ahead unless you think fixing 
mypy-protobuf would be faster.
   
   The names should all start with `beam_`.
   It probably makes sense to only touch Python related Jenkins jobs.
   List of jobs:
   https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381831)
Time Spent: 6h 10m  (was: 6h)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
>  line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32   File "/usr/lib/python3.7/distutils/cmd.py", line 313, in 
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
>  line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for 
> details): 1
> {code}
>  
> This is what I have tried so far to resolve this (without being successful):
>  * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter 
> 

[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381830=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381830
 ]

ASF GitHub Bot logged work on BEAM-8979:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:26
Start Date: 04/Feb/20 19:26
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce 
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-582074458
 
 
   I did a lot of digging into distutils, and it seems that there is already a 
[solution](https://bitbucket.org/pypa/distlib/src/6818e71809c2779e5ad82e6017ea570eb590186f/distlib/scripts.py?at=master#lines-124)
 for this problem. Some packages have it, for example:
   ```
   $ head 
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/easy_install
   #!/bin/sh
   '''exec' 
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/python2.7
 "$0" "$@"
   ' '''
   ```
   
   I'm not sure what's up with mypy-protobuf, but perhaps it's using another 
installation method?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381830)
Time Spent: 6h  (was: 5h 50m)

> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding 
> protoc-gen-mypy. The following tests are affected (there might be more):
>  * 
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
>  * 
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>  
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
>  (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto 
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto 
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32 
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
>  UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32   normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32   File "setup.py", line 295, in 
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
>  line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32   File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 966, in 
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File 
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
>  line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32   File "/usr/lib/python3.7/distutils/cmd.py", line 313, in 
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32   File "/usr/lib/python3.7/distutils/dist.py", line 985, in 
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32   File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32   File 
> 

[jira] [Commented] (BEAM-8919) Move JAVA_11_HOME and JAVA_8_HOME variables to Jenkins envs.

[Mikhail Gryzykhin (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030072#comment-17030072
 ] 

Mikhail Gryzykhin commented on BEAM-8919:
-

[~markliu] Can you take a look?

> Move JAVA_11_HOME and JAVA_8_HOME variables to Jenkins envs.
> 
>
> Key: BEAM-8919
> URL: https://issues.apache.org/jira/browse/BEAM-8919
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Priority: Minor
>
> Some tests that use different java versions rely on the following paths to 
> java home:
> final String JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
> final String JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
>  
> The paths itself should be held as jenkins env variables. Benefits: 
>  - easier to reuse
>  - no room for typo in the path
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381825
 ]

ASF GitHub Bot logged work on BEAM-9219:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:10
Start Date: 04/Feb/20 19:10
Worklog Time Spent: 10m 
  Work Description: davidwrede commented on issue #10745: [BEAM-9219] 
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#issuecomment-582067451
 
 
   @tvalentyn I incorporated the majority of your feedback, but left the venv 
instructions in for Python 3 as that seems to be the 
[guidance](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#installing-virtualenv)
 from python.org.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381825)
Time Spent: 2h 50m  (was: 2h 40m)

> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: David Wrede
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK 
> dependency pages more relevant and up-to-date while reducing the amount of 
> time it takes to provide that information. The current method of scraping and 
> copying dependencies into a table for every release is a non-trivial task 
> because of the semi-automated workflows done by the tech writers on the 
> website.
> In an effort to provide accurate dependency listings that are always in sync 
> with SDK releases, referring people to the appropriate places in the source 
> code (or through CLI commands) should provide people the information they are 
> looking for and not require the creation and maintenance of an automated 
> tooling solution to generate the dependency tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381823
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:04
Start Date: 04/Feb/20 19:04
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #10755: [BEAM-4150] 
Remove fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755#discussion_r374863985
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
 ##
 @@ -64,6 +64,13 @@ func UnmarshalPlan(desc *fnpb.ProcessBundleDescriptor) 
(*Plan, error) {
}
 
u := {UID: b.idgen.New()}
+   u.Coder, err = b.coders.Coder(cid) // Expected to be windowed 
coder
+   if err != nil {
+   return nil, err
+   }
+   if !coder.IsW(u.Coder) {
+   return nil, errors.Errorf("unwindowed coder %v on 
DataSource %v: %v", cid, id, u.Coder)
+   }
 
for key, pid := range transform.GetOutputs() {
 
 Review comment:
   Do GRPC datasources always only have a single output?
   
   I note that we aren't using the pid, in the new code.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381823)
Time Spent: 4h 20m  (was: 4h 10m)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381824
 ]

ASF GitHub Bot logged work on BEAM-4150:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:04
Start Date: 04/Feb/20 19:04
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #10755: [BEAM-4150] 
Remove fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755#discussion_r374863985
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
 ##
 @@ -64,6 +64,13 @@ func UnmarshalPlan(desc *fnpb.ProcessBundleDescriptor) 
(*Plan, error) {
}
 
u := {UID: b.idgen.New()}
+   u.Coder, err = b.coders.Coder(cid) // Expected to be windowed 
coder
+   if err != nil {
+   return nil, err
+   }
+   if !coder.IsW(u.Coder) {
+   return nil, errors.Errorf("unwindowed coder %v on 
DataSource %v: %v", cid, id, u.Coder)
+   }
 
for key, pid := range transform.GetOutputs() {
 
 Review comment:
   Do GRPC datasources always only have a single output?
   
   I note that we aren't using the pid, in the new code.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381824)
Time Spent: 4.5h  (was: 4h 20m)

> Standardize use of PCollection coder proto attribute
> 
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
>  Issue Type: Task
>  Components: beam-model
>Reporter: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw 
> ElementCoder. We should use the same convention (decided in discussion to be 
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the 
> attached windowing strategy, and the input/output ports should specify the 
> encoding directly rather than read the adjacent PCollection coder fields. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9236) Mark missing Schema based classes and methods as Experimental

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9236?focusedWorklogId=381821=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381821
 ]

ASF GitHub Bot logged work on BEAM-9236:


Author: ASF GitHub Bot
Created on: 04/Feb/20 19:01
Start Date: 04/Feb/20 19:01
Worklog Time Spent: 10m 
  Work Description: alexvanboxel commented on issue #10763: [BEAM-9236] 
Mark missing Schema based classes and methods as Experimental (2)
URL: https://github.com/apache/beam/pull/10763#issuecomment-582063853
 
 
   LGTM
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381821)
Time Spent: 3h 20m  (was: 3h 10m)

> Mark missing Schema based classes and methods as Experimental
> -
>
> Key: BEAM-9236
> URL: https://issues.apache.org/jira/browse/BEAM-9236
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381802
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:43
Start Date: 04/Feb/20 18:43
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582056186
 
 
   SQL Post Commit Tests failed.
   
   ```
   java.lang.ClassCastException: java.lang.Integer cannot be cast to 
java.lang.Long
at org.apache.beam.sdk.coders.VarLongCoder.encode(VarLongCoder.java:35)
at 
org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:239)
   ```
   
https://builds.apache.org/job/beam_PostCommit_SQL_PR/242/testReport/junit/org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog/DataCatalogGCSIT/testReadFromGCS/
   
   Asking in https://github.com/apache/beam/pull/10563#issuecomment-582055815 
for regression.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381802)
Time Spent: 2h  (was: 1h 50m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> 

[jira] [Work logged] (BEAM-7310) Confluent Schema Registry support in KafkaIO

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7310?focusedWorklogId=381801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381801
 ]

ASF GitHub Bot logged work on BEAM-7310:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:43
Start Date: 04/Feb/20 18:43
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10563: [BEAM-7310] Add 
support of Confluent Schema Registry for KafkaIO
URL: https://github.com/apache/beam/pull/10563#issuecomment-582055815
 
 
   @aromanenko-dev @iemejia "SQL Postcommit" started failing after this commit. 
Would you check whether the failure is related to this PR?
   
   
![image](https://user-images.githubusercontent.com/28604/73775512-05e5e980-4754-11ea-8908-2e1b032d7090.png)
   
   https://builds.apache.org/job/beam_PostCommit_SQL/3924/
   
   ```
   
org.apache.beam.sdk.extensions.sql.meta.provider.datacatalog.DataCatalogGCSIT.testReadFromGCS
   
   
   java.lang.ClassCastException: java.lang.Integer cannot be cast to 
java.lang.Long
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381801)
Time Spent: 13h  (was: 12h 50m)

> Confluent Schema Registry support in KafkaIO
> 
>
> Key: BEAM-7310
> URL: https://issues.apache.org/jira/browse/BEAM-7310
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Affects Versions: 2.12.0
>Reporter: Yohei Shimomae
>Assignee: Alexey Romanenko
>Priority: Minor
> Fix For: 2.20.0
>
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> Confluent Schema Registry is useful when we manage Avro Schema but  KafkaIO 
> does not support Confluent Schema Registry as discussed here.
> https://stackoverflow.com/questions/56035121/unable-to-connect-from-dataflow-job-to-schema-registry-when-schema-registry-requ
> https://lists.apache.org/thread.html/7695fccddebd08733b80ae1e43b79b636b63cd5fe583a2bdeecda6c4@%3Cuser.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8965) WriteToBigQuery failed in BundleBasedDirectRunner

[Chun Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chun Yang updated BEAM-8965:

Description: 
*{{WriteToBigQuery}}* fails in *{{BundleBasedDirectRunner}}* with error 
{{PCollection of size 2 with more than one element accessed as a singleton 
view.}}

Here is the code

 
{code:python}
with Pipeline() as p:
query_results = (
p 
| beam.io.Read(beam.io.BigQuerySource(
query='SELECT ... FROM ...')
)
query_results | beam.io.gcp.WriteToBigQuery(
table=,
method=WriteToBigQuery.Method.FILE_LOADS,
schema={"fields": []}
)
{code}
 

Here is the error

 
{code:none}
  File "apache_beam/runners/common.py", line 778, in 
apache_beam.runners.common.DoFnRunner.process
    def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 782, in 
apache_beam.runners.common.DoFnRunner.process
    self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 849, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented
    raise_with_traceback(new_exn)
  File "apache_beam/runners/common.py", line 780, in 
apache_beam.runners.common.DoFnRunner.process
    return self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 587, in 
apache_beam.runners.common.PerWindowInvoker.invoke_process
    self._invoke_process_per_window(
  File "apache_beam/runners/common.py", line 610, in 
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    [si[global_window] for si in self.side_inputs]))
  File 
"/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
 line 65, in __getitem__
    _FilteringIterable(self._iterable, target_window), self._view_options)
  File 
"/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
 line 443, in _from_runtime_iterable
    len(head), str(head[0]), str(head[1])))
ValueError: PCollection of size 2 with more than one element accessed as a 
singleton view. First two elements encountered are 
"gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f", 
"gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f". [while running 
'WriteToBigQuery/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)']
{code}
 

 

 

 

  was:
{{*{{WriteToBigQuery}}* failed in }}{{*BundleBasedDirectRunner*}}{{ with error 
PCollection of size 2 with more than one element accessed as a singleton view.}}

Here is the code

 
{code:java}
with Pipeline() as p:
query_results = (
p 
| beam.io.Read(beam.io.BigQuerySource(
query='SELECT ... FROM ...')
)
query_results | beam.io.gcp.WriteToBigQuery(
table=,
method=WriteToBigQuery.Method.FILE_LOADS,
schema={"fields": []}
)
{code}
 

Here is the error

 
{code:java}
  File "apache_beam/runners/common.py", line 778, in 
apache_beam.runners.common.DoFnRunner.process
    def process(self, windowed_value):
  File "apache_beam/runners/common.py", line 782, in 
apache_beam.runners.common.DoFnRunner.process
    self._reraise_augmented(exn)
  File "apache_beam/runners/common.py", line 849, in 
apache_beam.runners.common.DoFnRunner._reraise_augmented
    raise_with_traceback(new_exn)
  File "apache_beam/runners/common.py", line 780, in 
apache_beam.runners.common.DoFnRunner.process
    return self.do_fn_invoker.invoke_process(windowed_value)
  File "apache_beam/runners/common.py", line 587, in 
apache_beam.runners.common.PerWindowInvoker.invoke_process
    self._invoke_process_per_window(
  File "apache_beam/runners/common.py", line 610, in 
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
    [si[global_window] for si in self.side_inputs]))
  File 
"/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/transforms/sideinputs.py",
 line 65, in __getitem__
    _FilteringIterable(self._iterable, target_window), self._view_options)
  File 
"/home/wbai/terra/terra_py2/local/lib/python2.7/site-packages/apache_beam/pvalue.py",
 line 443, in _from_runtime_iterable
    len(head), str(head[0]), str(head[1])))
ValueError: PCollection of size 2 with more than one element accessed as a 
singleton view. First two elements encountered are 
"gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f", 
"gs://temp-dev/temp/bq_load/3edbf2172dd540edb5c8e9597206b10f". [while running 
'WriteToBigQuery/BigQueryBatchFileLoads/ParDo(WriteRecordsToFile)/ParDo(WriteRecordsToFile)']
{code}
 

 

 

 


> WriteToBigQuery failed in BundleBasedDirectRunner
> -
>
> Key: BEAM-8965
> URL: https://issues.apache.org/jira/browse/BEAM-8965
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp
>Affects Versions: 

[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381786
 ]

ASF GitHub Bot logged work on BEAM-9178:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:30
Start Date: 04/Feb/20 18:30
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #10634: [BEAM-9178] 
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374846150
 
 

 ##
 File path: 
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
 ##
 @@ -770,88 +765,34 @@ private RexNode convertResolvedFunctionCall(
   throw new UnsupportedOperationException(
   "Only support TUMBLE, HOP AND SESSION functions right now.");
   }
-} else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
-  op =
-  
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
-  functionCall.getFunction().getName());
-
+} else if ("ZetaSQL".equals(funGroup)) {
   if (op == null) {
-throw new UnsupportedOperationException(
-"Does not support ZetaSQL function: " + 
functionCall.getFunction().getName());
+TypeKind returnType =
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_NAME_TO_RETURN_TYPE.get(funName);
+if (returnType != null) {
+  op =
+  SqlOperators.createSimpleSqlFunction(
+  funName, ZetaSqlUtils.zetaSqlTypeToCalciteType(returnType));
+} else {
+  throw new UnsupportedOperationException("Does not support ZetaSQL 
function: " + funName);
+}
   }
 
-  // There are different processes to handle argument conversion because 
INTERVAL is not a
-  // type in ZetaSQL.
-  if 
(FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
-return convertTimestampAddFunction(functionCall, columnList, 
fieldList);
-  } else {
-for (ResolvedExpr expr : functionCall.getArgumentList()) {
-  operands.add(convertRexNodeFromResolvedExpr(expr, columnList, 
fieldList));
-}
+  for (ResolvedExpr expr : functionCall.getArgumentList()) {
+operands.add(convertRexNodeFromResolvedExpr(expr, columnList, 
fieldList));
   }
 } else {
-  throw new UnsupportedOperationException(
-  "Does not support function group: " + 
functionCall.getFunction().getGroup());
+  throw new UnsupportedOperationException("Does not support function 
group: " + funGroup);
 }
 
 SqlOperatorRewriter rewriter =
-
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
-functionCall.getFunction().getName());
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(funName);
 
 if (rewriter != null) {
-  ret = rewriter.apply(rexBuilder(), operands);
+  return rewriter.apply(rexBuilder(), operands);
 } else {
-  ret = rexBuilder().makeCall(op, operands);
-}
-return ret;
-  }
-
-  private RexNode convertTimestampAddFunction(
-  ResolvedFunctionCall functionCall,
-  List columnList,
-  List fieldList) {
-
-TimeUnit unit =
-TIME_UNIT_CASTING_MAP.get(
-((ResolvedLiteral) 
functionCall.getArgumentList().get(2)).getValue().getEnumValue());
-
-if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
 
 Review comment:
   Yeah right now it might work inconsistently. We can throw an exception 
during timestamp value conversion 
([here](https://github.com/apache/beam/blob/39c4a9fafb82dce126870a129c7848470d21006d/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUtils.java#L186))
 if we see microsecond precision used, but I am not sure if we should do that. 
One  down side of that is we might break functions unnecessarily (e.g. 
CURRENT_TIMESTAMP). Or maybe we can leave it as it is and see if we can fully 
fix the precision problem?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381786)
Time Spent: 1h 50m  (was: 1h 40m)

> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql-zetasql
>Reporter: Yueyang 

[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381785=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381785
 ]

ASF GitHub Bot logged work on BEAM-9178:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:30
Start Date: 04/Feb/20 18:30
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #10634: [BEAM-9178] 
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374846150
 
 

 ##
 File path: 
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
 ##
 @@ -770,88 +765,34 @@ private RexNode convertResolvedFunctionCall(
   throw new UnsupportedOperationException(
   "Only support TUMBLE, HOP AND SESSION functions right now.");
   }
-} else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
-  op =
-  
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
-  functionCall.getFunction().getName());
-
+} else if ("ZetaSQL".equals(funGroup)) {
   if (op == null) {
-throw new UnsupportedOperationException(
-"Does not support ZetaSQL function: " + 
functionCall.getFunction().getName());
+TypeKind returnType =
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_NAME_TO_RETURN_TYPE.get(funName);
+if (returnType != null) {
+  op =
+  SqlOperators.createSimpleSqlFunction(
+  funName, ZetaSqlUtils.zetaSqlTypeToCalciteType(returnType));
+} else {
+  throw new UnsupportedOperationException("Does not support ZetaSQL 
function: " + funName);
+}
   }
 
-  // There are different processes to handle argument conversion because 
INTERVAL is not a
-  // type in ZetaSQL.
-  if 
(FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
-return convertTimestampAddFunction(functionCall, columnList, 
fieldList);
-  } else {
-for (ResolvedExpr expr : functionCall.getArgumentList()) {
-  operands.add(convertRexNodeFromResolvedExpr(expr, columnList, 
fieldList));
-}
+  for (ResolvedExpr expr : functionCall.getArgumentList()) {
+operands.add(convertRexNodeFromResolvedExpr(expr, columnList, 
fieldList));
   }
 } else {
-  throw new UnsupportedOperationException(
-  "Does not support function group: " + 
functionCall.getFunction().getGroup());
+  throw new UnsupportedOperationException("Does not support function 
group: " + funGroup);
 }
 
 SqlOperatorRewriter rewriter =
-
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
-functionCall.getFunction().getName());
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(funName);
 
 if (rewriter != null) {
-  ret = rewriter.apply(rexBuilder(), operands);
+  return rewriter.apply(rexBuilder(), operands);
 } else {
-  ret = rexBuilder().makeCall(op, operands);
-}
-return ret;
-  }
-
-  private RexNode convertTimestampAddFunction(
-  ResolvedFunctionCall functionCall,
-  List columnList,
-  List fieldList) {
-
-TimeUnit unit =
-TIME_UNIT_CASTING_MAP.get(
-((ResolvedLiteral) 
functionCall.getArgumentList().get(2)).getValue().getEnumValue());
-
-if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
 
 Review comment:
   Yeah right now it might work inconsistently. We can throw an exception 
during timestamp value conversion 
([here](https://github.com/apache/beam/blob/39c4a9fafb82dce126870a129c7848470d21006d/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlUtils.java#L186))
 if we see microsecond precision used, but I am not sure if we should do that. 
One  down side of that is we might break functions unnecessarily (e.g. 
CURRENT_TIMESTAMP). Or maybe we can leave it as it is and see if we can fix the 
precision problem at once?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381785)
Time Spent: 1h 40m  (was: 1.5h)

> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql-zetasql
>Reporter: Yueyang 

[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381783=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381783
 ]

ASF GitHub Bot logged work on BEAM-9178:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:24
Start Date: 04/Feb/20 18:24
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #10634: [BEAM-9178] 
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374842900
 
 

 ##
 File path: 
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
 ##
 @@ -770,88 +765,34 @@ private RexNode convertResolvedFunctionCall(
   throw new UnsupportedOperationException(
   "Only support TUMBLE, HOP AND SESSION functions right now.");
   }
-} else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
-  op =
-  
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
-  functionCall.getFunction().getName());
-
+} else if ("ZetaSQL".equals(funGroup)) {
   if (op == null) {
-throw new UnsupportedOperationException(
-"Does not support ZetaSQL function: " + 
functionCall.getFunction().getName());
+TypeKind returnType =
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_NAME_TO_RETURN_TYPE.get(funName);
 
 Review comment:
   Great suggestion! Removing the manual map simplifies the code quite a lot.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381783)
Time Spent: 1.5h  (was: 1h 20m)

> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). 
> See the full list of functions below:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381782=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381782
 ]

ASF GitHub Bot logged work on BEAM-9178:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:23
Start Date: 04/Feb/20 18:23
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #10634: [BEAM-9178] 
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374842360
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQuerySqlDialect.java
 ##
 @@ -302,6 +316,38 @@ private void unparseTrim(SqlWriter writer, SqlCall call, 
int leftPrec, int right
 writer.endFunCall(trimFrame);
   }
 
+  /**
+   * For usage of INTERVAL, see https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#timestamp_add;>
+   * BQ TIMESTAMP_ADD function for example.
+   */
+  private void unparseFunctionsUsingInterval(
+  SqlWriter writer, SqlCall call, int leftPrec, int rightPrec) {
+// e.g. TIMESTAMP_ADD syntax:
+// TIMESTAMP_ADD(timestamp_expression, INTERVAL int64_expression date_part)
+int operandCount = call.operandCount();
+if (operandCount == 2) {
+  // operand0: timestamp_expression
+  // operand1: SqlIntervalLiteral (INTERVAL int64_expression date_part)
+  super.unparseCall(writer, call, leftPrec, rightPrec);
+} else if (operandCount == 3) {
+  // operand0: timestamp_expression
+  // operand1: int64_expression
+  // operand2: date_part
+  final SqlWriter.Frame frame = 
writer.startFunCall(call.getOperator().getName());
+  call.operand(0).unparse(writer, leftPrec, rightPrec);
+  writer.literal(",");
+  writer.literal("INTERVAL");
+  call.operand(1).unparse(writer, leftPrec, rightPrec);
+  call.operand(2).unparse(writer, leftPrec, rightPrec);
+  writer.endFunCall(frame);
+} else {
+  throw new IllegalArgumentException(
+  String.format(
+  "Unable to parse %s with %d operands.", 
call.getOperator().getName(), operandCount));
 
 Review comment:
   Fixed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381782)
Time Spent: 1h 20m  (was: 1h 10m)

> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). 
> See the full list of functions below:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381778
 ]

ASF GitHub Bot logged work on BEAM-9219:


Author: ASF GitHub Bot
Created on: 04/Feb/20 18:20
Start Date: 04/Feb/20 18:20
Worklog Time Spent: 10m 
  Work Description: soyrice commented on issue #10745: [BEAM-9219] 
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#issuecomment-582046178
 
 
   LGTM after updating "compile and runtime dependencies" in the Python deps 
doc to "dependencies"  per Valentyn's comment
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381778)
Time Spent: 2h 40m  (was: 2.5h)

> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: David Wrede
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK 
> dependency pages more relevant and up-to-date while reducing the amount of 
> time it takes to provide that information. The current method of scraping and 
> copying dependencies into a table for every release is a non-trivial task 
> because of the semi-automated workflows done by the tech writers on the 
> website.
> In an effort to provide accurate dependency listings that are always in sync 
> with SDK releases, referring people to the appropriate places in the source 
> code (or through CLI commands) should provide people the information they are 
> looking for and not require the creation and maintenance of an automated 
> tooling solution to generate the dependency tables.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=381773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381773
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:58
Start Date: 04/Feb/20 17:58
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10747: [BEAM-6857] 
Recategorize UsesTimerMap tests to ValidatesRunner
URL: https://github.com/apache/beam/pull/10747#issuecomment-582036586
 
 
   @rehmanmuradali it appears that the TimerMap PR did not work for Dataflow. 
Unfortunately due to a bug in the test description (one I didn't catch in code 
review), the test never ran on Dataflow.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381773)
Time Spent: 30h 20m  (was: 30h 10m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 30h 20m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6703) Support Java 11 in Jenkins

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6703?focusedWorklogId=381772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381772
 ]

ASF GitHub Bot logged work on BEAM-6703:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:57
Start Date: 04/Feb/20 17:57
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10689: [BEAM-6703] 
Make Dataflow ValidatesRunner test use Java 11 in test execution
URL: https://github.com/apache/beam/pull/10689#discussion_r374829194
 
 

 ##
 File path: 
.test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Dataflow_Java11.groovy
 ##
 @@ -20,26 +20,40 @@ import CommonJobProperties as commonJobProperties
 import PostcommitJobBuilder
 
 
-PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java11_ValidatesRunner_Dataflow',
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11',
   'Run Dataflow ValidatesRunner Java 11', 'Google Cloud Dataflow Runner 
ValidatesRunner Tests On Java 11', this) {
 
   description('Runs the ValidatesRunner suite on the Dataflow runner with Java 
11 worker harness.')
 
-  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
+  def JAVA_11_HOME = '/usr/lib/jvm/java-11-openjdk-amd64'
+  def JAVA_8_HOME = '/usr/lib/jvm/java-8-openjdk-amd64'
 
+  commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 270)
   publishers {
 archiveJunit('**/build/test-results/**/*.xml')
   }
 
   steps {
+gradle {
+  rootBuildScriptDir(commonJobProperties.checkoutDir)
 
 Review comment:
   I don't fully understand your response.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381772)
Time Spent: 19.5h  (was: 19h 20m)

> Support Java 11 in Jenkins
> --
>
> Key: BEAM-6703
> URL: https://issues.apache.org/jira/browse/BEAM-6703
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow, runner-direct
>Reporter: Michał Walenia
>Assignee: Michał Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 19.5h
>  Remaining Estimate: 0h
>
> In this issue I'll create a Jenkins job that compiles Dataflow and Direct 
> runners with tests using Java 8 and runs Validates Runner suites with Java 11 
> Runtime.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381767
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:51
Start Date: 04/Feb/20 17:51
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582033493
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381767)
Time Spent: 1h 20m  (was: 1h 10m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 12:09:44.298346 
> 

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381766
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:51
Start Date: 04/Feb/20 17:51
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582033457
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381766)
Time Spent: 1h 10m  (was: 1h)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381764=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381764
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:51
Start Date: 04/Feb/20 17:51
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582033367
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381764)
Time Spent: 50m  (was: 40m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 12:09:44.298346 
> -
> 

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381765
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:51
Start Date: 04/Feb/20 17:51
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582033395
 
 
   Run Java HadoopFormatIO Performance Test
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381765)
Time Spent: 1h  (was: 50m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-27 12:09:44.298346 
> 

[jira] [Work logged] (BEAM-8758) Beam Dependency Update Request: com.google.cloud:google-cloud-spanner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8758?focusedWorklogId=381770=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381770
 ]

ASF GitHub Bot logged work on BEAM-8758:


Author: ASF GitHub Bot
Created on: 04/Feb/20 17:51
Start Date: 04/Feb/20 17:51
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #10765: [BEAM-8758] 
Google-cloud-spanner upgrade to 1.49.1
URL: https://github.com/apache/beam/pull/10765#issuecomment-582033304
 
 
   Run Java PostCommit
   Run Java HadoopFormatIO Performance Test
   Run BigQueryIO Streaming Performance Test Java
   Run Dataflow ValidatesRunner
   Run Spark ValidatesRunner
   Run SQL Postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 381770)
Time Spent: 1h 50m  (was: 1h 40m)

> Beam Dependency Update Request: com.google.cloud:google-cloud-spanner
> -
>
> Key: BEAM-8758
> URL: https://issues.apache.org/jira/browse/BEAM-8758
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
>  - 2019-11-19 21:05:29.289016 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:11:08.926875 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.46.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:10:16.400168 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:10:17.656471 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:49.080960 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:09:23.346857 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-13 12:09:02.023131 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.47.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-20 12:08:38.419575 
> -
> Please consider upgrading the dependency 
> com.google.cloud:google-cloud-spanner. 
> The current version is 1.6.0. The latest version is 1.48.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
>