[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381406=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381406
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 06:20
Start Date: 04/Feb/20 06:20
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10734: [BEAM-8979]
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581760701
Nice find. Well now we know what the two failing jobs have in common: really
long names!
Shorter names would be a good change regardless. Those two job names feel a
bit like a word salad right now.
How about these:
- `Python_BigQueryIO_Write_Perf`
- `Python37_Dataflow_ParDo`
I don't actually know what these do so I just picked out the most important
sounding bits, and tried to harmonize them with our prevailing conventions.
Feel free to have a go at it.
It would also be good to add a comment about this issue to the jenkins
groovy code that generates these, or even add an assertion. Do you want to
take this issue or would you like me to?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381406)
Time Spent: 5h 50m (was: 5h 40m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5h 50m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
>
[jira] [Work logged] (BEAM-6857) Support dynamic timers
[
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=381396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381396
]
ASF GitHub Bot logged work on BEAM-6857:
Author: ASF GitHub Bot
Created on: 04/Feb/20 05:07
Start Date: 04/Feb/20 05:07
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10747: [BEAM-6857]
Recategorize UsesTimerMap tests to ValidatesRunner
URL: https://github.com/apache/beam/pull/10747#discussion_r374476134
##
File path: runners/google-cloud-dataflow-java/build.gradle
##
@@ -137,6 +137,7 @@ def commonExcludeCategories = [
'org.apache.beam.sdk.testing.UsesGaugeMetrics',
'org.apache.beam.sdk.testing.UsesSetState',
'org.apache.beam.sdk.testing.UsesMapState',
+ 'org.apache.beam.sdk.testing.UsesTimerMap',
Review comment:
If it was broken, we need to fix. Support for this was added for Dataflow,
though it now appears that the test was never being run :(
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381396)
Time Spent: 29h 50m (was: 29h 40m)
> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 29h 50m
> Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified
> in the DoFn. The user must provide a separate callback method per timer. For
> example:
>
> {code:java}
> DoFn()
> {
> @TimerId("timer1")
> private final TimerSpec timer1 = TimerSpecs.timer(...);
> @TimerId("timer2")
> private final TimerSpec timer2 = TimerSpecs.timer(...);
> .. set timers in processElement
> @OnTimer("timer1")
> public void onTimer1() { .}
> @OnTimer("timer2")
> public void onTimer2() {}
> }
> {code}
>
> However there are many cases where the user does not know the set of timers
> statically when writing their code. This happens when the timer tag should be
> based on the data. It also happens when writing a DSL on top of Beam, where
> the DSL author has to create DoFns but does not know statically which timers
> their users will want to set (e.g. Scio).
>
> The goal is to support dynamic timers. Something as follows;
>
> {code:java}
> DoFn()
> {
> @TimerId("timer")
> private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
> @ProcessElement process(@TimerId("timer") DynamicTimer timer)
> {
> timer.set("tag1'", ts);
>timer.set("tag2", ts);
> }
> @OnTimer("timer")
> public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6857) Support dynamic timers
[
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=381383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381383
]
ASF GitHub Bot logged work on BEAM-6857:
Author: ASF GitHub Bot
Created on: 04/Feb/20 04:35
Start Date: 04/Feb/20 04:35
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10747: [BEAM-6857]
Recategorize UsesTimerMap tests to ValidatesRunner
URL: https://github.com/apache/beam/pull/10747#discussion_r374470396
##
File path: runners/google-cloud-dataflow-java/build.gradle
##
@@ -137,6 +137,7 @@ def commonExcludeCategories = [
'org.apache.beam.sdk.testing.UsesGaugeMetrics',
'org.apache.beam.sdk.testing.UsesSetState',
'org.apache.beam.sdk.testing.UsesMapState',
+ 'org.apache.beam.sdk.testing.UsesTimerMap',
Review comment:
do we need to exclude for Dataflow?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381383)
Time Spent: 29.5h (was: 29h 20m)
> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 29.5h
> Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified
> in the DoFn. The user must provide a separate callback method per timer. For
> example:
>
> {code:java}
> DoFn()
> {
> @TimerId("timer1")
> private final TimerSpec timer1 = TimerSpecs.timer(...);
> @TimerId("timer2")
> private final TimerSpec timer2 = TimerSpecs.timer(...);
> .. set timers in processElement
> @OnTimer("timer1")
> public void onTimer1() { .}
> @OnTimer("timer2")
> public void onTimer2() {}
> }
> {code}
>
> However there are many cases where the user does not know the set of timers
> statically when writing their code. This happens when the timer tag should be
> based on the data. It also happens when writing a DSL on top of Beam, where
> the DSL author has to create DoFns but does not know statically which timers
> their users will want to set (e.g. Scio).
>
> The goal is to support dynamic timers. Something as follows;
>
> {code:java}
> DoFn()
> {
> @TimerId("timer")
> private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
> @ProcessElement process(@TimerId("timer") DynamicTimer timer)
> {
> timer.set("tag1'", ts);
>timer.set("tag2", ts);
> }
> @OnTimer("timer")
> public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-6857) Support dynamic timers
[
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=381384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381384
]
ASF GitHub Bot logged work on BEAM-6857:
Author: ASF GitHub Bot
Created on: 04/Feb/20 04:35
Start Date: 04/Feb/20 04:35
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10747: [BEAM-6857]
Recategorize UsesTimerMap tests to ValidatesRunner
URL: https://github.com/apache/beam/pull/10747#discussion_r374470501
##
File path:
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ParDoTest.java
##
@@ -4292,7 +4292,7 @@ public void onTimer(OutputReceiver r,
@TimerId("timer") Timer timer) {
public static class TimerFamilyTests extends SharedTestBase implements
Serializable {
@Test
-@Category({NeedsRunner.class, UsesTimersInParDo.class, UsesTimerMap.class})
+@Category({ValidatesRunner.class, UsesTimersInParDo.class,
UsesTimerMap.class})
Review comment:
Good catch - I missed this in the code review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381384)
Time Spent: 29h 40m (was: 29.5h)
> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 29h 40m
> Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified
> in the DoFn. The user must provide a separate callback method per timer. For
> example:
>
> {code:java}
> DoFn()
> {
> @TimerId("timer1")
> private final TimerSpec timer1 = TimerSpecs.timer(...);
> @TimerId("timer2")
> private final TimerSpec timer2 = TimerSpecs.timer(...);
> .. set timers in processElement
> @OnTimer("timer1")
> public void onTimer1() { .}
> @OnTimer("timer2")
> public void onTimer2() {}
> }
> {code}
>
> However there are many cases where the user does not know the set of timers
> statically when writing their code. This happens when the timer tag should be
> based on the data. It also happens when writing a DSL on top of Beam, where
> the DSL author has to create DoFns but does not know statically which timers
> their users will want to set (e.g. Scio).
>
> The goal is to support dynamic timers. Something as follows;
>
> {code:java}
> DoFn()
> {
> @TimerId("timer")
> private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
> @ProcessElement process(@TimerId("timer") DynamicTimer timer)
> {
> timer.set("tag1'", ts);
>timer.set("tag2", ts);
> }
> @OnTimer("timer")
> public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-4379) Make ParquetIO Read splittable
[ https://issues.apache.org/jira/browse/BEAM-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029516#comment-17029516 ] Steve Cosenza commented on BEAM-4379: - Hi [~rskraba]. Have you been able to make any progress on this? Thanks > Make ParquetIO Read splittable > -- > > Key: BEAM-4379 > URL: https://issues.apache.org/jira/browse/BEAM-4379 > Project: Beam > Issue Type: Improvement > Components: io-ideas, io-java-parquet >Reporter: Lukasz Gajowy >Assignee: Ryan Skraba >Priority: Major > > As the title stands - currently it is not splittable which is not optimal for > runners that support splitting. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381348 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 03:03 Start Date: 04/Feb/20 03:03 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374450518 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: We can fall back to topic's project if this fails. This also maintains the current functionality. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381348) Time Spent: 40m (was: 0.5h) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?focusedWorklogId=381343=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381343 ] ASF GitHub Bot logged work on BEAM-9230: Author: ASF GitHub Bot Created on: 04/Feb/20 02:50 Start Date: 04/Feb/20 02:50 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10758: [BEAM-9230] Enable CrossLanguageValidateRunner test for Spark runner URL: https://github.com/apache/beam/pull/10758#issuecomment-581717566 Run XVR_Spark PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381343) Time Spent: 50m (was: 40m) > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381345 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:50 Start Date: 04/Feb/20 02:50 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374450518 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -516,7 +524,7 @@ def get_subscription(cls, transform, project, short_topic_name, sub_client = pubsub.SubscriberClient() sub_name = sub_client.subscription_path( -project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) +sub_project, 'beam_%d_%x' % (int(time.time()), random.randrange(1 << 32))) Review comment: We can fall back to topic's project if this fails. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381345) Time Spent: 0.5h (was: 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381344 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:50 Start Date: 04/Feb/20 02:50 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#discussion_r374449591 ## File path: sdks/python/apache_beam/runners/direct/transform_evaluator.py ## @@ -499,13 +499,21 @@ def __init__(self, evaluation_context, applied_ptransform, if self.source.id_label: raise NotImplementedError( 'DirectRunner: id_label is not supported for PubSub reads') + +import google.auth + +_, sub_project = google.auth.default() Review comment: Beam has this pipeline option, project which is generally used to specify project explicitly. Asking user explicitly for resource avoid surprises. Shall we use it. https://github.com/apache/beam/blob/40c2be03303d7c37cabfae4d4da6258a2ce33579/sdks/python/apache_beam/options/pipeline_options.py#L473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381344) Time Spent: 0.5h (was: 20m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381342
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 02:48
Start Date: 04/Feb/20 02:48
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581716993
Unless you have another solution @chadrik, I believe we'll need to shorten
Jenkins job names. I think a limit of 20 characters should work.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381342)
Time Spent: 5h 40m (was: 5.5h)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381336 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:28 Start Date: 04/Feb/20 02:28 Worklog Time Spent: 10m Work Description: davidyan74 commented on issue #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762#issuecomment-581712749 R: @angoenka This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381336) Time Spent: 20m (was: 10m) > Allow usage of public Google PubSub topics in Python DirectRunner > - > > Key: BEAM-3453 > URL: https://issues.apache.org/jira/browse/BEAM-3453 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Affects Versions: 2.2.0 >Reporter: Charles Chen >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently, the Beam Python DirectRunner does not allow the usage of data from > public Google Cloud PubSub topics. We should allow this functionality so > that users can more easily test Beam Python's streaming functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3453) Allow usage of public Google PubSub topics in Python DirectRunner
[ https://issues.apache.org/jira/browse/BEAM-3453?focusedWorklogId=381334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381334 ] ASF GitHub Bot logged work on BEAM-3453: Author: ASF GitHub Bot Created on: 04/Feb/20 02:24 Start Date: 04/Feb/20 02:24 Worklog Time Spent: 10m Work Description: davidyan74 commented on pull request #10762: [BEAM-3453] Use default project when creating pubsub subscription URL: https://github.com/apache/beam/pull/10762 When creating a pubsub subscription, use the default project in GCP if available, rather than the project specified in the pubsub topic. R: @aaltay Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381307
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 02:00
Start Date: 04/Feb/20 02:00
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581706155
Okay so I ssh-ed into the Jenkins node and tried running the command:
```shell
-su:
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/protoc-gen-mypy:
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_:
bad interpreter: No such file or directory
```
Basically, the file has a shebang line that's getting truncated.
I found this:
https://github.com/pypa/virtualenv/issues/596#issuecomment-411485104
FYI the first line is this:
```
#!/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/python2.7
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381307)
Time Spent: 5.5h (was: 5h 20m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
>
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381306
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:59
Start Date: 04/Feb/20 01:59
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581706155
Okay so I ssh-ed into the Jenkins node and tried running the command:
```shell
-su:
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/protoc-gen-mypy:
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_:
bad interpreter: No such file or directory
```
Basically, the file has a shebang line that's getting truncated.
I found this:
https://github.com/pypa/virtualenv/issues/596#issuecomment-411485104
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381306)
Time Spent: 5h 20m (was: 5h 10m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381304=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381304
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:58
Start Date: 04/Feb/20 01:58
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10734: [BEAM-8979]
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581705852
hmmm... so my new code found the executable, but protoc still complained. I
wonder if it's not executable for some reason. I can't think of any
explanation for why that would happen.
```
INFO:gen_protos:Found protoc_gen_mypy at
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/protoc-gen-mypy
/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Write_Performance_Test_Python_Batch_PR/src/build/gradleenv/192237/bin/protoc-gen-mypy:
program not found or is not executable
```
Are the normal python tox tests passing?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381304)
Time Spent: 5h 10m (was: 5h)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[ https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381303 ] ASF GitHub Bot logged work on BEAM-4150: Author: ASF GitHub Bot Created on: 04/Feb/20 01:58 Start Date: 04/Feb/20 01:58 Worklog Time Spent: 10m Work Description: lostluck commented on issue #10755: [BEAM-4150] Remove fallback case for coder not specified within RemoteGrpcPort. URL: https://github.com/apache/beam/pull/10755#issuecomment-581705839 Run Go Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381303) Time Spent: 3h 40m (was: 3.5h) > Standardize use of PCollection coder proto attribute > > > Key: BEAM-4150 > URL: https://issues.apache.org/jira/browse/BEAM-4150 > Project: Beam > Issue Type: Task > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Time Spent: 3h 40m > Remaining Estimate: 0h > > In some places it's expected to be a WindowedCoder, in others the raw > ElementCoder. We should use the same convention (decided in discussion to be > the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the > attached windowing strategy, and the input/output ports should specify the > encoding directly rather than read the adjacent PCollection coder fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[
https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381305
]
ASF GitHub Bot logged work on BEAM-4150:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:58
Start Date: 04/Feb/20 01:58
Worklog Time Spent: 10m
Work Description: lostluck commented on pull request #10755: [BEAM-4150]
Remove fallback case for coder not specified within RemoteGrpcPort.
URL: https://github.com/apache/beam/pull/10755#discussion_r374436704
##
File path: sdks/go/pkg/beam/core/runtime/exec/translate.go
##
@@ -64,6 +64,13 @@ func UnmarshalPlan(desc *fnpb.ProcessBundleDescriptor)
(*Plan, error) {
}
u := {UID: b.idgen.New()}
+u.Coder, err = b.coders.Coder(cid) // Expected to be windowed coder
Review comment:
Please run gofmt on your code. Go uses Tabs for indentation, not spaces.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381305)
Time Spent: 3h 50m (was: 3h 40m)
> Standardize use of PCollection coder proto attribute
>
>
> Key: BEAM-4150
> URL: https://issues.apache.org/jira/browse/BEAM-4150
> Project: Beam
> Issue Type: Task
> Components: beam-model
>Reporter: Robert Bradshaw
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> In some places it's expected to be a WindowedCoder, in others the raw
> ElementCoder. We should use the same convention (decided in discussion to be
> the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the
> attached windowing strategy, and the input/output ports should specify the
> encoding directly rather than read the adjacent PCollection coder fields.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381301=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381301
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:47
Start Date: 04/Feb/20 01:47
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581703499
Run Python 3.7 Load Tests ParDo Dataflow Batch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381301)
Time Spent: 5h (was: 4h 50m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 5h
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381298
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:46
Start Date: 04/Feb/20 01:46
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581703369
The Jenkins logs also show that setupVirtualenv successful installed
`mypy-protobuf-1.12`. I'll try again with the added debug output.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381298)
Time Spent: 4.5h (was: 4h 20m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381299
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:46
Start Date: 04/Feb/20 01:46
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581703420
Run BigQueryIO Write Performance Test Python Batch
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381299)
Time Spent: 4h 40m (was: 4.5h)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381300
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:46
Start Date: 04/Feb/20 01:46
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581703443
retest this please
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381300)
Time Spent: 4h 50m (was: 4h 40m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381295=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381295
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:38
Start Date: 04/Feb/20 01:38
Worklog Time Spent: 10m
Work Description: chadrik commented on issue #10734: [BEAM-8979]
reintroduce mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581701787
> However, on Jenkins nodes the workspace should be cleaned up between runs
and the gradleenv/ directories live in said workspace.
Then what does `:sdks:python:setupVirtualenv UP-TO-DATE` indicate? Your
`./gradlew clean` seems to confirm that this is a caching issue, so what's the
next step to troubleshooting this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381295)
Time Spent: 4h 20m (was: 4h 10m)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including
[jira] [Work logged] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?focusedWorklogId=381294=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381294
]
ASF GitHub Bot logged work on BEAM-8979:
Author: ASF GitHub Bot
Created on: 04/Feb/20 01:29
Start Date: 04/Feb/20 01:29
Worklog Time Spent: 10m
Work Description: udim commented on issue #10734: [BEAM-8979] reintroduce
mypy-protobuf stub generation
URL: https://github.com/apache/beam/pull/10734#issuecomment-581699815
I did a `./gradlew clean` and then `./gradlew :sdks:python:sdist` and it
worked.
`setupVirtualenv` isn't cached in the Gradle sense (the output is not saved
in Gradle's cache), but it is skipped if the output directory is already
present. However, on Jenkins nodes the workspace should be cleaned up between
runs and the gradleenv/ directories live in said workspace.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381294)
Time Spent: 4h 10m (was: 4h)
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=381292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381292 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 04/Feb/20 01:21 Start Date: 04/Feb/20 01:21 Worklog Time Spent: 10m Work Description: pabloem commented on issue #10368: [BEAM-8335] Modify PipelineInstrument to add TestStream for unbounded PCollections URL: https://github.com/apache/beam/pull/10368#issuecomment-581697868 it looks like the tests are failing @rohdesamuel https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1432/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381292) Time Spent: 54h 40m (was: 54.5h) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 54h 40m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=381290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381290 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 04/Feb/20 00:57 Start Date: 04/Feb/20 00:57 Worklog Time Spent: 10m Work Description: pabloem commented on issue #10368: [BEAM-8335] Modify PipelineInstrument to add TestStream for unbounded PCollections URL: https://github.com/apache/beam/pull/10368#issuecomment-581692305 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381290) Time Spent: 54.5h (was: 54h 20m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 54.5h > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7746) Add type hints to python code
[ https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=381287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381287 ] ASF GitHub Bot logged work on BEAM-7746: Author: ASF GitHub Bot Created on: 04/Feb/20 00:51 Start Date: 04/Feb/20 00:51 Worklog Time Spent: 10m Work Description: udim commented on pull request #10730: [BEAM-7746] Remove use of @functools.total_ordering URL: https://github.com/apache/beam/pull/10730 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381287) Time Spent: 60h 10m (was: 60h) > Add type hints to python code > - > > Key: BEAM-7746 > URL: https://issues.apache.org/jira/browse/BEAM-7746 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Chad Dombrova >Assignee: Chad Dombrova >Priority: Major > Time Spent: 60h 10m > Remaining Estimate: 0h > > As a developer of the beam source code, I would like the code to use pep484 > type hints so that I can clearly see what types are required, get completion > in my IDE, and enforce code correctness via a static analyzer like mypy. > This may be considered a precursor to BEAM-7060 > Work has been started here: [https://github.com/apache/beam/pull/9056] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[ https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381286 ] ASF GitHub Bot logged work on BEAM-4150: Author: ASF GitHub Bot Created on: 04/Feb/20 00:32 Start Date: 04/Feb/20 00:32 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10755: [BEAM-4150] Remove fallback case for coder not specified within RemoteGrpcPort. URL: https://github.com/apache/beam/pull/10755#issuecomment-581686417 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381286) Time Spent: 3.5h (was: 3h 20m) > Standardize use of PCollection coder proto attribute > > > Key: BEAM-4150 > URL: https://issues.apache.org/jira/browse/BEAM-4150 > Project: Beam > Issue Type: Task > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Time Spent: 3.5h > Remaining Estimate: 0h > > In some places it's expected to be a WindowedCoder, in others the raw > ElementCoder. We should use the same convention (decided in discussion to be > the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the > attached windowing strategy, and the input/output ports should specify the > encoding directly rather than read the adjacent PCollection coder fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9231) Add Experimental portability kind and tag related classes
[ https://issues.apache.org/jira/browse/BEAM-9231?focusedWorklogId=381283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381283 ] ASF GitHub Bot logged work on BEAM-9231: Author: ASF GitHub Bot Created on: 04/Feb/20 00:31 Start Date: 04/Feb/20 00:31 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10739: [BEAM-9231] Add Experimental portability kind and tag related classes URL: https://github.com/apache/beam/pull/10739#issuecomment-581685996 I'm not sure if we need to mark runner libraries or anything outside of beam-sdks-java-core / beam-sdks-java-extensions-* / beam-sdks-java-io-* as `@Experimental`. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381283) Time Spent: 1h 20m (was: 1h 10m) > Add Experimental portability kind and tag related classes > - > > Key: BEAM-9231 > URL: https://issues.apache.org/jira/browse/BEAM-9231 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Ismaël Mejía >Assignee: Ismaël Mejía >Priority: Minor > Labels: portability > Time Spent: 1h 20m > Remaining Estimate: 0h > > For some extra context I was studying the evolution of our APIs between > versions in particular for beam-sdks-java-core and noticed that some parts > were not well classified as Experimental in particular classes (and > transforms) for portability and SplittableDoFn. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8095) pytest Py3.7 crashes on test_remote_runner_display_data
[ https://issues.apache.org/jira/browse/BEAM-8095?focusedWorklogId=381284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381284 ] ASF GitHub Bot logged work on BEAM-8095: Author: ASF GitHub Bot Created on: 04/Feb/20 00:31 Start Date: 04/Feb/20 00:31 Worklog Time Spent: 10m Work Description: udim commented on pull request #10759: [BEAM-8095] Remove no_xdist for test URL: https://github.com/apache/beam/pull/10759 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-8095) pytest Py3.7 crashes on test_remote_runner_display_data
[
https://issues.apache.org/jira/browse/BEAM-8095?focusedWorklogId=381285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381285
]
ASF GitHub Bot logged work on BEAM-8095:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:31
Start Date: 04/Feb/20 00:31
Worklog Time Spent: 10m
Work Description: udim commented on issue #10759: [BEAM-8095] Remove
no_xdist for test
URL: https://github.com/apache/beam/pull/10759#issuecomment-581686183
R: @tvalentyn
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381285)
Time Spent: 20m (was: 10m)
> pytest Py3.7 crashes on test_remote_runner_display_data
> ---
>
> Key: BEAM-8095
> URL: https://issues.apache.org/jira/browse/BEAM-8095
> Project: Beam
> Issue Type: Improvement
> Components: test-failures, testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Adding certain flags such as "-n 1" or "--debug" causes Python to abort.
> The --debug flag logs some information in a pytestdebug.log, but I'm not
> familiar with it and what it means if anything.
> {code}
> $ pytest
> apache_beam/runners/dataflow/dataflow_runner_test.py::DataflowRunnerTest::test_remote_runner_display_data
>
> ===
> test session starts
>
> platform linux -- Python 3.7.3rc1, pytest-5.1.1, py-1.8.0, pluggy-0.12.0
> rootdir: /usr/local/google/home/ehudm/src/beam/sdks/python, inifile:
> pytest.ini
> plugins: forked-1.0.2, xdist-1.29.0
> collected 1 item
>
>
>
>
> apache_beam/runners/dataflow/dataflow_runner_test.py .
>
>
>
> [100%]
>
> 1 passed in 0.11s
> =
> $ pytest
> apache_beam/runners/dataflow/dataflow_runner_test.py::DataflowRunnerTest::test_remote_runner_display_data
> --debug
> writing pytestdebug information to
> /usr/local/google/home/ehudm/src/beam/sdks/python/pytestdebug.log
> ===
> test session starts
>
> platform linux -- Python 3.7.3rc1, pytest-5.1.1, py-1.8.0, pluggy-0.12.0 --
> /usr/local/google/home/ehudm/virtualenvs/beam-py37/bin/python3.7
> using: pytest-5.1.1 pylib-1.8.0
> setuptools registered plugins:
> pytest-forked-1.0.2 at
> /usr/local/google/home/ehudm/virtualenvs/beam-py37/lib/python3.7/site-packages/pytest_forked/__init__.py
> pytest-xdist-1.29.0 at
> /usr/local/google/home/ehudm/virtualenvs/beam-py37/lib/python3.7/site-packages/xdist/plugin.py
> pytest-xdist-1.29.0 at
> /usr/local/google/home/ehudm/virtualenvs/beam-py37/lib/python3.7/site-packages/xdist/looponfail.py
> rootdir: /usr/local/google/home/ehudm/src/beam/sdks/python, inifile:
> pytest.ini
> plugins: forked-1.0.2, xdist-1.29.0
> collected 1 item
>
>
[jira] [Work logged] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?focusedWorklogId=381281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381281 ] ASF GitHub Bot logged work on BEAM-9230: Author: ASF GitHub Bot Created on: 04/Feb/20 00:28 Start Date: 04/Feb/20 00:28 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #10758: [BEAM-9230] Enable CrossLanguageValidateRunner test for Spark runner URL: https://github.com/apache/beam/pull/10758#issuecomment-581685378 Run Seed Job This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381281) Time Spent: 40m (was: 0.5h) > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381274
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374412359
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.util.VarInt;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+public final class AwsCoders {
Review comment:
Please add class comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381274)
Time Spent: 5h (was: 4h 50m)
> PublishResult returned by SnsIO is missing sdkResponseMetadata and
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field
> so the PublishResult returned by Beam returns null for
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible
> to check the HTTP status for errors, which is necessary since this is not
> handled in SnsIO.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381272
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374413206
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.util.VarInt;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ public static Coder fullSdkHttpMetadata() {
+return FullSdkHttpMetadataCoder.of();
+ }
+
+ public static Coder minimalSdkHttpMetadata() {
+return MinimalSdkHttpMetadataCoder.of();
+ }
+
+ private static class ResponseMetadataCoder extends Coder
+ implements Serializable {
+
+private static final ResponseMetadataCoder INSTANCE = new
ResponseMetadataCoder();
+
+private ResponseMetadataCoder() {}
+
+public static ResponseMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(ResponseMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
Review comment:
The encoding of this changed where before we assumed this would always exist
but now we handle the null case but we don't actually return the same object
that was provided when decoding. Would you rather handle the null case by
wrapping the response metadata coder returned in `responseMetadata` with the
[NullableCoder](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/NullableCoder.java)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381272)
Time Spent: 4h 50m (was: 4h 40m)
> PublishResult returned by SnsIO is missing sdkResponseMetadata and
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field
> so the PublishResult returned by Beam returns null for
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible
> to check the HTTP status for errors, which is necessary since
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381275
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374414686
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoders.java
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.SdkHttpMetadata;
+import com.amazonaws.services.sns.model.PublishResult;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.aws.coders.AwsCoders;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+public final class PublishResultCoders {
+
+ private PublishResultCoders() {}
+
+ public static Coder defaultPublishResult() {
+return DefaultPublishResultCoder.of();
+ }
+
+ public static Coder fullPublishResult() {
+return FullPublishResultCoder.of();
+ }
+
+ public static Coder minimalPublishResult() {
+return MinimalPublishResultCoder.of();
+ }
+
+ private static class DefaultPublishResultCoder extends Coder
+ implements Serializable {
+
+private static final DefaultPublishResultCoder INSTANCE = new
DefaultPublishResultCoder();
+
+private DefaultPublishResultCoder() {}
+
+static DefaultPublishResultCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(PublishResult value, OutputStream outStream)
+throws CoderException, IOException {
+ StringUtf8Coder.of().encode(value.getMessageId(), outStream);
+}
+
+@Override
+public PublishResult decode(InputStream inStream) throws CoderException,
IOException {
+ final String messageId = StringUtf8Coder.of().decode(inStream);
+ return new PublishResult().withMessageId(messageId);
+}
+
+@Override
+public List> getCoderArguments() {
+ return ImmutableList.of();
+}
+
+@Override
+public void verifyDeterministic() throws NonDeterministicException {
+ StringUtf8Coder.of().verifyDeterministic();
+}
+ }
+
+ private abstract static class AbstractPublishResultCoder extends
Coder
+ implements Serializable {
+
+private static final Coder RESPONSE_METADATA_CODER =
+AwsCoders.responseMetadata();
+
+protected abstract Coder getHttpMetadataCoder();
Review comment:
nit: stylewise I think its simpler to understand if the abstract class has a
constructor that takes the sdkhttpmetadata coder and you pass in the coder
instance in the concrete subclass implementations instead of asking authors to
override `getHttpMetadataCoder`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381275)
Time Spent: 5h 10m (was: 5h)
> PublishResult returned by SnsIO is missing sdkResponseMetadata and
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Affects Versions: 2.13.0,
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381273
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374413942
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sns/PublishResultCoders.java
##
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sns;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.SdkHttpMetadata;
+import com.amazonaws.services.sns.model.PublishResult;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.io.aws.coders.AwsCoders;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+
+public final class PublishResultCoders {
Review comment:
class comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381273)
Time Spent: 5h (was: 4h 50m)
> PublishResult returned by SnsIO is missing sdkResponseMetadata and
> sdkHttpMetadata
> --
>
> Key: BEAM-8374
> URL: https://issues.apache.org/jira/browse/BEAM-8374
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws
>Affects Versions: 2.13.0, 2.14.0, 2.15.0
>Reporter: Jonothan Farr
>Assignee: Jonothan Farr
>Priority: Minor
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Currently the PublishResultCoder in SnsIO only serializes the messageId field
> so the PublishResult returned by Beam returns null for
> getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible
> to check the HTTP status for errors, which is necessary since this is not
> handled in SnsIO.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381276
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374413483
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.util.VarInt;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ public static Coder fullSdkHttpMetadata() {
+return FullSdkHttpMetadataCoder.of();
+ }
+
+ public static Coder minimalSdkHttpMetadata() {
+return MinimalSdkHttpMetadataCoder.of();
+ }
+
+ private static class ResponseMetadataCoder extends Coder
+ implements Serializable {
+
+private static final ResponseMetadataCoder INSTANCE = new
ResponseMetadataCoder();
+
+private ResponseMetadataCoder() {}
+
+public static ResponseMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(ResponseMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
+ .encode(
+ ImmutableMap.of(
+ ResponseMetadata.AWS_REQUEST_ID, value != null ?
value.getRequestId() : ""),
+ outStream);
+}
+
+@Override
+public ResponseMetadata decode(InputStream inStream) throws
CoderException, IOException {
+ return new ResponseMetadata(
+ MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()).decode(inStream));
+}
+
+@Override
+public List> getCoderArguments() {
+ return ImmutableList.of();
+}
+
+@Override
+public void verifyDeterministic() throws NonDeterministicException {
+ MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()).verifyDeterministic();
+}
+ }
+
+ private static class FullSdkHttpMetadataCoder extends Coder
+ implements Serializable {
+
+private static final FullSdkHttpMetadataCoder INSTANCE = new
FullSdkHttpMetadataCoder();
+
+private FullSdkHttpMetadataCoder() {}
+
+public static FullSdkHttpMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(SdkHttpMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ VarInt.encode(value != null ? value.getHttpStatusCode() : 0, outStream);
Review comment:
Same suggestion for wrapping NullableCoder within `fullSdkHttpMetadata` and
not handling the null case here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[
https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381277=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381277
]
ASF GitHub Bot logged work on BEAM-8374:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:22
Start Date: 04/Feb/20 00:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #9758: [BEAM-8374]
Enable returning missing PublishResult fields in SnsIO.Write
URL: https://github.com/apache/beam/pull/9758#discussion_r374413590
##
File path:
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/coders/AwsCoders.java
##
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.coders;
+
+import com.amazonaws.ResponseMetadata;
+import com.amazonaws.http.HttpResponse;
+import com.amazonaws.http.SdkHttpMetadata;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.util.List;
+import java.util.Map;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderException;
+import org.apache.beam.sdk.coders.MapCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.coders.VarIntCoder;
+import org.apache.beam.sdk.util.VarInt;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
+
+public final class AwsCoders {
+
+ private AwsCoders() {}
+
+ public static Coder responseMetadata() {
+return ResponseMetadataCoder.of();
+ }
+
+ public static Coder fullSdkHttpMetadata() {
+return FullSdkHttpMetadataCoder.of();
+ }
+
+ public static Coder minimalSdkHttpMetadata() {
+return MinimalSdkHttpMetadataCoder.of();
+ }
+
+ private static class ResponseMetadataCoder extends Coder
+ implements Serializable {
+
+private static final ResponseMetadataCoder INSTANCE = new
ResponseMetadataCoder();
+
+private ResponseMetadataCoder() {}
+
+public static ResponseMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(ResponseMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
+ .encode(
+ ImmutableMap.of(
+ ResponseMetadata.AWS_REQUEST_ID, value != null ?
value.getRequestId() : ""),
+ outStream);
+}
+
+@Override
+public ResponseMetadata decode(InputStream inStream) throws
CoderException, IOException {
+ return new ResponseMetadata(
+ MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()).decode(inStream));
+}
+
+@Override
+public List> getCoderArguments() {
+ return ImmutableList.of();
+}
+
+@Override
+public void verifyDeterministic() throws NonDeterministicException {
+ MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()).verifyDeterministic();
+}
+ }
+
+ private static class FullSdkHttpMetadataCoder extends Coder
+ implements Serializable {
+
+private static final FullSdkHttpMetadataCoder INSTANCE = new
FullSdkHttpMetadataCoder();
+
+private FullSdkHttpMetadataCoder() {}
+
+public static FullSdkHttpMetadataCoder of() {
+ return INSTANCE;
+}
+
+@Override
+public void encode(SdkHttpMetadata value, OutputStream outStream)
+throws CoderException, IOException {
+ VarInt.encode(value != null ? value.getHttpStatusCode() : 0, outStream);
+ MapCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of())
+ .encode(value != null ? value.getHttpHeaders() : ImmutableMap.of(),
outStream);
+}
+
+@Override
+public SdkHttpMetadata decode(InputStream inStream) throws CoderException,
IOException {
+ final int httpStatusCode = VarInt.decodeInt(inStream);
+ final Map httpHeaders =
+ MapCoder.of(StringUtf8Coder.of(),
StringUtf8Coder.of()).decode(inStream);
+ HttpResponse httpResponse = new
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[ https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381271 ] ASF GitHub Bot logged work on BEAM-4150: Author: ASF GitHub Bot Created on: 04/Feb/20 00:20 Start Date: 04/Feb/20 00:20 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10755: [BEAM-4150] Remove fallback case for coder not specified within RemoteGrpcPort. URL: https://github.com/apache/beam/pull/10755#issuecomment-581683444 R: @lostluck @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381271) Time Spent: 3h 20m (was: 3h 10m) > Standardize use of PCollection coder proto attribute > > > Key: BEAM-4150 > URL: https://issues.apache.org/jira/browse/BEAM-4150 > Project: Beam > Issue Type: Task > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Time Spent: 3h 20m > Remaining Estimate: 0h > > In some places it's expected to be a WindowedCoder, in others the raw > ElementCoder. We should use the same convention (decided in discussion to be > the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the > attached windowing strategy, and the input/output ports should specify the > encoding directly rather than read the adjacent PCollection coder fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-4150) Standardize use of PCollection coder proto attribute
[ https://issues.apache.org/jira/browse/BEAM-4150?focusedWorklogId=381267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381267 ] ASF GitHub Bot logged work on BEAM-4150: Author: ASF GitHub Bot Created on: 04/Feb/20 00:20 Start Date: 04/Feb/20 00:20 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10755: [BEAM-4150] Remove fallback case for coder not specified within RemoteGrpcPort. URL: https://github.com/apache/beam/pull/10755#issuecomment-581683353 Tested JRH change internally within Google as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381267) Time Spent: 3h 10m (was: 3h) > Standardize use of PCollection coder proto attribute > > > Key: BEAM-4150 > URL: https://issues.apache.org/jira/browse/BEAM-4150 > Project: Beam > Issue Type: Task > Components: beam-model >Reporter: Robert Bradshaw >Priority: Major > Fix For: Not applicable > > Time Spent: 3h 10m > Remaining Estimate: 0h > > In some places it's expected to be a WindowedCoder, in others the raw > ElementCoder. We should use the same convention (decided in discussion to be > the raw ElementCoder) everywhere. The WindowCoder can be pulled out of the > attached windowing strategy, and the input/output ports should specify the > encoding directly rather than read the adjacent PCollection coder fields. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381266 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 04/Feb/20 00:14 Start Date: 04/Feb/20 00:14 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756#issuecomment-581681835 @lukecwik and @ibzib, Thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381266) Time Spent: 1h 50m (was: 1h 40m) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8335) Add streaming support to Interactive Beam
[ https://issues.apache.org/jira/browse/BEAM-8335?focusedWorklogId=381265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381265 ] ASF GitHub Bot logged work on BEAM-8335: Author: ASF GitHub Bot Created on: 04/Feb/20 00:12 Start Date: 04/Feb/20 00:12 Worklog Time Spent: 10m Work Description: pabloem commented on issue #10368: [BEAM-8335] Modify PipelineInstrument to add TestStream for unbounded PCollections URL: https://github.com/apache/beam/pull/10368#issuecomment-581681213 Run Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381265) Time Spent: 54h 20m (was: 54h 10m) > Add streaming support to Interactive Beam > - > > Key: BEAM-8335 > URL: https://issues.apache.org/jira/browse/BEAM-8335 > Project: Beam > Issue Type: Improvement > Components: runner-py-interactive >Reporter: Sam Rohde >Assignee: Sam Rohde >Priority: Major > Time Spent: 54h 20m > Remaining Estimate: 0h > > This issue tracks the work items to introduce streaming support to the > Interactive Beam experience. This will allow users to: > * Write and run a streaming job in IPython > * Automatically cache records from unbounded sources > * Add a replay experience that replays all cached records to simulate the > original pipeline execution > * Add controls to play/pause/stop/step individual elements from the cached > records > * Add ability to inspect/visualize unbounded PCollections -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381264
]
ASF GitHub Bot logged work on BEAM-9178:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:10
Start Date: 04/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #10634: [BEAM-9178]
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374412025
##
File path:
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
##
@@ -770,88 +765,34 @@ private RexNode convertResolvedFunctionCall(
throw new UnsupportedOperationException(
"Only support TUMBLE, HOP AND SESSION functions right now.");
}
-} else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
- op =
-
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- functionCall.getFunction().getName());
-
+} else if ("ZetaSQL".equals(funGroup)) {
if (op == null) {
-throw new UnsupportedOperationException(
-"Does not support ZetaSQL function: " +
functionCall.getFunction().getName());
+TypeKind returnType =
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_NAME_TO_RETURN_TYPE.get(funName);
+if (returnType != null) {
+ op =
+ SqlOperators.createSimpleSqlFunction(
+ funName, ZetaSqlUtils.zetaSqlTypeToCalciteType(returnType));
+} else {
+ throw new UnsupportedOperationException("Does not support ZetaSQL
function: " + funName);
+}
}
- // There are different processes to handle argument conversion because
INTERVAL is not a
- // type in ZetaSQL.
- if
(FUNCTION_FAMILY_DATE_ADD.contains(functionCall.getFunction().getName())) {
-return convertTimestampAddFunction(functionCall, columnList,
fieldList);
- } else {
-for (ResolvedExpr expr : functionCall.getArgumentList()) {
- operands.add(convertRexNodeFromResolvedExpr(expr, columnList,
fieldList));
-}
+ for (ResolvedExpr expr : functionCall.getArgumentList()) {
+operands.add(convertRexNodeFromResolvedExpr(expr, columnList,
fieldList));
}
} else {
- throw new UnsupportedOperationException(
- "Does not support function group: " +
functionCall.getFunction().getGroup());
+ throw new UnsupportedOperationException("Does not support function
group: " + funGroup);
}
SqlOperatorRewriter rewriter =
-
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(
-functionCall.getFunction().getName());
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR_REWRITER.get(funName);
if (rewriter != null) {
- ret = rewriter.apply(rexBuilder(), operands);
+ return rewriter.apply(rexBuilder(), operands);
} else {
- ret = rexBuilder().makeCall(op, operands);
-}
-return ret;
- }
-
- private RexNode convertTimestampAddFunction(
- ResolvedFunctionCall functionCall,
- List columnList,
- List fieldList) {
-
-TimeUnit unit =
-TIME_UNIT_CASTING_MAP.get(
-((ResolvedLiteral)
functionCall.getArgumentList().get(2)).getValue().getEnumValue());
-
-if ((unit == TimeUnit.MICROSECOND) || (unit == TimeUnit.NANOSECOND)) {
Review comment:
This 'works' now but it is still not supported in our type system, so it
might work inconsistently. (Microseconds will be dropped once we are out of
ZetaSQLCalc.) Should we be making sure ZetaSQLCalc has the correct hooks to
error if the user interacts with microsecond precision values?
cc: @kennknowles
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381264)
Time Spent: 1h 10m (was: 1h)
> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL).
> See the full list of functions below:
>
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381263 ] ASF GitHub Bot logged work on BEAM-9178: Author: ASF GitHub Bot Created on: 04/Feb/20 00:10 Start Date: 04/Feb/20 00:10 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10634: [BEAM-9178] Support all ZetaSQL TIMESTAMP functions URL: https://github.com/apache/beam/pull/10634#discussion_r374409030 ## File path: sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java ## @@ -52,7 +51,6 @@ import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedOrderByScan; import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedParameter; import com.google.zetasql.resolvedast.ResolvedNodes.ResolvedProjectScan; -import io.grpc.Status; Review comment: :+1: Thanks. This actually adds quite a bit of complexity elsewhere. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381263) Time Spent: 1h (was: 50m) > Support ZetaSQL TIMESTAMP functions in BeamSQL > -- > > Key: BEAM-9178 > URL: https://issues.apache.org/jira/browse/BEAM-9178 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). > See the full list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381261
]
ASF GitHub Bot logged work on BEAM-9178:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:10
Start Date: 04/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #10634: [BEAM-9178]
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374408543
##
File path:
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQuerySqlDialect.java
##
@@ -302,6 +316,38 @@ private void unparseTrim(SqlWriter writer, SqlCall call,
int leftPrec, int right
writer.endFunCall(trimFrame);
}
+ /**
+ * For usage of INTERVAL, see https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#timestamp_add;>
+ * BQ TIMESTAMP_ADD function for example.
+ */
+ private void unparseFunctionsUsingInterval(
+ SqlWriter writer, SqlCall call, int leftPrec, int rightPrec) {
+// e.g. TIMESTAMP_ADD syntax:
+// TIMESTAMP_ADD(timestamp_expression, INTERVAL int64_expression date_part)
+int operandCount = call.operandCount();
+if (operandCount == 2) {
+ // operand0: timestamp_expression
+ // operand1: SqlIntervalLiteral (INTERVAL int64_expression date_part)
+ super.unparseCall(writer, call, leftPrec, rightPrec);
+} else if (operandCount == 3) {
+ // operand0: timestamp_expression
+ // operand1: int64_expression
+ // operand2: date_part
+ final SqlWriter.Frame frame =
writer.startFunCall(call.getOperator().getName());
+ call.operand(0).unparse(writer, leftPrec, rightPrec);
+ writer.literal(",");
+ writer.literal("INTERVAL");
+ call.operand(1).unparse(writer, leftPrec, rightPrec);
+ call.operand(2).unparse(writer, leftPrec, rightPrec);
+ writer.endFunCall(frame);
+} else {
+ throw new IllegalArgumentException(
+ String.format(
+ "Unable to parse %s with %d operands.",
call.getOperator().getName(), operandCount));
Review comment:
s/parse/unparse/
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381261)
Time Spent: 50m (was: 40m)
> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL).
> See the full list of functions below:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[
https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381262
]
ASF GitHub Bot logged work on BEAM-9178:
Author: ASF GitHub Bot
Created on: 04/Feb/20 00:10
Start Date: 04/Feb/20 00:10
Worklog Time Spent: 10m
Work Description: apilloud commented on pull request #10634: [BEAM-9178]
Support all ZetaSQL TIMESTAMP functions
URL: https://github.com/apache/beam/pull/10634#discussion_r374410033
##
File path:
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
##
@@ -770,88 +765,34 @@ private RexNode convertResolvedFunctionCall(
throw new UnsupportedOperationException(
"Only support TUMBLE, HOP AND SESSION functions right now.");
}
-} else if (functionCall.getFunction().getGroup().equals("ZetaSQL")) {
- op =
-
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_TO_CALCITE_SQL_OPERATOR.get(
- functionCall.getFunction().getName());
-
+} else if ("ZetaSQL".equals(funGroup)) {
if (op == null) {
-throw new UnsupportedOperationException(
-"Does not support ZetaSQL function: " +
functionCall.getFunction().getName());
+TypeKind returnType =
+
SqlStdOperatorMappingTable.ZETASQL_FUNCTION_NAME_TO_RETURN_TYPE.get(funName);
Review comment:
Manually maintaining this list seems like no fun. What about using
`functionCall.getSignature().getResultType().getType()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381262)
> Support ZetaSQL TIMESTAMP functions in BeamSQL
> --
>
> Key: BEAM-9178
> URL: https://issues.apache.org/jira/browse/BEAM-9178
> Project: Beam
> Issue Type: New Feature
> Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL).
> See the full list of functions below:
> [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8374) PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[ https://issues.apache.org/jira/browse/BEAM-8374?focusedWorklogId=381260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381260 ] ASF GitHub Bot logged work on BEAM-8374: Author: ASF GitHub Bot Created on: 04/Feb/20 00:08 Start Date: 04/Feb/20 00:08 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #9758: [BEAM-8374] Enable returning missing PublishResult fields in SnsIO.Write URL: https://github.com/apache/beam/pull/9758#issuecomment-581680218 Sorry for the delay, taking a look now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381260) Time Spent: 4h 40m (was: 4.5h) > PublishResult returned by SnsIO is missing sdkResponseMetadata and > sdkHttpMetadata > -- > > Key: BEAM-8374 > URL: https://issues.apache.org/jira/browse/BEAM-8374 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.13.0, 2.14.0, 2.15.0 >Reporter: Jonothan Farr >Assignee: Jonothan Farr >Priority: Minor > Time Spent: 4h 40m > Remaining Estimate: 0h > > Currently the PublishResultCoder in SnsIO only serializes the messageId field > so the PublishResult returned by Beam returns null for > getSdkResponseMetadata() and getSdkHttpMetadata(). This makes it impossible > to check the HTTP status for errors, which is necessary since this is not > handled in SnsIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-3221) Model pipeline representation improvements
[ https://issues.apache.org/jira/browse/BEAM-3221?focusedWorklogId=381259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381259 ] ASF GitHub Bot logged work on BEAM-3221: Author: ASF GitHub Bot Created on: 03/Feb/20 23:56 Start Date: 03/Feb/20 23:56 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10754: [BEAM-3221] Expand/clarify timestamp comments within beam_runner_api.proto URL: https://github.com/apache/beam/pull/10754#issuecomment-581676818 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381259) Time Spent: 50m (was: 40m) > Model pipeline representation improvements > -- > > Key: BEAM-3221 > URL: https://issues.apache.org/jira/browse/BEAM-3221 > Project: Beam > Issue Type: Improvement > Components: beam-model >Reporter: Henning Rohde >Priority: Major > Labels: portability > Time Spent: 50m > Remaining Estimate: 0h > > Collections of various (breaking) tweaks to the Runner API, notably the > pipeline representation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8756) Beam Dependency Update Request: com.google.cloud:google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-8756?focusedWorklogId=381258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381258 ] ASF GitHub Bot logged work on BEAM-8756: Author: ASF GitHub Bot Created on: 03/Feb/20 23:54 Start Date: 03/Feb/20 23:54 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10674: [BEAM-8756] Google cloud client libraries to use 2019 versions URL: https://github.com/apache/beam/pull/10674#issuecomment-581676343 Thanks for the fix, sorry for the miss during review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381258) Time Spent: 3h 50m (was: 3h 40m) > Beam Dependency Update Request: com.google.cloud:google-cloud-core > -- > > Key: BEAM-8756 > URL: https://issues.apache.org/jira/browse/BEAM-8756 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > - 2019-11-19 21:05:22.063293 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:10:54.007619 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:01.481594 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:02.830621 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:05:33.624755 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:08.158708 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:46.788844 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:08:23.305323 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381256 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 23:43 Start Date: 03/Feb/20 23:43 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756#issuecomment-581673540 > We'll see during the next run for the nightly snapshot. This is definitely an improvement. Agreed, thanks for the prompt fix Tomo. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381256) Time Spent: 1h 40m (was: 1.5h) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381255 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 23:41 Start Date: 03/Feb/20 23:41 Worklog Time Spent: 10m Work Description: lukecwik commented on issue #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756#issuecomment-581673024 We'll see during the next run for the nightly snapshot. This is definitely an improvement. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381255) Time Spent: 1.5h (was: 1h 20m) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381254 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 23:39 Start Date: 03/Feb/20 23:39 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381254) Time Spent: 1h 20m (was: 1h 10m) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[
https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381253
]
ASF GitHub Bot logged work on BEAM-9246:
Author: ASF GitHub Bot
Created on: 03/Feb/20 23:39
Start Date: 03/Feb/20 23:39
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10756: [BEAM-9246]
Introducing google-http-client.version property in Java archetype
URL: https://github.com/apache/beam/pull/10756#discussion_r374403127
##
File path: sdks/java/maven-archetypes/examples/build.gradle
##
@@ -28,6 +28,7 @@ processResources {
'project.version': version,
'bigquery.version':
dependencies.create(project.library.java.google_api_services_bigquery).getVersion(),
'google-clients.version':
dependencies.create(project.library.java.google_api_client).getVersion(),
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381253)
Time Spent: 1h 10m (was: 1h)
> beam_PostRelease_NightlySnapshot failing
>
>
> Key: BEAM-9246
> URL: https://issues.apache.org/jira/browse/BEAM-9246
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Tomo Suzuki
>Priority: Major
> Labels: currently-failing
> Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Multiple tests are failing with error:
> 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could
> not find artifact com.google.http-client:google-http-client:jar:1.30.3 in
> test.release (https://repository.apache.org/content/repositories/snapshots)
> -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[
https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381246
]
ASF GitHub Bot logged work on BEAM-9246:
Author: ASF GitHub Bot
Created on: 03/Feb/20 23:30
Start Date: 03/Feb/20 23:30
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10756: [BEAM-9246]
Introducing google-http-client.version property in Java archetype
URL: https://github.com/apache/beam/pull/10756#discussion_r374400355
##
File path: sdks/java/maven-archetypes/examples/build.gradle
##
@@ -28,6 +28,7 @@ processResources {
'project.version': version,
'bigquery.version':
dependencies.create(project.library.java.google_api_services_bigquery).getVersion(),
'google-clients.version':
dependencies.create(project.library.java.google_api_client).getVersion(),
Review comment:
That's a nice idea.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381246)
Time Spent: 1h (was: 50m)
> beam_PostRelease_NightlySnapshot failing
>
>
> Key: BEAM-9246
> URL: https://issues.apache.org/jira/browse/BEAM-9246
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Tomo Suzuki
>Priority: Major
> Labels: currently-failing
> Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Multiple tests are failing with error:
> 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could
> not find artifact com.google.http-client:google-http-client:jar:1.30.3 in
> test.release (https://repository.apache.org/content/repositories/snapshots)
> -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-8813) Java updateOfflineRepository fails
[ https://issues.apache.org/jira/browse/BEAM-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver closed BEAM-8813. - Fix Version/s: Not applicable Resolution: Workaround > Java updateOfflineRepository fails > -- > > Key: BEAM-8813 > URL: https://issues.apache.org/jira/browse/BEAM-8813 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Priority: Major > Fix For: Not applicable > > > > Task :sdks:java:core:updateOfflineRepository FAILED > FAILURE: Build failed with an exception. > * What went wrong: > Execution failed for task ':sdks:java:core:updateOfflineRepository'. > > Could not find avro-tests.jar (org.apache.avro:avro:1.8.2). > Searched in the following locations: > > file:/[redacted]/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2-tests.jar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8813) Java updateOfflineRepository fails
[ https://issues.apache.org/jira/browse/BEAM-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029364#comment-17029364 ] Kyle Weaver commented on BEAM-8813: --- I fixed this by deleting my .m2 directory and retrying. > Java updateOfflineRepository fails > -- > > Key: BEAM-8813 > URL: https://issues.apache.org/jira/browse/BEAM-8813 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kyle Weaver >Priority: Major > > > Task :sdks:java:core:updateOfflineRepository FAILED > FAILURE: Build failed with an exception. > * What went wrong: > Execution failed for task ':sdks:java:core:updateOfflineRepository'. > > Could not find avro-tests.jar (org.apache.avro:avro:1.8.2). > Searched in the following locations: > > file:/[redacted]/.m2/repository/org/apache/avro/avro/1.8.2/avro-1.8.2-tests.jar -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381245 ] ASF GitHub Bot logged work on BEAM-9178: Author: ASF GitHub Bot Created on: 03/Feb/20 23:24 Start Date: 03/Feb/20 23:24 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10634: [BEAM-9178] Support all ZetaSQL TIMESTAMP functions URL: https://github.com/apache/beam/pull/10634#issuecomment-581668221 I'm guessing you didn't mean to create a merge commit here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381245) Time Spent: 40m (was: 0.5h) > Support ZetaSQL TIMESTAMP functions in BeamSQL > -- > > Key: BEAM-9178 > URL: https://issues.apache.org/jira/browse/BEAM-9178 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). > See the full list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?focusedWorklogId=381243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381243 ] ASF GitHub Bot logged work on BEAM-9230: Author: ASF GitHub Bot Created on: 03/Feb/20 23:22 Start Date: 03/Feb/20 23:22 Worklog Time Spent: 10m Work Description: ihji commented on issue #10758: [BEAM-9230] Enable CrossLanguageValidateRunner test for Spark runner URL: https://github.com/apache/beam/pull/10758#issuecomment-581667426 @chamikaramj need to run `Run Seed Job` and `Run XVR_Spark PostCommit` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381243) Time Spent: 0.5h (was: 20m) > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?focusedWorklogId=381240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381240 ] ASF GitHub Bot logged work on BEAM-9230: Author: ASF GitHub Bot Created on: 03/Feb/20 23:16 Start Date: 03/Feb/20 23:16 Worklog Time Spent: 10m Work Description: ihji commented on issue #10758: [BEAM-9230] Enable CrossLanguageValidateRunner test for Spark runner URL: https://github.com/apache/beam/pull/10758#issuecomment-581665649 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381240) Time Spent: 20m (was: 10m) > Enable CrossLanguageValidateRunner test for Spark runner > > > Key: BEAM-9230 > URL: https://issues.apache.org/jira/browse/BEAM-9230 > Project: Beam > Issue Type: Improvement > Components: runner-spark >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Enable CrossLanguageValidateRunner test for Spark runner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9230) Enable CrossLanguageValidateRunner test for Spark runner
[ https://issues.apache.org/jira/browse/BEAM-9230?focusedWorklogId=381239=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381239 ] ASF GitHub Bot logged work on BEAM-9230: Author: ASF GitHub Bot Created on: 03/Feb/20 23:15 Start Date: 03/Feb/20 23:15 Worklog Time Spent: 10m Work Description: ihji commented on pull request #10758: [BEAM-9230] Enable CrossLanguageValidateRunner test for Spark runner URL: https://github.com/apache/beam/pull/10758 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[
https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381238
]
ASF GitHub Bot logged work on BEAM-9246:
Author: ASF GitHub Bot
Created on: 03/Feb/20 23:14
Start Date: 03/Feb/20 23:14
Worklog Time Spent: 10m
Work Description: ibzib commented on pull request #10756: [BEAM-9246]
Introducing google-http-client.version property in Java archetype
URL: https://github.com/apache/beam/pull/10756#discussion_r374395160
##
File path: sdks/java/maven-archetypes/examples/build.gradle
##
@@ -28,6 +28,7 @@ processResources {
'project.version': version,
'bigquery.version':
dependencies.create(project.library.java.google_api_services_bigquery).getVersion(),
'google-clients.version':
dependencies.create(project.library.java.google_api_client).getVersion(),
Review comment:
Maybe we should rename `google-clients.version` to better distinguish from
`google-http-client`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381238)
Time Spent: 50m (was: 40m)
> beam_PostRelease_NightlySnapshot failing
>
>
> Key: BEAM-9246
> URL: https://issues.apache.org/jira/browse/BEAM-9246
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Tomo Suzuki
>Priority: Major
> Labels: currently-failing
> Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Multiple tests are failing with error:
> 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could
> not find artifact com.google.http-client:google-http-client:jar:1.30.3 in
> test.release (https://repository.apache.org/content/repositories/snapshots)
> -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381237 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 23:11 Start Date: 03/Feb/20 23:11 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756#issuecomment-581664332 Run Dataflow PostRelease This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381237) Time Spent: 40m (was: 0.5h) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 40m > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381235 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 23:10 Start Date: 03/Feb/20 23:10 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756#issuecomment-581664020 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381235) Time Spent: 0.5h (was: 20m) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8756) Beam Dependency Update Request: com.google.cloud:google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-8756?focusedWorklogId=381234=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381234 ] ASF GitHub Bot logged work on BEAM-8756: Author: ASF GitHub Bot Created on: 03/Feb/20 23:08 Start Date: 03/Feb/20 23:08 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10674: [BEAM-8756] Google cloud client libraries to use 2019 versions URL: https://github.com/apache/beam/pull/10674#issuecomment-581663256 Created PR to fix that. https://github.com/apache/beam/pull/10756 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381234) Time Spent: 3h 40m (was: 3.5h) > Beam Dependency Update Request: com.google.cloud:google-cloud-core > -- > > Key: BEAM-8756 > URL: https://issues.apache.org/jira/browse/BEAM-8756 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > - 2019-11-19 21:05:22.063293 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:10:54.007619 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:01.481594 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:02.830621 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:05:33.624755 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:08.158708 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:46.788844 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:08:23.305323 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > -
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=381229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381229 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 03/Feb/20 22:55 Start Date: 03/Feb/20 22:55 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381229) Time Spent: 23h 20m (was: 23h 10m) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 23h 20m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[
https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381228
]
ASF GitHub Bot logged work on BEAM-9246:
Author: ASF GitHub Bot
Created on: 03/Feb/20 22:46
Start Date: 03/Feb/20 22:46
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10756: [BEAM-9246]
Introducing google-http-client.version property in Java archetype
URL: https://github.com/apache/beam/pull/10756#discussion_r374385290
##
File path:
sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
##
@@ -424,7 +425,7 @@
com.google.http-client
google-http-client
- ${google-clients.version}
Review comment:
Problem was that google-api-client and google-http-client versions have
diverged.
- The latest google-api-client version is 1.30.X.
- The latest google-http-client version is 1.34.X.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381228)
Time Spent: 20m (was: 10m)
> beam_PostRelease_NightlySnapshot failing
>
>
> Key: BEAM-9246
> URL: https://issues.apache.org/jira/browse/BEAM-9246
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Tomo Suzuki
>Priority: Major
> Labels: currently-failing
> Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Multiple tests are failing with error:
> 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could
> not find artifact com.google.http-client:google-http-client:jar:1.30.3 in
> test.release (https://repository.apache.org/content/repositories/snapshots)
> -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381227 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 03/Feb/20 22:43 Start Date: 03/Feb/20 22:43 Worklog Time Spent: 10m Work Description: udim commented on pull request #10735: [BEAM-8280][BEAM-8629] Make IOTypeHints immutable URL: https://github.com/apache/beam/pull/10735 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381227) Time Spent: 4h (was: 3h 50m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?focusedWorklogId=381223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381223 ] ASF GitHub Bot logged work on BEAM-9246: Author: ASF GitHub Bot Created on: 03/Feb/20 22:39 Start Date: 03/Feb/20 22:39 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10756: [BEAM-9246] Introducing google-http-client.version property in Java archetype URL: https://github.com/apache/beam/pull/10756 Fixes https://issues.apache.org/jira/browse/BEAM-9246 . Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[ https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381221 ] ASF GitHub Bot logged work on BEAM-9219: Author: ASF GitHub Bot Created on: 03/Feb/20 22:39 Start Date: 03/Feb/20 22:39 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10745: [BEAM-9219] Streamline creation of Python and Java dependencies pages URL: https://github.com/apache/beam/pull/10745#discussion_r374376538 ## File path: website/src/documentation/sdks/python-dependencies.md ## @@ -26,460 +26,8 @@ behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution. -To see the compile and runtime dependencies for your Beam SDK version, expand -the relevant section below. +To see the compile and runtime dependencies for your Beam SDK version, core SDK dependencies are listed in `https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py` under `REQUIRED_PACKAGES`. Review comment: Python doesn't have a notion of compile-time dependency, I'd say just dependencies. Same as above, I would rewrite this sentence slightly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381221) Time Spent: 2h 20m (was: 2h 10m) > Streamline creation of Python and Java dependencies pages > - > > Key: BEAM-9219 > URL: https://issues.apache.org/jira/browse/BEAM-9219 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: David Wrede >Priority: Minor > Time Spent: 2h 20m > Remaining Estimate: 0h > > This issue is about the need to address keeping both Python and Java SDK > dependency pages more relevant and up-to-date while reducing the amount of > time it takes to provide that information. The current method of scraping and > copying dependencies into a table for every release is a non-trivial task > because of the semi-automated workflows done by the tech writers on the > website. > In an effort to provide accurate dependency listings that are always in sync > with SDK releases, referring people to the appropriate places in the source > code (or through CLI commands) should provide people the information they are > looking for and not require the creation and maintenance of an automated > tooling solution to generate the dependency tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381218
]
ASF GitHub Bot logged work on BEAM-9219:
Author: ASF GitHub Bot
Created on: 03/Feb/20 22:38
Start Date: 03/Feb/20 22:38
Worklog Time Spent: 10m
Work Description: davidwrede commented on pull request #10745:
[BEAM-9219] Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r374382170
##
File path: website/src/documentation/sdks/python-dependencies.md
##
@@ -26,460 +26,8 @@ behavior in the service. If you are using any of these
packages in your code, be
aware that some libraries are not forward-compatible and you may need to pin to
the listed versions that will be in scope during execution.
-To see the compile and runtime dependencies for your Beam SDK version,
expand
-the relevant section below.
+To see the compile and runtime dependencies for your Beam SDK version, core
SDK dependencies are listed in
`https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py`
under `REQUIRED_PACKAGES`.
-2.17.0
+If you installed any [extra features]({{ site.baseurl
}}/get-started/quickstart-py#extra-requirements) along with `apache-beam`, such
as `gcp` or `test`, you should review the lists under `REQUIRED_TEST_PACKAGES`,
`GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies.
-Beam SDK for Python 2.17.0 has the following compile and runtime
dependencies.
-
-
-PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.3.0,0.3.1
- fastavro=0.21.4,0.22
- funcsigs=1.0.2,2; python_version
"3.0"
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,1.1.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,1.1.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.12.1,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.15.1,0.16.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- python-dateutil=2.8.0,3
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.16.0
-
-Beam SDK for Python 2.16.0 has the following compile and
- runtime dependencies.
-
-
- PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.3.0,0.3.1
- fastavro=0.21.4,0.22
- funcsigs=1.0.2,2; python_version
"3.0"
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,1.1.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,1.1.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.12.1,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.11.1,0.15.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- python-dateutil=2.8.0,3
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- pyyaml=3.12,4.0.0
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.15.0
-
-Beam SDK for Python 2.15.0 has the following compile and
- runtime dependencies.
-
- PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.2.9,0.2.10
- fastavro=0.21.4,0.22
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,0.33.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,0.40.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.8,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.11.1,0.15.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- pyyaml=3.12,4.0.0
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.14.0
-
-Beam SDK for Python
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381220
]
ASF GitHub Bot logged work on BEAM-9219:
Author: ASF GitHub Bot
Created on: 03/Feb/20 22:39
Start Date: 03/Feb/20 22:39
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #10745: [BEAM-9219]
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r374375451
##
File path: website/src/documentation/sdks/python-dependencies.md
##
@@ -26,460 +26,8 @@ behavior in the service. If you are using any of these
packages in your code, be
aware that some libraries are not forward-compatible and you may need to pin to
the listed versions that will be in scope during execution.
-To see the compile and runtime dependencies for your Beam SDK version,
expand
-the relevant section below.
+To see the compile and runtime dependencies for your Beam SDK version, core
SDK dependencies are listed in
`https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py`
under `REQUIRED_PACKAGES`.
-2.17.0
+If you installed any [extra features]({{ site.baseurl
}}/get-started/quickstart-py#extra-requirements) along with `apache-beam`, such
as `gcp` or `test`, you should review the lists under `REQUIRED_TEST_PACKAGES`,
`GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies.
-Beam SDK for Python 2.17.0 has the following compile and runtime
dependencies.
-
-
-PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.3.0,0.3.1
- fastavro=0.21.4,0.22
- funcsigs=1.0.2,2; python_version
"3.0"
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,1.1.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,1.1.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.12.1,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.15.1,0.16.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- python-dateutil=2.8.0,3
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.16.0
-
-Beam SDK for Python 2.16.0 has the following compile and
- runtime dependencies.
-
-
- PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.3.0,0.3.1
- fastavro=0.21.4,0.22
- funcsigs=1.0.2,2; python_version
"3.0"
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,1.1.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,1.1.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.12.1,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.11.1,0.15.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- python-dateutil=2.8.0,3
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- pyyaml=3.12,4.0.0
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.15.0
-
-Beam SDK for Python 2.15.0 has the following compile and
- runtime dependencies.
-
- PackageVersion
- avro-python3=1.8.1,2.0.0; python_version =
"3.0"
- avro=1.8.1,2.0.0; python_version
"3.0"
- cachetools=3.1.0,4
- crcmod=1.7,2.0
- dill=0.2.9,0.2.10
- fastavro=0.21.4,0.22
- future=0.16.0,1.0.0
- futures=3.2.0,4.0.0; python_version
"3.0"
- google-apitools=0.5.28,0.5.29
- google-cloud-bigquery=1.6.0,1.18.0
- google-cloud-bigtable=0.31.1,0.33.0
- google-cloud-core=0.28.1,2
- google-cloud-datastore=1.7.1,1.8.0
- google-cloud-pubsub=0.39.0,0.40.0
- googledatastore=7.0.1,7.1; python_version
"3.0"
- grpcio=1.8,2
- hdfs=2.1.0,3.0.0
- httplib2=0.8,=0.12.0
- mock=1.0.1,3.0.0
- oauth2client=2.0.1,4
- proto-google-cloud-datastore-v1=0.90.0,=0.90.4;
python_version "3.0"
- protobuf=3.5.0.post1,4
- pyarrow=0.11.1,0.15.0; python_version = "3.0"
or platform_system != "Windows"
- pydot=1.2.0,2
- pymongo=3.8.0,4.0.0
- pytz=2018.3
- pyvcf=0.6.8,0.7.0; python_version
"3.0"
- pyyaml=3.12,4.0.0
- typing=3.6.0,3.7.0; python_version
"3.5.0"
-
-
-
-
-2.14.0
-
-Beam SDK for Python
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381222
]
ASF GitHub Bot logged work on BEAM-9219:
Author: ASF GitHub Bot
Created on: 03/Feb/20 22:39
Start Date: 03/Feb/20 22:39
Worklog Time Spent: 10m
Work Description: tvalentyn commented on pull request #10745: [BEAM-9219]
Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r374382310
##
File path: website/src/documentation/sdks/python-dependencies.md
##
@@ -26,460 +26,8 @@ behavior in the service. If you are using any of these
packages in your code, be
aware that some libraries are not forward-compatible and you may need to pin to
the listed versions that will be in scope during execution.
-To see the compile and runtime dependencies for your Beam SDK version,
expand
-the relevant section below.
+To see the compile and runtime dependencies for your Beam SDK version, core
SDK dependencies are listed in
`https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py`
under `REQUIRED_PACKAGES`.
-2.17.0
+If you installed any [extra features]({{ site.baseurl
}}/get-started/quickstart-py#extra-requirements) along with `apache-beam`, such
as `gcp` or `test`, you should review the lists under `REQUIRED_TEST_PACKAGES`,
`GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies.
Review comment:
I would say: "If you require extra features ... such as ...,you should
review ... for additional dependencies"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381222)
Time Spent: 2.5h (was: 2h 20m)
> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: David Wrede
>Priority: Minor
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK
> dependency pages more relevant and up-to-date while reducing the amount of
> time it takes to provide that information. The current method of scraping and
> copying dependencies into a table for every release is a non-trivial task
> because of the semi-automated workflows done by the tech writers on the
> website.
> In an effort to provide accurate dependency listings that are always in sync
> with SDK releases, referring people to the appropriate places in the source
> code (or through CLI commands) should provide people the information they are
> looking for and not require the creation and maintenance of an automated
> tooling solution to generate the dependency tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[ https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381219 ] ASF GitHub Bot logged work on BEAM-9219: Author: ASF GitHub Bot Created on: 03/Feb/20 22:39 Start Date: 03/Feb/20 22:39 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10745: [BEAM-9219] Streamline creation of Python and Java dependencies pages URL: https://github.com/apache/beam/pull/10745#discussion_r374372668 ## File path: website/src/documentation/sdks/java-dependencies.md ## @@ -26,336 +26,12 @@ behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution. -To see the compile and runtime dependencies for your Beam SDK version, expand -the relevant section below. +To see the compile and runtime dependencies for your Beam SDK version, dependencies are listed in `https://raw.githubusercontent.com/apache/beam/v/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy` under `project.ext.library`. Review comment: It looks like we might want to rephrase the first sentence. Should we say "Compile and runtime dependencies for your Beam SDK version dependencies are listed in ..." This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381219) Time Spent: 2h 10m (was: 2h) > Streamline creation of Python and Java dependencies pages > - > > Key: BEAM-9219 > URL: https://issues.apache.org/jira/browse/BEAM-9219 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: David Wrede >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > This issue is about the need to address keeping both Python and Java SDK > dependency pages more relevant and up-to-date while reducing the amount of > time it takes to provide that information. The current method of scraping and > copying dependencies into a table for every release is a non-trivial task > because of the semi-automated workflows done by the tech writers on the > website. > In an effort to provide accurate dependency listings that are always in sync > with SDK releases, referring people to the appropriate places in the source > code (or through CLI commands) should provide people the information they are > looking for and not require the creation and maintenance of an automated > tooling solution to generate the dependency tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[
https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381217
]
ASF GitHub Bot logged work on BEAM-9219:
Author: ASF GitHub Bot
Created on: 03/Feb/20 22:37
Start Date: 03/Feb/20 22:37
Worklog Time Spent: 10m
Work Description: davidwrede commented on pull request #10745:
[BEAM-9219] Streamline creation of Python and Java dependencies pages
URL: https://github.com/apache/beam/pull/10745#discussion_r374381699
##
File path: website/src/documentation/sdks/python-dependencies.md
##
@@ -26,460 +26,8 @@ behavior in the service. If you are using any of these
packages in your code, be
aware that some libraries are not forward-compatible and you may need to pin to
the listed versions that will be in scope during execution.
-To see the compile and runtime dependencies for your Beam SDK version,
expand
-the relevant section below.
+To see the compile and runtime dependencies for your Beam SDK version, core
SDK dependencies are listed in
`https://raw.githubusercontent.com/apache/beam/v/sdks/python/setup.py`
under `REQUIRED_PACKAGES`.
-2.17.0
+If you installed any [extra features]({{ site.baseurl
}}/get-started/quickstart-py#extra-requirements) along with `apache-beam`, such
as `gcp` or `test`, you should review the lists under `REQUIRED_TEST_PACKAGES`,
`GCP_REQUIREMENTS`, or `INTERACTIVE_BEAM` for additional dependencies.
Review comment:
Made the sed change.
The other lists live within the setup.py file right below
`REQUIRED_PACKAGES`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381217)
Time Spent: 1h 50m (was: 1h 40m)
> Streamline creation of Python and Java dependencies pages
> -
>
> Key: BEAM-9219
> URL: https://issues.apache.org/jira/browse/BEAM-9219
> Project: Beam
> Issue Type: Improvement
> Components: website
>Reporter: David Wrede
>Priority: Minor
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This issue is about the need to address keeping both Python and Java SDK
> dependency pages more relevant and up-to-date while reducing the amount of
> time it takes to provide that information. The current method of scraping and
> copying dependencies into a table for every release is a non-trivial task
> because of the semi-automated workflows done by the tech writers on the
> website.
> In an effort to provide accurate dependency listings that are always in sync
> with SDK releases, referring people to the appropriate places in the source
> code (or through CLI commands) should provide people the information they are
> looking for and not require the creation and maintenance of an automated
> tooling solution to generate the dependency tables.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[ https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381215=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381215 ] ASF GitHub Bot logged work on BEAM-9219: Author: ASF GitHub Bot Created on: 03/Feb/20 22:36 Start Date: 03/Feb/20 22:36 Worklog Time Spent: 10m Work Description: davidwrede commented on pull request #10745: [BEAM-9219] Streamline creation of Python and Java dependencies pages URL: https://github.com/apache/beam/pull/10745#discussion_r374381034 ## File path: website/src/documentation/sdks/java-dependencies.md ## @@ -26,336 +26,12 @@ behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution. -To see the compile and runtime dependencies for your Beam SDK version, expand -the relevant section below. +To see the compile and runtime dependencies for your Beam SDK version, dependencies are listed in `https://raw.githubusercontent.com/apache/beam/v/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy` under `project.ext.library`. Review comment: I addressed the feedback, but left the placeholder as lowercase per the Google style guide. Given the additional example and note below the URL, I *think* it will be clear to users what the placeholder is and what to replace it with. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381215) Time Spent: 1h 40m (was: 1.5h) > Streamline creation of Python and Java dependencies pages > - > > Key: BEAM-9219 > URL: https://issues.apache.org/jira/browse/BEAM-9219 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: David Wrede >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > > This issue is about the need to address keeping both Python and Java SDK > dependency pages more relevant and up-to-date while reducing the amount of > time it takes to provide that information. The current method of scraping and > copying dependencies into a table for every release is a non-trivial task > because of the semi-automated workflows done by the tech writers on the > website. > In an effort to provide accurate dependency listings that are always in sync > with SDK releases, referring people to the appropriate places in the source > code (or through CLI commands) should provide people the information they are > looking for and not require the creation and maintenance of an automated > tooling solution to generate the dependency tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7256) Add support for allowDiskUse (AggregationOptions) in MongoDbIO
[
https://issues.apache.org/jira/browse/BEAM-7256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029349#comment-17029349
]
Jeff Yemin commented on BEAM-7256:
--
This is possible with the {{MongoCollection}} - based API:
{code}
MongoCollection coll = ...
coll.aggregate(pipeline).allowDiskUse(true)
{code}
> Add support for allowDiskUse (AggregationOptions) in MongoDbIO
> ---
>
> Key: BEAM-7256
> URL: https://issues.apache.org/jira/browse/BEAM-7256
> Project: Beam
> Issue Type: Improvement
> Components: io-java-mongodb
>Affects Versions: 2.12.0
>Reporter: Javier Cornejo
>Priority: Major
> Attachments: Screen Shot 2019-05-09 at 12.30.51.png
>
>
> When a read is executed over a collection that exceed the memory limit of
> 104857600 an exception occurs. This is declared by mongodb and is possible to
> control the error passing a AggregationOptions allowDiskUse true so mongo can
> sort with disk usage.
> This should be happen only when aggregations are added to read but now is
> happening even without aggregation at all.
> Please let me know how can help with this improvement / bug.
>
> !Screen Shot 2019-05-09 at 12.30.51.png!
> {code:java}
> PCollection> updateColls = p.apply("Reading Ops
> Collection: " + key, MongoDbIO .read() .withUri(options.getMongoDBUri())
> .withDatabase("local") .withCollection("oplog.rs") .withBucketAuto(true) //
> .withQueryFn( // FindQuery.create().withFilters( // Filters.and( //
> Filters.gt("ts", ts.format(dtf)), // Filters.eq("ns",
> options.getMongoDBDBName() + "" + key), // Filters.eq("op", "u") // ) // ) //
> // AggregationQuery.create().withMongoDbPipeline(updatedDocsOplogAggregation)
> // ) )
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[
https://issues.apache.org/jira/browse/BEAM-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029346#comment-17029346
]
Tomo Suzuki commented on BEAM-9246:
---
Suspect:
in
sdks/java/maven-archetypes/examples/build/resources/main/archetype-resources/pom.xml
:
{code:java}
com.google.http-client
google-http-client
${google-clients.version}
com.google.guava
guava-jdk5
{code}
> beam_PostRelease_NightlySnapshot failing
>
>
> Key: BEAM-9246
> URL: https://issues.apache.org/jira/browse/BEAM-9246
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kyle Weaver
>Assignee: Tomo Suzuki
>Priority: Major
> Labels: currently-failing
> Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png
>
>
> Multiple tests are failing with error:
> 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not
> resolve dependencies for project org.example:word-count-beam:jar:0.1: Could
> not find artifact com.google.http-client:google-http-client:jar:1.30.3 in
> test.release (https://repository.apache.org/content/repositories/snapshots)
> -> [Help 1]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029339#comment-17029339 ] Tomo Suzuki commented on BEAM-9246: --- The artifact is indeed unavailable in Maven Central. I thought the change did not touch google-http-client. Digging further. > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomo Suzuki updated BEAM-9246: -- Attachment: Screen Shot 2020-02-03 at 5.20.15 PM.png > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > Attachments: Screen Shot 2020-02-03 at 5.20.15 PM.png > > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9246: -- Status: Open (was: Triage Needed) > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9246) beam_PostRelease_NightlySnapshot failing
[ https://issues.apache.org/jira/browse/BEAM-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver reassigned BEAM-9246: - Assignee: Tomo Suzuki > beam_PostRelease_NightlySnapshot failing > > > Key: BEAM-9246 > URL: https://issues.apache.org/jira/browse/BEAM-9246 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kyle Weaver >Assignee: Tomo Suzuki >Priority: Major > Labels: currently-failing > > Multiple tests are failing with error: > 03:03:04 [ERROR] Failed to execute goal on project word-count-beam: Could not > resolve dependencies for project org.example:word-count-beam:jar:0.1: Could > not find artifact com.google.http-client:google-http-client:jar:1.30.3 in > test.release (https://repository.apache.org/content/repositories/snapshots) > -> [Help 1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8756) Beam Dependency Update Request: com.google.cloud:google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-8756?focusedWorklogId=381199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381199 ] ASF GitHub Bot logged work on BEAM-8756: Author: ASF GitHub Bot Created on: 03/Feb/20 22:03 Start Date: 03/Feb/20 22:03 Worklog Time Spent: 10m Work Description: udim commented on issue #10674: [BEAM-8756] Google cloud client libraries to use 2019 versions URL: https://github.com/apache/beam/pull/10674#issuecomment-581640502 This broke nightly snapshots: https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/874/console https://issues.apache.org/jira/browse/BEAM-9246 Any idea what needs to be done? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381199) Time Spent: 3h 20m (was: 3h 10m) > Beam Dependency Update Request: com.google.cloud:google-cloud-core > -- > > Key: BEAM-8756 > URL: https://issues.apache.org/jira/browse/BEAM-8756 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > - 2019-11-19 21:05:22.063293 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:10:54.007619 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:01.481594 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:02.830621 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:05:33.624755 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:08.158708 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:46.788844 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:08:23.305323 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8756) Beam Dependency Update Request: com.google.cloud:google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-8756?focusedWorklogId=381205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381205 ] ASF GitHub Bot logged work on BEAM-8756: Author: ASF GitHub Bot Created on: 03/Feb/20 22:06 Start Date: 03/Feb/20 22:06 Worklog Time Spent: 10m Work Description: ibzib commented on issue #10674: [BEAM-8756] Google cloud client libraries to use 2019 versions URL: https://github.com/apache/beam/pull/10674#issuecomment-581641846 cc self @ibzib This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381205) Time Spent: 3.5h (was: 3h 20m) > Beam Dependency Update Request: com.google.cloud:google-cloud-core > -- > > Key: BEAM-8756 > URL: https://issues.apache.org/jira/browse/BEAM-8756 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > - 2019-11-19 21:05:22.063293 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:10:54.007619 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:01.481594 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.91.3 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:02.830621 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:05:33.624755 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:08.158708 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:08:46.788844 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:08:23.305323 > - > Please consider upgrading the dependency > com.google.cloud:google-cloud-core. > The current version is 1.61.0. The latest version is 1.92.2 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:09:28.216096 > - >
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381204=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381204 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 03/Feb/20 22:06 Start Date: 03/Feb/20 22:06 Worklog Time Spent: 10m Work Description: udim commented on issue #10735: [BEAM-8280][BEAM-8629] Make IOTypeHints immutable URL: https://github.com/apache/beam/pull/10735#issuecomment-581641703 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381204) Time Spent: 3h 50m (was: 3h 40m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 50m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[ https://issues.apache.org/jira/browse/BEAM-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Cwik closed BEAM-8691. --- Fix Version/s: 2.20.0 Resolution: Fixed > Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core > -- > > Key: BEAM-8691 > URL: https://issues.apache.org/jira/browse/BEAM-8691 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: 2.20.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > - 2019-11-15 19:39:51.523448 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:05:43.901882 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:11:30.163557 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:37.979355 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:39.422837 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:11.312353 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:45.627449 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:23.823869 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-20 12:09:00.213436 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-27 12:10:06.151868 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.13.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[ https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=381197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381197 ] ASF GitHub Bot logged work on BEAM-8691: Author: ASF GitHub Bot Created on: 03/Feb/20 21:51 Start Date: 03/Feb/20 21:51 Worklog Time Spent: 10m Work Description: lukecwik commented on pull request #10714: [BEAM-8691] bigtable-client-core dependency upgrade URL: https://github.com/apache/beam/pull/10714 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381197) Time Spent: 7h 50m (was: 7h 40m) > Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core > -- > > Key: BEAM-8691 > URL: https://issues.apache.org/jira/browse/BEAM-8691 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: 2.20.0 > > Time Spent: 7h 50m > Remaining Estimate: 0h > > - 2019-11-15 19:39:51.523448 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:05:43.901882 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:11:30.163557 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:10:37.979355 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:10:39.422837 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:11.312353 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:09:45.627449 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:23.823869 > - > Please consider upgrading the dependency > com.google.cloud.bigtable:bigtable-client-core. > The current version is 1.8.0. The latest version is 1.12.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The
[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=381196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381196
]
ASF GitHub Bot logged work on BEAM-8691:
Author: ASF GitHub Bot
Created on: 03/Feb/20 21:51
Start Date: 03/Feb/20 21:51
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10714: [BEAM-8691]
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#discussion_r374361544
##
File path: sdks/java/io/google-cloud-platform/build.gradle
##
@@ -41,9 +41,14 @@ dependencies {
compile library.java.google_auth_library_credentials
compile library.java.google_auth_library_oauth2_http
compile library.java.google_cloud_bigquery_storage
- compile library.java.google_cloud_bigtable_client_core
+ compile(library.java.google_cloud_bigtable_client_core) {
Review comment:
I wasn't aware that the build would fail before the dependency mediation
kicked in.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381196)
Time Spent: 7h 40m (was: 7.5h)
> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
> Fix For: 2.20.0
>
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> - 2019-11-15 19:39:51.523448
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-11-19 21:05:43.901882
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-02 12:11:30.163557
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-09 12:10:37.979355
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-23 12:10:39.422837
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-30 14:06:11.312353
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2020-01-06 12:09:45.627449
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
>
[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite
[ https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=381192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381192 ] ASF GitHub Bot logged work on BEAM-7961: Author: ASF GitHub Bot Created on: 03/Feb/20 21:45 Start Date: 03/Feb/20 21:45 Worklog Time Spent: 10m Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests for all runner native transforms for XLang URL: https://github.com/apache/beam/pull/10051#issuecomment-581633831 @chamikaramj ready to go :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381192) Time Spent: 23h 10m (was: 23h) > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite > -- > > Key: BEAM-7961 > URL: https://issues.apache.org/jira/browse/BEAM-7961 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 23h 10m > Remaining Estimate: 0h > > Add tests for all runner native transforms and some widely used composite > transforms to cross-language validates runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9178) Support ZetaSQL TIMESTAMP functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-9178?focusedWorklogId=381185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381185 ] ASF GitHub Bot logged work on BEAM-9178: Author: ASF GitHub Bot Created on: 03/Feb/20 21:13 Start Date: 03/Feb/20 21:13 Worklog Time Spent: 10m Work Description: robinyqiu commented on issue #10634: [BEAM-9178] Support all ZetaSQL TIMESTAMP functions URL: https://github.com/apache/beam/pull/10634#issuecomment-581620899 Friendly ping. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381185) Time Spent: 0.5h (was: 20m) > Support ZetaSQL TIMESTAMP functions in BeamSQL > -- > > Key: BEAM-9178 > URL: https://issues.apache.org/jira/browse/BEAM-9178 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Yueyang Qiu >Assignee: Yueyang Qiu >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Support *all* TIMESTAMP functions defined in ZetaSQL (BigQuery Standard SQL). > See the full list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/timestamp_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381184 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 03/Feb/20 21:12 Start Date: 03/Feb/20 21:12 Worklog Time Spent: 10m Work Description: udim commented on issue #10735: [BEAM-8280][BEAM-8629] Make IOTypeHints immutable URL: https://github.com/apache/beam/pull/10735#issuecomment-581620461 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381184) Time Spent: 3h 40m (was: 3.5h) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381182 ] ASF GitHub Bot logged work on BEAM-8280: Author: ASF GitHub Bot Created on: 03/Feb/20 20:50 Start Date: 03/Feb/20 20:50 Worklog Time Spent: 10m Work Description: chadrik commented on issue #10735: [BEAM-8280][BEAM-8629] Make IOTypeHints immutable URL: https://github.com/apache/beam/pull/10735#issuecomment-581612089 looks good! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381182) Time Spent: 3.5h (was: 3h 20m) > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=381174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381174
]
ASF GitHub Bot logged work on BEAM-8691:
Author: ASF GitHub Bot
Created on: 03/Feb/20 20:11
Start Date: 03/Feb/20 20:11
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10714: [BEAM-8691]
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#discussion_r374230141
##
File path: sdks/java/io/google-cloud-platform/build.gradle
##
@@ -41,9 +41,14 @@ dependencies {
compile library.java.google_auth_library_credentials
compile library.java.google_auth_library_oauth2_http
compile library.java.google_cloud_bigquery_storage
- compile library.java.google_cloud_bigtable_client_core
+ compile(library.java.google_cloud_bigtable_client_core) {
Review comment:
These exclusions are for Maven's version range constraint. The dependency
was transitively requesting `io.grpc:grpc-core:jar:[1.26.0,1.26.0]`, while
another dependency requests `io.grpc:grpc-core:jar:[1.25.0,1.25.0]`. The
conflict would fail the build before Maven's dependency mediation (that would
choose the closest one). Detail in
https://issues.apache.org/jira/browse/BEAM-8691?focusedCommentId=17025456=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17025456
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381174)
Time Spent: 7h 20m (was: 7h 10m)
> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> - 2019-11-15 19:39:51.523448
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-11-19 21:05:43.901882
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-02 12:11:30.163557
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-09 12:10:37.979355
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-23 12:10:39.422837
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-30 14:06:11.312353
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
>
[jira] [Work logged] (BEAM-8691) Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[
https://issues.apache.org/jira/browse/BEAM-8691?focusedWorklogId=381176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381176
]
ASF GitHub Bot logged work on BEAM-8691:
Author: ASF GitHub Bot
Created on: 03/Feb/20 20:11
Start Date: 03/Feb/20 20:11
Worklog Time Spent: 10m
Work Description: suztomo commented on pull request #10714: [BEAM-8691]
bigtable-client-core dependency upgrade
URL: https://github.com/apache/beam/pull/10714#discussion_r374230141
##
File path: sdks/java/io/google-cloud-platform/build.gradle
##
@@ -41,9 +41,14 @@ dependencies {
compile library.java.google_auth_library_credentials
compile library.java.google_auth_library_oauth2_http
compile library.java.google_cloud_bigquery_storage
- compile library.java.google_cloud_bigtable_client_core
+ compile(library.java.google_cloud_bigtable_client_core) {
Review comment:
These exclusions are for Maven's version range constraint. The dependency
was transitively requesting `io.grpc:grpc-core:jar:[1.26.0,1.26.0]`, while
another dependency requests `io.grpc:grpc-core:jar:[1.25.0,1.25.0]`. This
version range conflict would fail the build before Maven's dependency mediation
(that would choose the closest one) happens. Detail in
https://issues.apache.org/jira/browse/BEAM-8691?focusedCommentId=17025456=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17025456
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381176)
Time Spent: 7.5h (was: 7h 20m)
> Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
> --
>
> Key: BEAM-8691
> URL: https://issues.apache.org/jira/browse/BEAM-8691
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> - 2019-11-15 19:39:51.523448
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-11-19 21:05:43.901882
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-02 12:11:30.163557
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-09 12:10:37.979355
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-23 12:10:39.422837
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
> |https://beam.apache.org/contribute/dependencies/]for more information.
> Do Not Modify The Description Above.
> - 2019-12-30 14:06:11.312353
> -
> Please consider upgrading the dependency
> com.google.cloud.bigtable:bigtable-client-core.
> The current version is 1.8.0. The latest version is 1.12.1
> cc:
> Please refer to [Beam Dependency Guide
>
[jira] [Work logged] (BEAM-9219) Streamline creation of Python and Java dependencies pages
[ https://issues.apache.org/jira/browse/BEAM-9219?focusedWorklogId=381171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381171 ] ASF GitHub Bot logged work on BEAM-9219: Author: ASF GitHub Bot Created on: 03/Feb/20 19:57 Start Date: 03/Feb/20 19:57 Worklog Time Spent: 10m Work Description: aaltay commented on issue #10745: [BEAM-9219] Streamline creation of Python and Java dependencies pages URL: https://github.com/apache/beam/pull/10745#issuecomment-581590216 > > Should we keep the old information in the same page for previous releases? Removing might be a regression for existing users. We could hide in a collapsed box maybe? > > I worry that keeping these tables will lead to the expectation that we'll create more tables. Based on the feedback of the Java deps page, it seems to be confusing to just have tables for some of the old versions. Sounds good. If the new procedure could get the same information as the tables let's drop it. And if it cannot, that means we need to improve the new process. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381171) Time Spent: 1.5h (was: 1h 20m) > Streamline creation of Python and Java dependencies pages > - > > Key: BEAM-9219 > URL: https://issues.apache.org/jira/browse/BEAM-9219 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: David Wrede >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > This issue is about the need to address keeping both Python and Java SDK > dependency pages more relevant and up-to-date while reducing the amount of > time it takes to provide that information. The current method of scraping and > copying dependencies into a table for every release is a non-trivial task > because of the semi-automated workflows done by the tech writers on the > website. > In an effort to provide accurate dependency listings that are always in sync > with SDK releases, referring people to the appropriate places in the source > code (or through CLI commands) should provide people the information they are > looking for and not require the creation and maintenance of an automated > tooling solution to generate the dependency tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[
https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381170
]
ASF GitHub Bot logged work on BEAM-8280:
Author: ASF GitHub Bot
Created on: 03/Feb/20 19:55
Start Date: 03/Feb/20 19:55
Worklog Time Spent: 10m
Work Description: chadrik commented on pull request #10735:
[BEAM-8280][BEAM-8629] Make IOTypeHints immutable
URL: https://github.com/apache/beam/pull/10735#discussion_r374308055
##
File path: sdks/python/apache_beam/typehints/decorators.py
##
@@ -217,7 +220,10 @@ def get_signature(func):
return signature
-class IOTypeHints(object):
+class IOTypeHints(NamedTuple('IOTypeHints', [
+('input_types', Optional[Tuple[Tuple[Any, ...], Dict[str, Any]]]),
+('output_types', Optional[Tuple[Tuple[Any, ...], Dict[str, Any]]]),
+('origin', List[str])])):
Review comment:
I assumed that mypy would need the intermediate class _or_ would want that
class's name to match the name provided to `NamedTuple()`. For example, both
of these are invalid:
```python
class Foo(Generic[TypeVar('T')]): # TypeVar needs to be defined above
pass
T = TypeVar('T_with_suffix') # TypeVar name needs to match variable name
class Bar(Generic[T]):
pass
```
I just did my own isolated test of this and you're right, mypy has no issue
with it, so it appears there's nothing special about the name given to
`NamedTuple`, as far as mypy is concerned.
Thanks for taking the time to test that. I was trying to give some quick
feedback without getting too drawn in, but once I _did_ get drawn in, I should
have gone back and checked my original assumption!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381170)
Time Spent: 3h 20m (was: 3h 10m)
> re-enable IOTypeHints.from_callable
> ---
>
> Key: BEAM-8280
> URL: https://issues.apache.org/jira/browse/BEAM-8280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> See https://issues.apache.org/jira/browse/BEAM-8279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9085) Investigate performance difference between Python 2/3 on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029222#comment-17029222
]
Valentyn Tymofieiev edited comment on BEAM-9085 at 2/3/20 7:54 PM:
---
Somewhat faster way to obtain the same information is to pass: --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp
if gprof2dot is installed, it creates an SVG diagram.
Full command:
{noformat}
python setup.py nosetests --nocapture --test-pipeline-options=" --iterations=10
--number_of_counters=1 --number_of_counter_operations=1 --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp --project=big-query-project
--publish_to_big_query=false --metrics_dataset=python_load_tests
--metrics_table=pardo --input_options='{ \"num_records\": 10, \"key_size\":
10, \"value_size\":90, \"bundle_size_distribution_type\": \"const\",
\"bundle_size_distribution_param\": 1, \"force_initial_num_bundles\": 0 }'"
--tests apache_beam.testing.load_tests.pardo_test
{noformat}
was (Author: tvalentyn):
Somewhat faster way to obtain the same information is to pass: --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp
Full command:
{noformat}
python setup.py nosetests --nocapture --test-pipeline-options=" --iterations=10
--number_of_counters=1 --number_of_counter_operations=1 --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp --project=big-query-project
--publish_to_big_query=false --metrics_dataset=python_load_tests
--metrics_table=pardo --input_options='{ \"num_records\": 10, \"key_size\":
10, \"value_size\":90, \"bundle_size_distribution_type\": \"const\",
\"bundle_size_distribution_param\": 1, \"force_initial_num_bundles\": 0 }'"
--tests apache_beam.testing.load_tests.pardo_test
{noformat}
> Investigate performance difference between Python 2/3 on Dataflow
> -
>
> Key: BEAM-9085
> URL: https://issues.apache.org/jira/browse/BEAM-9085
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kamil Wasilewski
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Tests show that the performance of core Beam operations in Python 3.x on
> Dataflow can be a few time slower than in Python 2.7. We should investigate
> what's the cause of the problem.
> Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A
> dashboard with runtime results can be found here [2].
> [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py
> [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9220) Add use_runner_v2 argument for dataflow
[ https://issues.apache.org/jira/browse/BEAM-9220?focusedWorklogId=381169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381169 ] ASF GitHub Bot logged work on BEAM-9220: Author: ASF GitHub Bot Created on: 03/Feb/20 19:52 Start Date: 03/Feb/20 19:52 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10720: [BEAM-9220] Adding argument use_runner_v2 for dataflow unified worker URL: https://github.com/apache/beam/pull/10720 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 381169) Time Spent: 1h 40m (was: 1.5h) > Add use_runner_v2 argument for dataflow > --- > > Key: BEAM-9220 > URL: https://issues.apache.org/jira/browse/BEAM-9220 > Project: Beam > Issue Type: New Feature > Components: runner-dataflow >Reporter: Ankur Goenka >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9085) Investigate performance difference between Python 2/3 on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-9085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029222#comment-17029222
]
Valentyn Tymofieiev commented on BEAM-9085:
---
Somewhat faster way to obtain the same information is to pass: --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp
Full command:
{noformat}
python setup.py nosetests --nocapture --test-pipeline-options=" --iterations=10
--number_of_counters=1 --number_of_counter_operations=1 --profile_cpu
--profile_sample_rate=1 --profile_location=/tmp --project=big-query-project
--publish_to_big_query=false --metrics_dataset=python_load_tests
--metrics_table=pardo --input_options='{ \"num_records\": 10, \"key_size\":
10, \"value_size\":90, \"bundle_size_distribution_type\": \"const\",
\"bundle_size_distribution_param\": 1, \"force_initial_num_bundles\": 0 }'"
--tests apache_beam.testing.load_tests.pardo_test
{noformat}
> Investigate performance difference between Python 2/3 on Dataflow
> -
>
> Key: BEAM-9085
> URL: https://issues.apache.org/jira/browse/BEAM-9085
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kamil Wasilewski
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Tests show that the performance of core Beam operations in Python 3.x on
> Dataflow can be a few time slower than in Python 2.7. We should investigate
> what's the cause of the problem.
> Currently, we have one ParDo test that is run both in Py3 and Py2 [1]. A
> dashboard with runtime results can be found here [2].
> [1] sdks/python/apache_beam/testing/load_tests/pardo_test.py
> [2] https://apache-beam-testing.appspot.com/explore?dashboard=5678187241537536
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[
https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381168
]
ASF GitHub Bot logged work on BEAM-8280:
Author: ASF GitHub Bot
Created on: 03/Feb/20 19:50
Start Date: 03/Feb/20 19:50
Worklog Time Spent: 10m
Work Description: udim commented on pull request #10735:
[BEAM-8280][BEAM-8629] Make IOTypeHints immutable
URL: https://github.com/apache/beam/pull/10735#discussion_r374305378
##
File path: sdks/python/apache_beam/typehints/decorators.py
##
@@ -228,18 +234,30 @@ class IOTypeHints(object):
May be None. The list and dict correspond to args and kwargs.
output_types: (tuple, dict) List of typing types, and an optional
dictionary
(unused). Only the first element of the list is used. May be None.
+origin: (List[str]) Stack of tracebacks of method calls used to create this
+ instance.
"""
- __slots__ = ('input_types', 'output_types')
- def __init__(self,
- input_types=None, # type: Optional[Tuple[Tuple[Any, ...],
Dict[str, Any]]]
- output_types=None # type: Optional[Tuple[Tuple[Any, ...],
Dict[str, Any]]]
- ):
-self.input_types = input_types
-self.output_types = output_types
+ traceback_limit = 5
- @staticmethod
- def from_callable(fn):
+ @classmethod
+ def _make_traceback(cls, base): # type: (Optional[IOTypeHints]) -> List[str]
+# Omit this method and the IOTypeHints method that called it.
+num_frames_skip = 2
+tb = traceback.format_stack(limit=cls.traceback_limit + num_frames_skip)
+tb_lines = 'TH>' + ''.join(tb[:-num_frames_skip]).replace('\n', '\nTH>')
+
+res = [tb_lines + '\nbased on: ' + str(base)]
+if base is not None:
+ res += base.origin
+return res
+
+ @classmethod
+ def empty(cls):
+return IOTypeHints(None, None, [])
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381168)
Time Spent: 3h 10m (was: 3h)
> re-enable IOTypeHints.from_callable
> ---
>
> Key: BEAM-8280
> URL: https://issues.apache.org/jira/browse/BEAM-8280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> See https://issues.apache.org/jira/browse/BEAM-8279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8280) re-enable IOTypeHints.from_callable
[
https://issues.apache.org/jira/browse/BEAM-8280?focusedWorklogId=381163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381163
]
ASF GitHub Bot logged work on BEAM-8280:
Author: ASF GitHub Bot
Created on: 03/Feb/20 19:39
Start Date: 03/Feb/20 19:39
Worklog Time Spent: 10m
Work Description: udim commented on pull request #10735:
[BEAM-8280][BEAM-8629] Make IOTypeHints immutable
URL: https://github.com/apache/beam/pull/10735#discussion_r374300256
##
File path: sdks/python/apache_beam/typehints/decorators.py
##
@@ -217,7 +220,10 @@ def get_signature(func):
return signature
-class IOTypeHints(object):
+class IOTypeHints(NamedTuple('IOTypeHints', [
+('input_types', Optional[Tuple[Tuple[Any, ...], Dict[str, Any]]]),
+('output_types', Optional[Tuple[Tuple[Any, ...], Dict[str, Any]]]),
+('origin', List[str])])):
Review comment:
Trip up how?
I see this error in lint (should be fixed in next commit):
```
apache_beam/typehints/decorators.py:353: error: Value of type
"Optional[Tuple[Tuple[Any, ...], Dict[str, Any]]]" is not indexable [index]
```
This is the line:
```
output_type = self.output_types[0][0]
```
So it seems that mypy is correctly parsing these type hints.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381163)
Time Spent: 3h (was: 2h 50m)
> re-enable IOTypeHints.from_callable
> ---
>
> Key: BEAM-8280
> URL: https://issues.apache.org/jira/browse/BEAM-8280
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> See https://issues.apache.org/jira/browse/BEAM-8279
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-3221) Model pipeline representation improvements
[
https://issues.apache.org/jira/browse/BEAM-3221?focusedWorklogId=381162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-381162
]
ASF GitHub Bot logged work on BEAM-3221:
Author: ASF GitHub Bot
Created on: 03/Feb/20 19:34
Start Date: 03/Feb/20 19:34
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10754: [BEAM-3221]
Expand/clarify timestamp comments within beam_runner_api.proto
URL: https://github.com/apache/beam/pull/10754#discussion_r374297835
##
File path: model/pipeline/src/main/proto/beam_runner_api.proto
##
@@ -37,15 +37,21 @@ import "google/protobuf/timestamp.proto";
message BeamConstants {
enum Constants {
// All timestamps in milliseconds since Jan 1, 1970.
+
+// All timestamps of elements or window boundaries must be within
+// the interval [MIN_TIMESTAMP_MILLIS, MAX_TIMESTAMP_MILLIS].
+// The smallest representable timestamp of an element or a window boundary.
MIN_TIMESTAMP_MILLIS = 0 [(beam_constant) = "-9223372036854775"];
+// The largest representable timestamp of an element or a window boundary.
MAX_TIMESTAMP_MILLIS = 1 [(beam_constant) = "9223372036854775"];
+
// The maximum timestamp for the global window.
-// Triggers use maxTimestamp to set timers' timestamp. Timers fires when
+// Triggers use max timestamp to set timers' timestamp. Timers fire when
// the watermark passes their timestamps. So, the timestamp needs to be
// smaller than the MAX_TIMESTAMP_MILLIS.
// One standard day is subtracted from MAX_TIMESTAMP_MILLIS to make sure
-// the maxTimestamp is smaller than MAX_TIMESTAMP_MILLIS even after
rounding up
-// to seconds or minutes. See also GlobalWindow in the Java SDK.
+// the max timestamp is smaller than MAX_TIMESTAMP_MILLIS even after
rounding up
+// to seconds or minutes.
GLOBAL_WINDOW_MAX_TIMESTAMP_MILLIS = 2 [(beam_constant) =
"9223371950454775"];
Review comment:
We would need to define an EnumValueOption for each type we would like to
support.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 381162)
Time Spent: 40m (was: 0.5h)
> Model pipeline representation improvements
> --
>
> Key: BEAM-3221
> URL: https://issues.apache.org/jira/browse/BEAM-3221
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
>Reporter: Henning Rohde
>Priority: Major
> Labels: portability
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Collections of various (breaking) tweaks to the Runner API, notably the
> pipeline representation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
