date:20191230



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364905
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 07:39
Start Date: 31/Dec/19 07:39
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#issuecomment-569882002
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364905)
Time Spent: 13h 10m  (was: 13h)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364893=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364893
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 31/Dec/19 06:54
Start Date: 31/Dec/19 06:54
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10482: [BEAM-5600] Add 
unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#issuecomment-569876432
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364893)
Time Spent: 2h 20m  (was: 2h 10m)

> Splitting for SplittableDoFn should be exposed within runner shared libraries
> -
>
> Key: BEAM-5600
> URL: https://issues.apache.org/jira/browse/BEAM-5600
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364870
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:45
Start Date: 31/Dec/19 05:45
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569868458
 
 
   Retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364870)
Time Spent: 3h 20m  (was: 3h 10m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364869
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:44
Start Date: 31/Dec/19 05:44
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569868458
 
 
   Retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364869)
Time Spent: 3h 10m  (was: 3h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364868
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:40
Start Date: 31/Dec/19 05:40
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10464: 
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362151892
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   Do you think we should add the WireCoderSetting property for each input and 
output edge explicitly in the proto? 
   
   I was thinking the setting is not a necessary properties for each edge so 
added the wire_coder_settings as an ExecutableStage level property. The 
wire_coder_settings can be an empty list.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364868)
Time Spent: 13h  (was: 12h 50m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364866=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364866
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:39
Start Date: 31/Dec/19 05:39
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10464: 
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362142107
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   Please see the new PR: 
   https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364866)
Time Spent: 12h 50m  (was: 12h 40m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364865
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:37
Start Date: 31/Dec/19 05:37
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#issuecomment-569856179
 
 
   I can not reopen this PR, So , I have open a new PR here: 
https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364865)
Time Spent: 12h 40m  (was: 12.5h)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 12h 40m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364861=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364861
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 05:35
Start Date: 31/Dec/19 05:35
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10464: 
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464
 
 
   In this PR would address multiple inputs/outputs for wire coder settings as 
suggested in https://github.com/apache/beam/pull/9979
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364850=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364850
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:47
Start Date: 31/Dec/19 03:47
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10484: [BEAM-7951] 
Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484#issuecomment-569858254
 
 
   I have rebase the code.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364850)
Time Spent: 12h 20m  (was: 12h 10m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 12h 20m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364845=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364845
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:38
Start Date: 31/Dec/19 03:38
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10484: 
[BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484#discussion_r362142926
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   Hi @lukecwik. Do you think we should add the WireCoderSetting property for 
each input and output edge explicitly in the proto? 
   I was thinking the setting is not a necessary properties for each edge so 
added the wire_coder_settings as an ExecutableStage level property. The 
wire_coder_settings can be an empty list. 
   What do you think?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364845)
Time Spent: 12h 10m  (was: 12h)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364844=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364844
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:37
Start Date: 31/Dec/19 03:37
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10484: 
[BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484#discussion_r362142854
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   
![image](https://user-images.githubusercontent.com/22488084/71609233-e7735800-2bc1-11ea-88cd-bc7cf547b964.png)
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364844)
Time Spent: 12h  (was: 11h 50m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364841=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364841
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:28
Start Date: 31/Dec/19 03:28
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10464: 
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362142107
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   Please see the new PR: 
   https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364841)
Time Spent: 11h 50m  (was: 11h 40m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364839=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364839
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:25
Start Date: 31/Dec/19 03:25
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#issuecomment-569856179
 
 
   I can not reopen this PR, So , I have open a new PR here: 
https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364839)
Time Spent: 11h 40m  (was: 11.5h)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364837
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 03:06
Start Date: 31/Dec/19 03:06
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#issuecomment-569850965
 
 
   Sorry, I accidentally deleted the branch, So, reopen the PR here: 
   https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364837)
Time Spent: 11.5h  (was: 11h 20m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364834=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364834
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:37
Start Date: 31/Dec/19 02:37
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#issuecomment-569850965
 
 
   Sorry, I accidentally deleted the branch, So, reopen the PR here: 
   https://github.com/apache/beam/pull/10484
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364834)
Time Spent: 11h 20m  (was: 11h 10m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364831=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364831
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:36
Start Date: 31/Dec/19 02:36
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10484: 
[BEAM-7951] Improve the docs for beam_runner_api.proto and WindowedVa…
URL: https://github.com/apache/beam/pull/10484
 
 
   In this PR would address multiple inputs/outputs for wire coder settings as 
suggested in #9979
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364829
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:31
Start Date: 31/Dec/19 02:31
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10464: 
[BEAM-7951] Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364829)
Time Spent: 11h  (was: 10h 50m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364826=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364826
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:07
Start Date: 31/Dec/19 02:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10482: [BEAM-5600] Add 
unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#issuecomment-569847995
 
 
   Run Python2_PVR_Flink PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364826)
Time Spent: 2h 10m  (was: 2h)

> Splitting for SplittableDoFn should be exposed within runner shared libraries
> -
>
> Key: BEAM-5600
> URL: https://issues.apache.org/jira/browse/BEAM-5600
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364825=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364825
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:07
Start Date: 31/Dec/19 02:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10482: [BEAM-5600] Add 
unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#issuecomment-569847979
 
 
   Run Portable_Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364825)
Time Spent: 2h  (was: 1h 50m)

> Splitting for SplittableDoFn should be exposed within runner shared libraries
> -
>
> Key: BEAM-5600
> URL: https://issues.apache.org/jira/browse/BEAM-5600
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
>  Labels: portability
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364824=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364824
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 31/Dec/19 02:07
Start Date: 31/Dec/19 02:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10482: [BEAM-5600] Add 
unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#issuecomment-569847967
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364824)
Time Spent: 1h 50m  (was: 1h 40m)

> Splitting for SplittableDoFn should be exposed within runner shared libraries
> -
>
> Key: BEAM-5600
> URL: https://issues.apache.org/jira/browse/BEAM-5600
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (BEAM-8882) Allow Dataflow to automatically choose portability or not.

2019-12-30 Thread Udi Meiri (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reopened BEAM-8882:
-

Reopening since we found what seems to be a bug with this feature.

Consider rolling back the cherrypick to 2.18.

> Allow Dataflow to automatically choose portability or not.
> --
>
> Key: BEAM-8882
> URL: https://issues.apache.org/jira/browse/BEAM-8882
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Critical
> Fix For: 2.18.0
>
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> We would like the Dataflow service to be able to automatically choose whether 
> to run pipelines in a portable way. In order to do this, we need to provide 
> more information even if portability is not explicitly requested. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364820=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364820
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 31/Dec/19 01:45
Start Date: 31/Dec/19 01:45
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364820)
Time Spent: 2h 20m  (was: 2h 10m)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364805=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364805
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 31/Dec/19 00:01
Start Date: 31/Dec/19 00:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10458: [BEAM-7951]  
Improve the docs for `beam_runner_api.proto` and `WindowedValue.java`
URL: https://github.com/apache/beam/pull/10458
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364805)
Time Spent: 10h 50m  (was: 10h 40m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364804
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 31/Dec/19 00:01
Start Date: 31/Dec/19 00:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364804)
Time Spent: 2h 10m  (was: 2h)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8487) Python typehints: support forward references



 [ 
https://issues.apache.org/jira/browse/BEAM-8487?focusedWorklogId=364803=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364803
 ]

ASF GitHub Bot logged work on BEAM-8487:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:53
Start Date: 30/Dec/19 23:53
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9888: [BEAM-8487] 
Convert forward references to Any
URL: https://github.com/apache/beam/pull/9888
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364803)
Time Spent: 1h  (was: 50m)

> Python typehints: support forward references
> 
>
> Key: BEAM-8487
> URL: https://issues.apache.org/jira/browse/BEAM-8487
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Typehints may be given as string literals: 
> https://www.python.org/dev/peps/pep-0484/#forward-references
> These are currently not evaluated and result in errors.
> Example 1:
> {code}
>   def test_typed_callable_string_hints(self):
> def do_fn(element: 'int') -> 'typehints.List[str]':
>   return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > return issubclass(sub, base)
> E TypeError: issubclass() arg 2 must be a class or tuple of classes
> typehints.py:1168: TypeError
> {code}
> Example 2:
> {code}
>   def test_typed_dofn_string_hints(self):
> class MyDoFn(beam.DoFn):
>   def process(self, element: 'int') -> 'typehints.List[str]':
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(MyDoFn())
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> This results in:
> {code}
> > raise ValueError('%s is not iterable' % type_hint)
> E ValueError: typehints.List[str] is not iterable
> typehints.py:1194: ValueError
> {code}
> where the non-iterable entity the error refers to is a string literal 
> ("typehints.List[str]").



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364802=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364802
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:50
Start Date: 30/Dec/19 23:50
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483#issuecomment-569829577
 
 
   LGTM for the Go changes, since the Go SDK doesn't yet support sub element 
splitting on fractions.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364802)
Time Spent: 2h 10m  (was: 2h)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite



 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=364799=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364799
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:45
Start Date: 30/Dec/19 23:45
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-568606886
 
 
   Run XVR_Flink PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364799)
Time Spent: 11h 50m  (was: 11h 40m)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite



 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=364798=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364798
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:45
Start Date: 30/Dec/19 23:45
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-569828720
 
 
   Run XVR_Flink PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364798)
Time Spent: 11h 40m  (was: 11.5h)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite



 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=364800=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364800
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:45
Start Date: 30/Dec/19 23:45
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-569828774
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364800)
Time Spent: 12h  (was: 11h 50m)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7961) Add tests for all runner native transforms and some widely used composite transforms to cross-language validates runner test suite



 [ 
https://issues.apache.org/jira/browse/BEAM-7961?focusedWorklogId=364801=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364801
 ]

ASF GitHub Bot logged work on BEAM-7961:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:45
Start Date: 30/Dec/19 23:45
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10051: [BEAM-7961] Add tests 
for all runner native transforms for XLang
URL: https://github.com/apache/beam/pull/10051#issuecomment-569828774
 
 
   run java precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364801)
Time Spent: 12h 10m  (was: 12h)

> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite
> --
>
> Key: BEAM-7961
> URL: https://issues.apache.org/jira/browse/BEAM-7961
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> Add tests for all runner native transforms and some widely used composite 
> transforms to cross-language validates runner test suite



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364797=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364797
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:37
Start Date: 30/Dec/19 23:37
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483#issuecomment-569827476
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364797)
Time Spent: 2h  (was: 1h 50m)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364796=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364796
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:21
Start Date: 30/Dec/19 23:21
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483#issuecomment-569824981
 
 
   Run Portable_Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364796)
Time Spent: 1h 50m  (was: 1h 40m)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364795=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364795
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 23:20
Start Date: 30/Dec/19 23:20
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483#issuecomment-569824821
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364795)
Time Spent: 1h 40m  (was: 1.5h)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364793=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364793
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 22:52
Start Date: 30/Dec/19 22:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483#issuecomment-569819678
 
 
   R: @boyuanzz 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364793)
Time Spent: 1.5h  (was: 1h 20m)

> Support Portable SplittableDoFn for batch
> -
>
> Key: BEAM-5605
> URL: https://issues.apache.org/jira/browse/BEAM-5605
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Luke Cwik
>Priority: Major
>  Labels: portability
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Roll-up item tracking work towards supporting portable SplittableDoFn for 
> batch



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5605) Support Portable SplittableDoFn for batch



 [ 
https://issues.apache.org/jira/browse/BEAM-5605?focusedWorklogId=364792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364792
 ]

ASF GitHub Bot logged work on BEAM-5605:


Author: ASF GitHub Bot
Created on: 30/Dec/19 22:52
Start Date: 30/Dec/19 22:52
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10483: [BEAM-5605] 
Increase precision of fraction used during splitting.
URL: https://github.com/apache/beam/pull/10483
 
 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build

[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests



 [ 
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=364781=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364781
 ]

ASF GitHub Bot logged work on BEAM-8575:


Author: ASF GitHub Bot
Created on: 30/Dec/19 22:07
Start Date: 30/Dec/19 22:07
Worklog Time Spent: 10m 
  Work Description: liumomo315 commented on pull request #10447: 
[BEAM-8575] Refactor test_do_fn_with_windowing_in_finish_bundle to work with 
Dataflow runner
URL: https://github.com/apache/beam/pull/10447#discussion_r362107855
 
 

 ##
 File path: sdks/python/apache_beam/transforms/ptransform_test.py
 ##
 @@ -347,14 +347,14 @@ def finish_bundle(self):
 
 pipeline = TestPipeline()
 result = (pipeline
-  | 'Start' >> beam.Create([x for x in range(3)])
+  | 'Start' >> beam.Create([1])
   | beam.ParDo(MyDoFn())
   | WindowInto(windowfn)
   | 'create tuple' >> beam.Map(
   lambda v, t=beam.DoFn.TimestampParam, 
w=beam.DoFn.WindowParam:
   (v, t, w.start, w.end)))
-expected_process = [('process'+ str(x), Timestamp(5), Timestamp(4),
- Timestamp(6)) for x in range(3)]
+expected_process = [('process'+ str(1), Timestamp(5), Timestamp(4),
 
 Review comment:
   Sorry I missed this comment earlier. Committed the suggestion. Thanks!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364781)
Time Spent: 40h 20m  (was: 40h 10m)

> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
>  Time Spent: 40h 20m
>  Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to 
> improve test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-9039) Fix Datachannel stuckness on errors



 [ 
https://issues.apache.org/jira/browse/BEAM-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-9039.

Resolution: Fixed

> Fix Datachannel stuckness on errors
> ---
>
> Key: BEAM-9039
> URL: https://issues.apache.org/jira/browse/BEAM-9039
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Catch all task for any data channel stuckness issues, in particular if any 
> happen on errors. Re-open this issue if a new one is found.
> The last known one I have is a race condition on DataChannel.readErr 
> Close off a race condition where a closing DataChannel might have new readers 
> created for it while it is failing, causing stuckness in the bundles.
> In particular, the c.readErr must be interacted while c.mu is held.
>  Otherwise something like the following happens.
>  Given a channel C, and goroutines G1,G2.
>  # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
>  # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
> Readers are closed. The channel is officially closed with A.forceRecreate, 
> removing it from the DataManager cache.
>  # G1 calls A.makeReader, and creates a new reader there.
> There could be an arbitrary number of G1s.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364779=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364779
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:50
Start Date: 30/Dec/19 21:50
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362103280
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) &&

[jira] [Work logged] (BEAM-9039) Fix Datachannel stuckness on errors



 [ 
https://issues.apache.org/jira/browse/BEAM-9039?focusedWorklogId=364778=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364778
 ]

ASF GitHub Bot logged work on BEAM-9039:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:50
Start Date: 30/Dec/19 21:50
Worklog Time Spent: 10m 
  Work Description: lostluck commented on pull request #10456: [BEAM-9039] 
Fix race on reading channel readErr
URL: https://github.com/apache/beam/pull/10456
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364778)
Time Spent: 20m  (was: 10m)

> Fix Datachannel stuckness on errors
> ---
>
> Key: BEAM-9039
> URL: https://issues.apache.org/jira/browse/BEAM-9039
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Catch all task for any data channel stuckness issues, in particular if any 
> happen on errors. Re-open this issue if a new one is found.
> The last known one I have is a race condition on DataChannel.readErr 
> Close off a race condition where a closing DataChannel might have new readers 
> created for it while it is failing, causing stuckness in the bundles.
> In particular, the c.readErr must be interacted while c.mu is held.
>  Otherwise something like the following happens.
>  Given a channel C, and goroutines G1,G2.
>  # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
>  # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
> Readers are closed. The channel is officially closed with A.forceRecreate, 
> removing it from the DataManager cache.
>  # G1 calls A.makeReader, and creates a new reader there.
> There could be an arbitrary number of G1s.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9039) Fix Datachannel stuckness on errors



 [ 
https://issues.apache.org/jira/browse/BEAM-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke updated BEAM-9039:
---
Description: 
Catch all task for any data channel stuckness issues, in particular if any 
happen on errors. Re-open this issue if a new one is found.

The last known one I have is a race condition on DataChannel.readErr 

Close off a race condition where a closing DataChannel might have new readers 
created for it while it is failing, causing stuckness in the bundles.

In particular, the c.readErr must be interacted while c.mu is held.
 Otherwise something like the following happens.
 Given a channel C, and goroutines G1,G2.
 # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
 # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
Readers are closed. The channel is officially closed with A.forceRecreate, 
removing it from the DataManager cache.
 # G1 calls A.makeReader, and creates a new reader there.

There could be an arbitrary number of G1s.

 

  was:
Catch all task for any data channel stuckness issues, in particular if any 
happen on errors.

The last known one I have is a race condition on DataChannel.readErr 

Close off a race condition where a closing DataChannel might have new readers 
created for it while it is failing, causing stuckness in the bundles.

In particular, the c.readErr must be interacted while c.mu is held.
Otherwise something like the following happens.
Given a channel C, and goroutines G1,G2.
 # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
 # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
Readers are closed. The channel is officially closed with A.forceRecreate, 
removing it from the DataManager cache.
 # G1 calls A.makeReader, and creates a new reader there.

There could be an arbitrary number of G1s.

 


> Fix Datachannel stuckness on errors
> ---
>
> Key: BEAM-9039
> URL: https://issues.apache.org/jira/browse/BEAM-9039
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Catch all task for any data channel stuckness issues, in particular if any 
> happen on errors. Re-open this issue if a new one is found.
> The last known one I have is a race condition on DataChannel.readErr 
> Close off a race condition where a closing DataChannel might have new readers 
> created for it while it is failing, causing stuckness in the bundles.
> In particular, the c.readErr must be interacted while c.mu is held.
>  Otherwise something like the following happens.
>  Given a channel C, and goroutines G1,G2.
>  # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
>  # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
> Readers are closed. The channel is officially closed with A.forceRecreate, 
> removing it from the DataManager cache.
>  # G1 calls A.makeReader, and creates a new reader there.
> There could be an arbitrary number of G1s.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9039) Fix Datachannel stuckness on errors



 [ 
https://issues.apache.org/jira/browse/BEAM-9039?focusedWorklogId=364777=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364777
 ]

ASF GitHub Bot logged work on BEAM-9039:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:49
Start Date: 30/Dec/19 21:49
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #10456: [BEAM-9039] Fix 
race on reading channel readErr
URL: https://github.com/apache/beam/pull/10456#issuecomment-569806113
 
 
   Thanks! I filed one as a catch all for any further stuckness issues. It 
should be closed with this fix, but we can re-open if we find more.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364777)
Remaining Estimate: 0h
Time Spent: 10m

> Fix Datachannel stuckness on errors
> ---
>
> Key: BEAM-9039
> URL: https://issues.apache.org/jira/browse/BEAM-9039
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Catch all task for any data channel stuckness issues, in particular if any 
> happen on errors.
> The last known one I have is a race condition on DataChannel.readErr 
> Close off a race condition where a closing DataChannel might have new readers 
> created for it while it is failing, causing stuckness in the bundles.
> In particular, the c.readErr must be interacted while c.mu is held.
> Otherwise something like the following happens.
> Given a channel C, and goroutines G1,G2.
>  # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
>  # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
> Readers are closed. The channel is officially closed with A.forceRecreate, 
> removing it from the DataManager cache.
>  # G1 calls A.makeReader, and creates a new reader there.
> There could be an arbitrary number of G1s.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests



 [ 
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=364776=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364776
 ]

ASF GitHub Bot logged work on BEAM-8575:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:48
Start Date: 30/Dec/19 21:48
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10447: [BEAM-8575] 
Refactor test_do_fn_with_windowing_in_finish_bundle to work with Dataflow runner
URL: https://github.com/apache/beam/pull/10447#discussion_r362104383
 
 

 ##
 File path: sdks/python/apache_beam/transforms/ptransform_test.py
 ##
 @@ -347,14 +347,14 @@ def finish_bundle(self):
 
 pipeline = TestPipeline()
 result = (pipeline
-  | 'Start' >> beam.Create([x for x in range(3)])
+  | 'Start' >> beam.Create([1])
   | beam.ParDo(MyDoFn())
   | WindowInto(windowfn)
   | 'create tuple' >> beam.Map(
   lambda v, t=beam.DoFn.TimestampParam, 
w=beam.DoFn.WindowParam:
   (v, t, w.start, w.end)))
-expected_process = [('process'+ str(x), Timestamp(5), Timestamp(4),
- Timestamp(6)) for x in range(3)]
+expected_process = [('process'+ str(1), Timestamp(5), Timestamp(4),
 
 Review comment:
   it's not important I think. we can merge without.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364776)
Time Spent: 40h 10m  (was: 40h)

> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
>  Time Spent: 40h 10m
>  Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to 
> improve test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9039) Fix Datachannel stuckness on errors

Robert Burke created BEAM-9039:
--

 Summary: Fix Datachannel stuckness on errors
 Key: BEAM-9039
 URL: https://issues.apache.org/jira/browse/BEAM-9039
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-go
Affects Versions: Not applicable
Reporter: Robert Burke
Assignee: Robert Burke
 Fix For: Not applicable


Catch all task for any data channel stuckness issues, in particular if any 
happen on errors.

The last known one I have is a race condition on DataChannel.readErr 

Close off a race condition where a closing DataChannel might have new readers 
created for it while it is failing, causing stuckness in the bundles.

In particular, the c.readErr must be interacted while c.mu is held.
Otherwise something like the following happens.
Given a channel C, and goroutines G1,G2.
 # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
 # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
Readers are closed. The channel is officially closed with A.forceRecreate, 
removing it from the DataManager cache.
 # G1 calls A.makeReader, and creates a new reader there.

There could be an arbitrary number of G1s.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9039) Fix Datachannel stuckness on errors



 [ 
https://issues.apache.org/jira/browse/BEAM-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke updated BEAM-9039:
---
Status: Open  (was: Triage Needed)

> Fix Datachannel stuckness on errors
> ---
>
> Key: BEAM-9039
> URL: https://issues.apache.org/jira/browse/BEAM-9039
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>
> Catch all task for any data channel stuckness issues, in particular if any 
> happen on errors.
> The last known one I have is a race condition on DataChannel.readErr 
> Close off a race condition where a closing DataChannel might have new readers 
> created for it while it is failing, causing stuckness in the bundles.
> In particular, the c.readErr must be interacted while c.mu is held.
> Otherwise something like the following happens.
> Given a channel C, and goroutines G1,G2.
>  # G1 A request for a new reader on C arrives, checks C.readErr finds it null.
>  # G2 An error occurs on reading. The lock is acquired, and C.readErr is set. 
> Readers are closed. The channel is officially closed with A.forceRecreate, 
> removing it from the DataManager cache.
>  # G1 calls A.makeReader, and creates a new reader there.
> There could be an arbitrary number of G1s.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364773=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364773
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:45
Start Date: 30/Dec/19 21:45
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362103705
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) && 
connectedClient.get(workerId).isDone())

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364772
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:43
Start Date: 30/Dec/19 21:43
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362103280
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) &&

[jira] [Work logged] (BEAM-8575) Add more Python validates runner tests



 [ 
https://issues.apache.org/jira/browse/BEAM-8575?focusedWorklogId=364771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364771
 ]

ASF GitHub Bot logged work on BEAM-8575:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:41
Start Date: 30/Dec/19 21:41
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #10447: [BEAM-8575] 
Refactor test_do_fn_with_windowing_in_finish_bundle to work with Dataflow runner
URL: https://github.com/apache/beam/pull/10447#discussion_r362102838
 
 

 ##
 File path: sdks/python/apache_beam/transforms/ptransform_test.py
 ##
 @@ -347,14 +347,14 @@ def finish_bundle(self):
 
 pipeline = TestPipeline()
 result = (pipeline
-  | 'Start' >> beam.Create([x for x in range(3)])
+  | 'Start' >> beam.Create([1])
   | beam.ParDo(MyDoFn())
   | WindowInto(windowfn)
   | 'create tuple' >> beam.Map(
   lambda v, t=beam.DoFn.TimestampParam, 
w=beam.DoFn.WindowParam:
   (v, t, w.start, w.end)))
-expected_process = [('process'+ str(x), Timestamp(5), Timestamp(4),
- Timestamp(6)) for x in range(3)]
+expected_process = [('process'+ str(1), Timestamp(5), Timestamp(4),
 
 Review comment:
   @liumomo315 
   Shall I add this suggestion to the change?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364771)
Time Spent: 40h  (was: 39h 50m)

> Add more Python validates runner tests
> --
>
> Key: BEAM-8575
> URL: https://issues.apache.org/jira/browse/BEAM-8575
> Project: Beam
>  Issue Type: Test
>  Components: sdk-py-core, testing
>Reporter: wendy liu
>Assignee: wendy liu
>Priority: Major
>  Time Spent: 40h
>  Remaining Estimate: 0h
>
> This is the umbrella issue to track the work of adding more Python tests to 
> improve test coverage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364765=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364765
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:27
Start Date: 30/Dec/19 21:27
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10482: [BEAM-5600] 
Add unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482
 
 
   This creates an unsupported API to RemoteBundle and a default split handler 
that throws.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364767
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:27
Start Date: 30/Dec/19 21:27
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10482: [BEAM-5600] Add 
unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#issuecomment-569801244
 
 
   R: @boyuanzz @mxm 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364767)
Time Spent: 1h 40m  (was: 1.5h)

> Splitting for SplittableDoFn should be exposed within runner shared libraries
> -
>
> Key: BEAM-5600
> URL: https://issues.apache.org/jira/browse/BEAM-5600
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5600) Splitting for SplittableDoFn should be exposed within runner shared libraries



 [ 
https://issues.apache.org/jira/browse/BEAM-5600?focusedWorklogId=364766=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364766
 ]

ASF GitHub Bot logged work on BEAM-5600:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:27
Start Date: 30/Dec/19 21:27
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10482: [BEAM-5600] 
Add unimplemented split API to Runner side SDF libraries.
URL: https://github.com/apache/beam/pull/10482#discussion_r362100325
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/SdkHarnessClient.java
 ##
 @@ -231,122 +234,114 @@ public ActiveBundle newBundle(
   return fnApiDataService.receive(
   LogicalEndpoint.of(bundleId, ptransformId), receiver.getCoder(), 
receiver.getReceiver());
 }
-  }
 
-  /** An active bundle for a particular {@link 
BeamFnApi.ProcessBundleDescriptor}. */
-  public static class ActiveBundle implements RemoteBundle {
-private final String bundleId;
-private final CompletionStage response;
-private final Map inputReceivers;
-private final Map outputClients;
-private final StateDelegator.Registration stateRegistration;
-private final BundleProgressHandler progressHandler;
-private final BundleCheckpointHandler checkpointHandler;
-private final BundleFinalizationHandler finalizationHandler;
-
-private ActiveBundle(
-String bundleId,
-CompletionStage response,
-Map inputReceivers,
-Map outputClients,
-StateDelegator.Registration stateRegistration,
-BundleProgressHandler progressHandler,
-BundleCheckpointHandler checkpointHandler,
-BundleFinalizationHandler finalizationHandler) {
-  this.bundleId = bundleId;
-  this.response = response;
-  this.inputReceivers = inputReceivers;
-  this.outputClients = outputClients;
-  this.stateRegistration = stateRegistration;
-  this.progressHandler = progressHandler;
-  this.checkpointHandler = checkpointHandler;
-  this.finalizationHandler = finalizationHandler;
-}
+/** An active bundle for a particular {@link 
BeamFnApi.ProcessBundleDescriptor}. */
+public class ActiveBundle implements RemoteBundle {
+  private final String bundleId;
+  private final CompletionStage response;
+  private final Map inputReceivers;
+  private final Map outputClients;
+  private final StateDelegator.Registration stateRegistration;
+  private final BundleProgressHandler progressHandler;
+  private final BundleSplitHandler splitHandler;
+  private final BundleCheckpointHandler checkpointHandler;
+  private final BundleFinalizationHandler finalizationHandler;
+
+  private ActiveBundle(
+  String bundleId,
+  CompletionStage response,
+  Map inputReceivers,
+  Map outputClients,
+  StateDelegator.Registration stateRegistration,
+  BundleProgressHandler progressHandler,
+  BundleSplitHandler splitHandler,
+  BundleCheckpointHandler checkpointHandler,
+  BundleFinalizationHandler finalizationHandler) {
+this.bundleId = bundleId;
+this.response = response;
+this.inputReceivers = inputReceivers;
+this.outputClients = outputClients;
+this.stateRegistration = stateRegistration;
+this.progressHandler = progressHandler;
+this.splitHandler = splitHandler;
+this.checkpointHandler = checkpointHandler;
+this.finalizationHandler = finalizationHandler;
+  }
 
-/** Returns an id used to represent this bundle. */
-@Override
-public String getId() {
-  return bundleId;
-}
+  /** Returns an id used to represent this bundle. */
+  @Override
+  public String getId() {
+return bundleId;
+  }
 
-/**
- * Get a map of PCollection ids to {@link FnDataReceiver receiver}s which 
consume input
- * elements, forwarding them to the remote environment.
- */
-@Override
-public Map getInputReceivers() {
-  return (Map) inputReceivers;
-}
+  /**
+   * Get a map of PCollection ids to {@link FnDataReceiver receiver}s 
which consume input
+   * elements, forwarding them to the remote environment.
+   */
+  @Override
+  public Map getInputReceivers() {
+return (Map) inputReceivers;
+  }
 
-/**
- * Blocks until bundle processing is finished. This is comprised of:
- *
- * 
- *   closing each {@link #getInputReceivers() input receiver}.
- *   waiting for the SDK to say that processing the bundle is finished.
- *   waiting for all inbound data clients to complete
- * 
- *
- * This method will throw an exception if bundle processing has failed. 
{@link
- *

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364757=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364757
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 30/Dec/19 21:05
Start Date: 30/Dec/19 21:05
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10458: [BEAM-7951]  
Improve the docs for `beam_runner_api.proto` and `WindowedValue.java`
URL: https://github.com/apache/beam/pull/10458#issuecomment-569796388
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364757)
Time Spent: 10h 40m  (was: 10.5h)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364748=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364748
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784715
 
 
   Run SQL Postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364748)
Time Spent: 2h  (was: 1h 50m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364744=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364744
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784606
 
 
   Run Java HadoopFormatIO Performance Test
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364744)
Time Spent: 1h 20m  (was: 1h 10m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364743=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364743
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784591
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364743)
Time Spent: 1h 10m  (was: 1h)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364747=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364747
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784694
 
 
   Run Spark ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364747)
Time Spent: 1h 50m  (was: 1h 40m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364745
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784641
 
 
   Run BigQueryIO Streaming Performance Test Java
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364745)
Time Spent: 1.5h  (was: 1h 20m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364746
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:17
Start Date: 30/Dec/19 20:17
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#issuecomment-569784675
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364746)
Time Spent: 1h 40m  (was: 1.5h)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364741=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364741
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:16
Start Date: 30/Dec/19 20:16
Worklog Time Spent: 10m 
  Work Description: suztomo commented on pull request #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#discussion_r362085410
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
 ##
 @@ -252,11 +253,11 @@ public void testLengthPrefixAndReplaceForRunnerNetwork() 
{
   }
 }
 
-Set expectedInstructions =
+ImmutableSet expectedInstructions =
 ImmutableSet.of(
-prefixedReadNode.getParallelInstruction(), 
prefixedReadNodeOut.getInstructionOutput());
+prefixedReadNodeOut.getInstructionOutput(), 
prefixedReadNode.getParallelInstruction());
 
 Review comment:
   Added `assertThat(..., containsInAnyOrder(...))` by introducing a new 
matcher.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364741)
Time Spent: 50m  (was: 40m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364742=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364742
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:16
Start Date: 30/Dec/19 20:16
Worklog Time Spent: 10m 
  Work Description: suztomo commented on pull request #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#discussion_r362083552
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
 ##
 @@ -252,11 +255,11 @@ public void testLengthPrefixAndReplaceForRunnerNetwork() 
{
   }
 }
 
-Set expectedInstructions =
-ImmutableSet.of(
-prefixedReadNode.getParallelInstruction(), 
prefixedReadNodeOut.getInstructionOutput());
-
-assertEquals(expectedInstructions, prefixedInstructions);
+assertThat(
+prefixedInstructions.build(),
+containsInAnyOrder(
+jsonOf(prefixedReadNodeOut.getInstructionOutput()),
+jsonOf(prefixedReadNode.getParallelInstruction(;
 
 Review comment:
   Created `GenericJsonMatcher.jsonOf`, to leverage Hamcrest's 
`containsInAnyOrder`.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364742)
Time Spent: 1h  (was: 50m)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8956) Unify Contributor Docs



 [ 
https://issues.apache.org/jira/browse/BEAM-8956?focusedWorklogId=364738=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364738
 ]

ASF GitHub Bot logged work on BEAM-8956:


Author: ASF GitHub Bot
Created on: 30/Dec/19 20:01
Start Date: 30/Dec/19 20:01
Worklog Time Spent: 10m 
  Work Description: suztomo commented on pull request #10366: [BEAM-8956] 
Begin unifying contributor instructions into a single location
URL: https://github.com/apache/beam/pull/10366#discussion_r362082634
 
 

 ##
 File path: website/src/contribute/index.md
 ##
 @@ -89,6 +89,13 @@ sudo apt-get install \
 
 You also need to [install Go](https://golang.org/doc/install]).
 
+Once Go is installed, install goavro:
+
+```
+$ export GOPATH=`pwd`/sdks/go/examples/.gogradle/project_gopath
+$ go get github.com/linkedin/goavro
 
 Review comment:
   Thank you for picking up my comment.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364738)
Time Spent: 2h 10m  (was: 2h)

> Unify Contributor Docs
> --
>
> Key: BEAM-8956
> URL: https://issues.apache.org/jira/browse/BEAM-8956
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.18.0
>Reporter: elharo
>Assignee: elharo
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Right now we have overlapping and sometimes contradictory docs on how to 
> setup and build BEAM in four different places I've found:
>  
>  README.md
>  CONTRIBUTING.md
>  [https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ]
>  [https://beam.apache.org/contribute/]
>  
>  We should probably pick one as the source of truth and rewrite the
>  other three to simply point to it. I propose putting all checkout,
>  build, test, commit, and push instructions in CONTRIBUTING.md in the
>  repo.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-7468) Add Python global window side input pattern

2019-12-30 Thread Monika Kovacs (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005796#comment-17005796
 ] 

Monika Kovacs edited comment on BEAM-7468 at 12/30/19 7:59 PM:
---

h4. 7.3.4. Single global window

If your {{PCollection}} is bounded (the size is fixed), you can assign all the 
elements to a single global window. The following example code shows how to set 
a single global window for a {{PCollection}}:
{quote}{{from apache_beam import windowsession_windowed_items = (items | 
'window' >> beam.WindowInto(window.GlobalWindows()))}}
{quote}


was (Author: monika101):
h4. 7.3.4. Single global window

If your {{PCollection}} is bounded (the size is fixed), you can assign all the 
elements to a single global window. The following example code shows how to set 
a single global window for a {{PCollection}}:

{{from apache_beam import windowsession_windowed_items = (items | 'window' >> 
beam.WindowInto(window.GlobalWindows()))}}

> Add Python global window side input pattern
> ---
>
> Key: BEAM-7468
> URL: https://issues.apache.org/jira/browse/BEAM-7468
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Cyrus Maden
>Assignee: Reza ardeshir rokni
>Priority: Minor
>
> Tracker for documenting a Python version of the side input pattern "Using 
> global window side inputs in non-global windows"
> [https://beam.apache.org|https://beam.apache.org/documentation/pipelines/design-your-pipeline/]
>  
> [/documentation/patterns/side-input-patterns/#using-global-window-side-inputs-in-non-global-windows|http://localhost:4000/documentation/patterns/side-input-patterns/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-7468) Add Python global window side input pattern

2019-12-30 Thread Monika Kovacs (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-7468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005796#comment-17005796
 ] 

Monika Kovacs commented on BEAM-7468:
-

h4. 7.3.4. Single global window

If your {{PCollection}} is bounded (the size is fixed), you can assign all the 
elements to a single global window. The following example code shows how to set 
a single global window for a {{PCollection}}:

{{from apache_beam import windowsession_windowed_items = (items | 'window' >> 
beam.WindowInto(window.GlobalWindows()))}}

> Add Python global window side input pattern
> ---
>
> Key: BEAM-7468
> URL: https://issues.apache.org/jira/browse/BEAM-7468
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Cyrus Maden
>Assignee: Reza ardeshir rokni
>Priority: Minor
>
> Tracker for documenting a Python version of the side input pattern "Using 
> global window side inputs in non-global windows"
> [https://beam.apache.org|https://beam.apache.org/documentation/pipelines/design-your-pipeline/]
>  
> [/documentation/patterns/side-input-patterns/#using-global-window-side-inputs-in-non-global-windows|http://localhost:4000/documentation/patterns/side-input-patterns/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9038) Documentation mistake on 7.3.4. Single Global window, mistakenly refers to Session windows python code

2019-12-30 Thread Monika Kovacs (Jira)

Monika Kovacs created BEAM-9038:
---

 Summary: Documentation mistake on 7.3.4. Single Global window, 
mistakenly refers to Session windows python code
 Key: BEAM-9038
 URL: https://issues.apache.org/jira/browse/BEAM-9038
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Monika Kovacs






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9034) Update environment_id for ExternalTransform in Python SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-9034?focusedWorklogId=364735=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364735
 ]

ASF GitHub Bot logged work on BEAM-9034:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:50
Start Date: 30/Dec/19 19:50
Worklog Time Spent: 10m 
  Work Description: ihji commented on issue #10469: [BEAM-9034] Update 
environment_id for ExternalTransform in Python SDK
URL: https://github.com/apache/beam/pull/10469#issuecomment-56912
 
 
   CC: @chamikaramj 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364735)
Time Spent: 0.5h  (was: 20m)

> Update environment_id for ExternalTransform in Python SDK
> -
>
> Key: BEAM-9034
> URL: https://issues.apache.org/jira/browse/BEAM-9034
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> ExternalTransform in  Python SDK also needs environment_id field.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364734=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364734
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:49
Start Date: 30/Dec/19 19:49
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569764504
 
 
   Retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364734)
Time Spent: 3h  (was: 2h 50m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9012) Include `-> None` on Pipeline and PipelineOptions `init` methods for pytype compatibility



 [ 
https://issues.apache.org/jira/browse/BEAM-9012?focusedWorklogId=364731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364731
 ]

ASF GitHub Bot logged work on BEAM-9012:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:44
Start Date: 30/Dec/19 19:44
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #10466: [BEAM-9012] 
Change __init__ hints so they work with pytype
URL: https://github.com/apache/beam/pull/10466
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364731)
Time Spent: 1h 10m  (was: 1h)

> Include `-> None` on Pipeline and PipelineOptions `__init__` methods for 
> pytype compatibility
> -
>
> Key: BEAM-9012
> URL: https://issues.apache.org/jira/browse/BEAM-9012
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> mypy [made a decision|https://github.com/python/mypy/issues/604] to allow 
> init methods to omit {{\-> None}} return type annotations, but pytype has no 
> such feature. I think we should include {{\-> None}} annotations for pytype 
> compatibility.
> cc: [~chadrik]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-7746) Add type hints to python code

2019-12-30 Thread Udi Meiri (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-7746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005782#comment-17005782
 ] 

Udi Meiri commented on BEAM-7746:
-

WDYT about adding a pytype (https://github.com/google/pytype) tox target as 
well?
This will help maintain type hint compatibility with Google-internal tests.

> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 36h 10m
>  Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484 
> type hints so that I can clearly see what types are required, get completion 
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here:  [https://github.com/apache/beam/pull/9056]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364726=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364726
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:34
Start Date: 30/Dec/19 19:34
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362076484
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) && 
connectedClient.get(workerId).isDone())

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364727=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364727
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:34
Start Date: 30/Dec/19 19:34
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362076554
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
+   * @param requestObserver The outbound request observer this client uses to 
send new status
+   * requests to its corresponding SDK Harness.
+   * @return {@link WorkerStatusClient}
+   */
+  public static WorkerStatusClient forRequestObserver(
+  String workerId, StreamObserver requestObserver) {
+return new WorkerStatusClient(workerId, requestObserver);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness. A random id will
+   * be used to specify the request_id field.
+   *
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  public CompletableFuture getWorkerStatus() {
+WorkerStatusRequest request =
+WorkerStatusRequest.newBuilder().setId(idGenerator.getId()).build();
+return getWorkerStatus(request);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness with request.
+   *
+   * @param request WorkerStatusRequest to be sent to SDK Harness.
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  CompletableFuture getWorkerStatus(WorkerStatusRequest 
request) {
+CompletableFuture future = new CompletableFuture<>();
+this.responseQueue.put(request.getId(), future);
+this.requestReceiver.onNext(request);
+return future;
+  }
+
+  @Override
+  public void close() throws IOException {
+if (isClosed.getAndSet(true)) {
+  return;
+}
+for (CompletableFuture pendingResponse : 
responseQueue.values()) {
+  pendingResponse.completeExceptionally(
+  new RuntimeException("Fn Status Api client shut down while waiting 
for the request"));
+}
+responseQueue.clear();
+requestReceiver.onCompleted();
+  }
+
+  /** Check if the client

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364724
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:30
Start Date: 30/Dec/19 19:30
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362075574
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) && 
connectedClient.get(workerId).isDone())

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364725
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 19:30
Start Date: 30/Dec/19 19:30
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362075666
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) && 
connectedClient.get(workerId).isDone())

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364717=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364717
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 18:58
Start Date: 30/Dec/19 18:58
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569764504
 
 
   Retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364717)
Time Spent: 2h 50m  (was: 2h 40m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364716=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364716
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 18:56
Start Date: 30/Dec/19 18:56
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569749330
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364716)
Time Spent: 2h 40m  (was: 2.5h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7949) Add time-based cache threshold support in the data service of the Python SDK harness



 [ 
https://issues.apache.org/jira/browse/BEAM-7949?focusedWorklogId=364700=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364700
 ]

ASF GitHub Bot logged work on BEAM-7949:


Author: ASF GitHub Bot
Created on: 30/Dec/19 18:08
Start Date: 30/Dec/19 18:08
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10246: [BEAM-7949] 
Add time-based cache threshold support in the data service of the Python SDK 
harness
URL: https://github.com/apache/beam/pull/10246#discussion_r362056607
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py
 ##
 @@ -200,6 +203,29 @@ def _get_state_cache_size(pipeline_options):
   return 0
 
 
+def _get_data_buffer_time_limit_ms(pipeline_options):
+  """Defines the time limt of the outbound data buffering.
+
+  Note: data_buffer_time_limit_ms is an experimental flag and might
+  not be available in future releases.
+
+  Returns:
+an int indicating the time limit in milliseconds of the the outbound
+  data buffering. Default is 0 (disabled)
+  """
+  experiments = pipeline_options.view_as(DebugOptions).experiments
+  experiments = experiments if experiments else []
+
+  for experiment in experiments:
+# There should only be 1 match so returning from the loop
+if re.match(r'data_buffer_time_limit_ms=', experiment):
+  return int(
+  re.match(
+  r'data_buffer_time_limit_ms=(?P.*)',
 
 Review comment:
   Should we be making sure that the experiments have the same string constant 
across languages 
([beam_fn_api_data_buffer_time_limit](https://github.com/apache/beam/blob/4c18cb4ada2650552a0006dfffd68d0775dd76c6/sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/BeamFnDataBufferingOutboundObserver.java#L39))?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364700)
Time Spent: 3h 50m  (was: 3h 40m)

> Add time-based cache threshold support in the data service of the Python SDK 
> harness
> 
>
> Key: BEAM-7949
> URL: https://issues.apache.org/jira/browse/BEAM-7949
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> Currently only size-based cache threshold is supported in the data service of 
> Python SDK harness. It should also support the time-based cache threshold. 
> This is very important, especially for streaming jobs which are sensitive to 
> the delay. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364699
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 30/Dec/19 18:03
Start Date: 30/Dec/19 18:03
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10458: [BEAM-7951]  
Improve the docs for `beam_runner_api.proto` and `WindowedValue.java`
URL: https://github.com/apache/beam/pull/10458#issuecomment-569750414
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364699)
Time Spent: 10.5h  (was: 10h 20m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder



 [ 
https://issues.apache.org/jira/browse/BEAM-7951?focusedWorklogId=364698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364698
 ]

ASF GitHub Bot logged work on BEAM-7951:


Author: ASF GitHub Bot
Created on: 30/Dec/19 18:01
Start Date: 30/Dec/19 18:01
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10464: [BEAM-7951] 
Supports multiple inputs/outputs for wire coder settings.
URL: https://github.com/apache/beam/pull/10464#discussion_r362055043
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -1362,4 +1347,27 @@ message ExecutableStagePayload {
 // (Required) The local name of this timer for the PTransform that 
references it.
 string local_name = 2;
   }
+
+  // Settings that decide the coder type of wire coder.
+  message WireCoderSetting {
+// (Required) The URN of the wire coder.
+// Note that only windowed value coder or parameterized windowed value 
coder are supported.
+string urn = 1;
+
+// (Optional) The data specifying any parameters to the URN. If
+// the URN is beam:coder:windowed_value:v1, this may be omitted. If the 
URN is
+// beam:coder:param_windowed_value:v1, the payload is an encoded windowed
+// value using the beam:coder:windowed_value:v1 coder parameterized by
+// a beam:coder:bytes:v1 element coder and the window coder that this
+// param_windowed_value coder uses.
+bytes payload = 2;
+
+// (Required) The target(PCollection or Timer) this setting applies to.
+oneof target {
 
 Review comment:
   We could add WireCoderSetting to the TimerId message explicitly instead of 
having target here.
   
   We could also add  `input_wire_coder_setting` and make `outputs` a 
`map`.
   
   WDYT?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364698)
Time Spent: 10h 20m  (was: 10h 10m)

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364697
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:58
Start Date: 30/Dec/19 17:58
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569749330
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364697)
Time Spent: 2.5h  (was: 2h 20m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9020) LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality



 [ 
https://issues.apache.org/jira/browse/BEAM-9020?focusedWorklogId=364693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364693
 ]

ASF GitHub Bot logged work on BEAM-9020:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:49
Start Date: 30/Dec/19 17:49
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10467: [BEAM-9020] 
LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
URL: https://github.com/apache/beam/pull/10467#discussion_r362051797
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/graph/LengthPrefixUnknownCodersTest.java
 ##
 @@ -252,11 +253,11 @@ public void testLengthPrefixAndReplaceForRunnerNetwork() 
{
   }
 }
 
-Set expectedInstructions =
+ImmutableSet expectedInstructions =
 ImmutableSet.of(
-prefixedReadNode.getParallelInstruction(), 
prefixedReadNodeOut.getInstructionOutput());
+prefixedReadNodeOut.getInstructionOutput(), 
prefixedReadNode.getParallelInstruction());
 
 Review comment:
   The intent isn't to have a set in JSON but to match the expected vs the 
received which is why set was being used instead of a list so this test was 
attempting to do something like `assertThat(..., contains(...))`.
   
   Please update this so that we don't care about the order of the nodes when 
matching.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364693)
Time Spent: 40m  (was: 0.5h)

> LengthPrefixUnknownCodersTest to avoid relying on AbstractMap's equality
> 
>
> Key: BEAM-9020
> URL: https://issues.apache.org/jira/browse/BEAM-9020
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Tomo Suzuki
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In an attempt for BEAM-8695 LengthPrefixUnknownCodersTest failed when trying 
> to upgrade google-http-client v1.34.0, because LengthPrefixUnknownCodersTest 
> relies on the equality of {{CloudObject}} with Map.
> Class hierarchy:
> {noformat}
> CloudObject < GenericJson < GenericData < AbstractMap
> {noformat}
> It was working fine as long as CloudObject's equality inherits 
> AbstractMap.equality. {{GenericData}} did not override equals method in 
> google-http-client v1.28.0 and earlier. The comparison was checking only key 
> and value of a Map.
> {code:java}
> assertEquals(
> CloudObjects.asCloudObject(prefixedWindowedValueCoder, null),   // This 
> is a CloudObject
> lengthPrefixedCoderCloudObject); // This is a 
> Map
> {code}
> However, with google-http-client v1.29.0 or higher, GenericData has its own 
> {{equals}} method 
> ([PR#589|https://github.com/googleapis/google-http-java-client/pull/589])  
> that checks {{classInfo}} and thus the comparisons between a CloudObject and 
> a Map always fail.
> Test failures when I tried to upgrade google-http-client 1.34.0 
> ([Jenkins|https://builds.apache.org/job/beam_PreCommit_Java_Commit/9288/#showFailuresLink]):
> {noformat}
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodeSuccessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForLengthPrefixCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForSideInputInfos
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixParDoInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixInstructionOutputCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixWriteInstructionCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceUnknownCoder
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixAndReplaceForRunnerNetwork
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixForInstructionOutputNodeWithGrpcNodePredecessor
> org.apache.beam.runners.dataflow.worker.graph.LengthPrefixUnknownCodersTest.testLengthPrefixReadInstructionCoder
>

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner



 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=364692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364692
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:48
Start Date: 30/Dec/19 17:48
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r362052126
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+if (connectedClient.containsKey(workerId) && 
connectedClient.get(workerId).isDone())

[jira] [Resolved] (BEAM-8195) Quota exceeded for create requests

2019-12-30 Thread Yifan Zou (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yifan Zou resolved BEAM-8195.
-
Fix Version/s: Not applicable
   Resolution: Fixed

> Quota exceeded for create requests
> --
>
> Key: BEAM-8195
> URL: https://issues.apache.org/jira/browse/BEAM-8195
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core, testing
>Reporter: Ahmet Altay
>Assignee: Yifan Zou
>Priority: Critical
> Fix For: Not applicable
>
>
> Post commits failied with the following error:
> HttpError accessing 
> :
>  response: <{'server': 'ESF', '-content-encoding': 'gzip', 'content-type': 
> 'application/json; charset=UTF-8', 'content-length': '598', 
> 'transfer-encoding': 'chunked', 'cache-control': 'private', 
> 'x-xss-protection': '0', 'date': 'Tue, 10 Sep 2019 12:02:24 GMT', 'vary': 
> 'Origin, X-Origin, Referer', 'x-frame-options': 'SAMEORIGIN', 'status': 
> '429', 'x-content-type-options': 'nosniff'}>, content <{
>   "error": {
> "code": 429,
> "message": "Quota exceeded for quota metric 
> 'dataflow.googleapis.com/create_requests' and limit 
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
> consumer 'project_number:844138762903'.",
> "status": "RESOURCE_EXHAUSTED",
> "details": [
>   {
> Could we increase the quota?
> /cc [~alanmyrvold] [~kenn]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8798) Contributing docs omit many steps

2019-12-30 Thread Tomo Suzuki (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-8798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005479#comment-17005479
 ] 

Tomo Suzuki commented on BEAM-8798:
---

The goVet problem is solved by :


{noformat}
$ go version # I think the version should be >= 1.12
go version go1.13.4 linux/amd64
$ export GOPATH=`pwd`/sdks/go/examples/.gogradle/project_gopath
$ go get github.com/linkedin/goavro # Why isn't this automatically installed?
$ ls $GOPATH/src/github.com
golang  linkedin
$ ./gradlew :sdks:go:examples:goVet --stacktrace
...
BUILD SUCCESSFUL in 52s
10 actionable tasks: 10 executed

{noformat}


> Contributing docs omit many steps
> -
>
> Key: BEAM-8798
> URL: https://issues.apache.org/jira/browse/BEAM-8798
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.16.0
>Reporter: Elliotte Rusty Harold
>Priority: Major
>
> So far I've noticed the following packages and tools that are not available 
> by default on a Debian system, need to be installed to build and test beam, 
> and are not yet mentioned in all our docs:
> * tox
> * Go
> * Python (which versions?)
> * Docker
> There might be more. The docs such as README.md should be updated to include 
> these. This covers README.md, CONTRIBUTING.md, 
> https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ, and 
> https://beam.apache.org/contribute/
> That we have four such docs is itself an issue. We should probably 
> consolidate these. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364690
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:08
Start Date: 30/Dec/19 17:08
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569736381
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364690)
Time Spent: 2h 20m  (was: 2h 10m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364689
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:08
Start Date: 30/Dec/19 17:08
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569732483
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364689)
Time Spent: 2h 10m  (was: 2h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364688
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:08
Start Date: 30/Dec/19 17:08
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569736381
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364688)
Time Spent: 2h  (was: 1h 50m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9030) Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)



 [ 
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=364687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364687
 ]

ASF GitHub Bot logged work on BEAM-9030:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:07
Start Date: 30/Dec/19 17:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10463: [BEAM-9030] 
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r362039967
 
 

 ##
 File path: buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring.groovy
 ##
 @@ -28,23 +28,44 @@ class GrpcVendoring {
   static List dependencies() {
 return [
   'com.google.guava:guava:26.0-jre',
-  'com.google.protobuf:protobuf-java:3.7.1',
-  'com.google.protobuf:protobuf-java-util:3.7.1',
-  'com.google.code.gson:gson:2.7',
-  'io.grpc:grpc-auth:1.21.0',
-  'io.grpc:grpc-core:1.21.0',
-  'io.grpc:grpc-context:1.21.0',
-  'io.grpc:grpc-netty:1.21.0',
-  'io.grpc:grpc-protobuf:1.21.0',
-  'io.grpc:grpc-stub:1.21.0',
-  'io.netty:netty-transport-native-epoll:4.1.34.Final',
+  'com.google.protobuf:protobuf-java:3.11.0',
+  'com.google.protobuf:protobuf-java-util:3.11.0',
+  'com.google.code.gson:gson:2.8.6',
+  'io.grpc:grpc-auth:1.26.0',
+  'io.grpc:grpc-core:1.26.0',
+  'io.grpc:grpc-context:1.26.0',
+  'io.grpc:grpc-netty:1.26.0',
+  'io.grpc:grpc-protobuf:1.26.0',
+  'io.grpc:grpc-stub:1.26.0',
+  'io.netty:netty-transport-native-epoll:4.1.42.Final',
   // tcnative version from 
https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
-  'io.netty:netty-tcnative-boringssl-static:2.0.22.Final',
-  'com.google.auth:google-auth-library-credentials:0.13.0',
-  'io.grpc:grpc-testing:1.21.0',
+  'io.netty:netty-tcnative-boringssl-static:2.0.26.Final',
+  'com.google.auth:google-auth-library-credentials:0.18.0',
+  'io.grpc:grpc-testing:1.26.0',
   'com.google.api.grpc:proto-google-common-protos:1.12.0',
-  'io.opencensus:opencensus-api:0.21.0',
-  'io.opencensus:opencensus-contrib-grpc-metrics:0.21.0',
+  'io.opencensus:opencensus-api:0.24.0',
+  'io.opencensus:opencensus-contrib-grpc-metrics:0.24.0',
+  'io.perfmark:perfmark-api:0.19.0',
+  'avalon-framework:avalon-framework:4.1.5',
+  'com.github.jponge:lzma-java:1.3',
+  'com.google.protobuf.nano:protobuf-javanano:3.0.0-alpha-5',
+  'com.jcraft:jzlib:1.1.3',
+  'com.ning:compress-lzf:1.0.3',
+  'commons-logging:commons-logging:1.2',
+  'javax.jms:jms-api:1.1-rev-1',
+  'javax.mail:mail:1.4.3',
+  'javax.servlet:servlet-api:2.5',
+  'log4j:log4j:1.2.17',
 
 Review comment:
   We should make these runtime dependencies of the vendored artifact (instead 
of shading/relocating) so that logging still works:
   * log4j:log4j:1.2.17
   * org.apache.logging.log4j:log4j-api:2.6.2
   * commons-logging:commons-logging:1.2
   * logkit:logkit:1.0.1
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364687)
Time Spent: 3.5h  (was: 3h 20m)

> Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
>  Issue Type: Bug
>  Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster 
> repeatedly, the meta space usage of the task manager of the Flink cluster 
> will continuously increase (about 40MB each time). The reason is that the 
> Beam classes are loaded with the user class loader in Flink and there are 
> problems with the implementation of `ProcessManager`(from Beam) and 
> `ThreadPoolCache`(from netty) which may cause the user class loader could not 
> be garbage collected even after the job finished which causes the meta space 
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track 
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed 
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC 
> 1.22.0 has already

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364686
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 17:06
Start Date: 30/Dec/19 17:06
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569696004
 
 
   run dataflow validatesrunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364686)
Time Spent: 1h 50m  (was: 1h 40m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=364685=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364685
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 30/Dec/19 16:54
Start Date: 30/Dec/19 16:54
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on pull request #9779: [BEAM-8388] 
Updating avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364685)
Remaining Estimate: 20h 40m  (was: 20h 50m)
Time Spent: 3h 20m  (was: 3h 10m)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 3h 20m
>  Remaining Estimate: 20h 40m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2



 [ 
https://issues.apache.org/jira/browse/BEAM-8388?focusedWorklogId=364684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364684
 ]

ASF GitHub Bot logged work on BEAM-8388:


Author: ASF GitHub Bot
Created on: 30/Dec/19 16:54
Start Date: 30/Dec/19 16:54
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #9779: [BEAM-8388] 
Updating avro dependency from 1.8.2 to 1.9.1
URL: https://github.com/apache/beam/pull/9779#issuecomment-569732532
 
 
   This pull request has been closed due to lack of activity. If you think that 
is incorrect, or the pull request requires review, you can revive the PR at any 
time.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364684)
Remaining Estimate: 20h 50m  (was: 21h)
Time Spent: 3h 10m  (was: 3h)

> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
>   Original Estimate: 24h
>  Time Spent: 3h 10m
>  Remaining Estimate: 20h 50m
>
> Update build dependency to 1.9.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364683
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 16:53
Start Date: 30/Dec/19 16:53
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569732483
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364683)
Time Spent: 1h 40m  (was: 1.5h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9030) Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)



 [ 
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=364677=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364677
 ]

ASF GitHub Bot logged work on BEAM-9030:


Author: ASF GitHub Bot
Created on: 30/Dec/19 16:35
Start Date: 30/Dec/19 16:35
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10463: [BEAM-9030] 
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r362034347
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoringOld.groovy
 ##
 @@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.gradle
+
+import org.gradle.api.Project
+
+/**
+ * Utilities for working with our vendored version of gRPC.
+ */
+class GrpcVendoringOld {
 
 Review comment:
   You could add some helpful links to `vendor/README.md`.
   
   Also, now I remember/understand why `GrpcVendoringOld` needs to exist (since 
the proto generated code needs to be updated to use the vendored version)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364677)
Time Spent: 3h 20m  (was: 3h 10m)

> Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
>  Issue Type: Bug
>  Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster 
> repeatedly, the meta space usage of the task manager of the Flink cluster 
> will continuously increase (about 40MB each time). The reason is that the 
> Beam classes are loaded with the user class loader in Flink and there are 
> problems with the implementation of `ProcessManager`(from Beam) and 
> `ThreadPoolCache`(from netty) which may cause the user class loader could not 
> be garbage collected even after the job finished which causes the meta space 
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track 
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed 
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC 
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the 
> version of GRPC to 1.22.0+ (currently 1.21.0).
>  
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9030) Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)



 [ 
https://issues.apache.org/jira/browse/BEAM-9030?focusedWorklogId=364676=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364676
 ]

ASF GitHub Bot logged work on BEAM-9030:


Author: ASF GitHub Bot
Created on: 30/Dec/19 16:31
Start Date: 30/Dec/19 16:31
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10463: [BEAM-9030] 
Bump grpc to 1.26.0
URL: https://github.com/apache/beam/pull/10463#discussion_r362033421
 
 

 ##
 File path: 
buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoringOld.groovy
 ##
 @@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.gradle
+
+import org.gradle.api.Project
+
+/**
+ * Utilities for working with our vendored version of gRPC.
+ */
+class GrpcVendoringOld {
 
 Review comment:
   There is a release guide for vendored artifacts here: 
https://s.apache.org/beam-release-vendored-artifacts (would greatly appreciate 
it if someone could add it to the beam site since it is the remaining work on 
https://issues.apache.org/jira/browse/BEAM-6438).
   
   Also, the linkage tool is useful for all dependency upgrades which is useful 
for more then just when creating vendored dependencies so documentation should 
go in some shared location.
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364676)
Time Spent: 3h 10m  (was: 3h)

> Bump the version of GRPC to 1.22.0+(May be latest 1.26.0, currently 1.21.0)
> ---
>
> Key: BEAM-9030
> URL: https://issues.apache.org/jira/browse/BEAM-9030
> Project: Beam
>  Issue Type: Bug
>  Components: java-fn-execution, runner-flink
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> When submitting a Python word count job to a Flink session/standalone cluster 
> repeatedly, the meta space usage of the task manager of the Flink cluster 
> will continuously increase (about 40MB each time). The reason is that the 
> Beam classes are loaded with the user class loader in Flink and there are 
> problems with the implementation of `ProcessManager`(from Beam) and 
> `ThreadPoolCache`(from netty) which may cause the user class loader could not 
> be garbage collected even after the job finished which causes the meta space 
> memory leak eventually. You can refer to FLINK-15338[1] for more information.
> Regarding to `ProcessManager`, I have created a JIRA BEAM-9006[2] to track 
> it. Regarding to `ThreadPoolCache`, it is a Netty problem and has been fixed 
> in NETTY#8955[3]. Netty 4.1.35 Final has already included this fix and GRPC 
> 1.22.0 has already dependents on Netty 4.1.35 Final. So we need to bump the 
> version of GRPC to 1.22.0+ (currently 1.21.0).
>  
> What do you think?
> [1] https://issues.apache.org/jira/browse/FLINK-15338
> [2] https://issues.apache.org/jira/browse/BEAM-9006
> [3] [https://github.com/netty/netty/pull/8955]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers



 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=364651=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364651
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 30/Dec/19 14:38
Start Date: 30/Dec/19 14:38
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10316: [BEAM-6857] 
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#issuecomment-569696004
 
 
   run dataflow validatesrunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 364651)
Time Spent: 1.5h  (was: 1h 20m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:

[
https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8002:

Description:

- 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-16 12:03:47.037326 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-23 12:04:12.360299 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-30 12:05:19.680040 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-07 12:04:48.065365 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-14 12:04:37.533903 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-21 12:05:15.993810 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-10-28 12:07:10.976374 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1 -> 5.6.3. The latest version is 6.0-rc-1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-04 12:05:57.793513 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 5.6.4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-11 12:05:24.076889 -

Please consider upgrading the dependency Gradle:.

The current version is 5.2.1. The latest version is 6.0

cc:
Please refer to [Beam

[jira] [Updated] (BEAM-8779) Beam Dependency Update Request: software.amazon.awssdk:sns



 [ 
https://issues.apache.org/jira/browse/BEAM-8779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8779:

Description: 


 - 2019-11-19 21:16:49.149055 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:24:08.775701 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:46.101446 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:33.501651 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:19:09.662186 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.41 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-11-19 21:16:49.149055 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:24:08.775701 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:46.101446 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:33.501651 -

Please consider upgrading the dependency software.amazon.awssdk:sns. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: software.amazon.awssdk:sns
> --
>
> Key: BEAM-8779
> URL: https://issues.apache.org/jira/browse/BEAM-8779
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-11-19 21:16:49.149055 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sns. 
> The current version is 2.5.71. The latest version is 2.10.19 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:24:08.775701 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sns. 
> The current version is 2.5.71. The latest version is 2.10.26 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The

[jira] [Updated] (BEAM-6901) Beam Dependency Update Request: sqlline:sqlline



 [ 
https://issues.apache.org/jira/browse/BEAM-6901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-6901:

Description: 


 - 2019-03-25 04:23:12.026437 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-01 12:15:11.859956 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-08 12:15:43.829522 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-15 12:35:59.407782 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-04-22 12:13:31.922817 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.7.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-12 22:55:03.099214 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-12 23:32:43.232674 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-15 19:48:15.961831 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-18 12:13:32.145836 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-18 20:50:42.825702 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 20:08:43.049616 -

Please consider upgrading the dependency sqlline:sqlline. 

The current version is 1.4.0. The latest version is 1.9.0 

cc: [~kenn], [~kedin], [~apilloud], [~amaliujia], 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-11-19 21:16:52.815933

[jira] [Updated] (BEAM-8857) Beam Dependency Update Request: software.amazon.awssdk:sqs



 [ 
https://issues.apache.org/jira/browse/BEAM-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8857:

Description: 


 - 2019-12-02 12:24:16.109021 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:53.668958 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:40.910919 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:19:17.637354 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.41 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-12-02 12:24:16.109021 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:53.668958 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:40.910919 -

Please consider upgrading the dependency software.amazon.awssdk:sqs. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: software.amazon.awssdk:sqs
> --
>
> Key: BEAM-8857
> URL: https://issues.apache.org/jira/browse/BEAM-8857
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-12-02 12:24:16.109021 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sqs. 
> The current version is 2.5.71. The latest version is 2.10.26 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:23:53.668958 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sqs. 
> The current version is 2.5.71. The latest version is 2.10.30 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:24:40.910919 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sqs. 
> The current version is 2.5.71. The latest version is 2.10.40 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:19:17.637354 
> -
> Please consider upgrading the dependency software.amazon.awssdk:sqs. 
> The current version is 2.5.71. The latest version is 2.10.41 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more

[jira] [Updated] (BEAM-8778) Beam Dependency Update Request: software.amazon.awssdk:sdk-core



 [ 
https://issues.apache.org/jira/browse/BEAM-8778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8778:

Description: 


 - 2019-11-19 21:16:45.434855 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:24:01.285164 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:38.577293 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:26.029788 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:19:01.746810 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.41 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-11-19 21:16:45.434855 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:24:01.285164 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:38.577293 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:26.029788 -

Please consider upgrading the dependency 
software.amazon.awssdk:sdk-core. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: software.amazon.awssdk:sdk-core
> ---
>
> Key: BEAM-8778
> URL: https://issues.apache.org/jira/browse/BEAM-8778
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-11-19 21:16:45.434855 
> -
> Please consider upgrading the dependency 
> software.amazon.awssdk:sdk-core. 
> The current version is 2.5.71. The latest version is 2.10.19 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:24:01.285164 
> -
> Please consider upgrading the dependency 
> software.amazon.awssdk:sdk-core. 
> The current version is 2.5.71. The latest version is 2.10.26 
> cc: 
>  Please refer to [Beam Dependency Guide 
>

[jira] [Updated] (BEAM-8724) Beam Dependency Update Request: org.slf4j:slf4j-simple

[
https://issues.apache.org/jira/browse/BEAM-8724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-8724:

Description:

- 2019-11-15 19:47:47.257481 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:07.039581 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:10.542496 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:22:42.211988 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:29.735248 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-30 14:18:02.884139 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-11-15 19:47:47.257481 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-11-19 21:16:07.039581 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-02 12:23:10.542496 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-09 12:22:42.211988 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-12-23 12:23:29.735248 -

Please consider upgrading the dependency org.slf4j:slf4j-simple.

The current version is 1.7.25. The latest version is 2.0.0-alpha1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

> Beam Dependency Update Request: org.slf4j:slf4j-simple
> --
>
> Key: BEAM-8724
> URL: https://issues.apache.org/jira/browse/BEAM-8724
> Project: Beam
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>

[jira] [Updated] (BEAM-8777) Beam Dependency Update Request: software.amazon.awssdk:dynamodb



 [ 
https://issues.apache.org/jira/browse/BEAM-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8777:

Description: 


 - 2019-11-19 21:16:41.659704 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:23:53.778671 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:30.990902 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:18.421491 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-30 14:18:53.835313 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.41 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-11-19 21:16:41.659704 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.19 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-02 12:23:53.778671 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.26 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-09 12:23:30.990902 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.30 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-12-23 12:24:18.421491 -

Please consider upgrading the dependency 
software.amazon.awssdk:dynamodb. 

The current version is 2.5.71. The latest version is 2.10.40 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: software.amazon.awssdk:dynamodb
> ---
>
> Key: BEAM-8777
> URL: https://issues.apache.org/jira/browse/BEAM-8777
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-11-19 21:16:41.659704 
> -
> Please consider upgrading the dependency 
> software.amazon.awssdk:dynamodb. 
> The current version is 2.5.71. The latest version is 2.10.19 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:23:53.778671 
> -
> Please consider upgrading the dependency 
> software.amazon.awssdk:dynamodb. 
> The current version is 2.5.71. The latest version is 2.10.26 
> cc: 
>  Please refer to [Beam Dependency Guide 
>

[jira] [Updated] (BEAM-8928) Beam Dependency Update Request: ru.yandex.clickhouse:clickhouse-jdbc