[jira] [Work logged] (BEAM-8937) Add a Jenkins job running GroupByKey load test on Java with Flink in Portability mode

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8937?focusedWorklogId=367332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367332
 ]

ASF GitHub Bot logged work on BEAM-8937:


Author: ASF GitHub Bot
Created on: 07/Jan/20 07:53
Start Date: 07/Jan/20 07:53
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #10495: [BEAM-8937] Add 
Jenkins job definitions for GroupByKey Java load test on Flink
URL: https://github.com/apache/beam/pull/10495#issuecomment-571476739
 
 
   Retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367332)
Time Spent: 15h 10m  (was: 15h)

> Add a Jenkins job running GroupByKey load test on Java with Flink in 
> Portability mode
> -
>
> Key: BEAM-8937
> URL: https://issues.apache.org/jira/browse/BEAM-8937
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 15h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367326
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 07/Jan/20 07:30
Start Date: 07/Jan/20 07:30
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571470248
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367326)
Time Spent: 9h 20m  (was: 9h 10m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367319
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 07/Jan/20 07:12
Start Date: 07/Jan/20 07:12
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571465367
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367319)
Time Spent: 9h 10m  (was: 9h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367318
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 07/Jan/20 07:11
Start Date: 07/Jan/20 07:11
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571465367
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367318)
Time Spent: 9h  (was: 8h 50m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=367309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367309
 ]

ASF GitHub Bot logged work on BEAM-9059:


Author: ASF GitHub Bot
Created on: 07/Jan/20 06:35
Start Date: 07/Jan/20 06:35
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on pull request #10511: 
[BEAM-9059] Migrate PTransformTranslation to use string constants
URL: https://github.com/apache/beam/pull/10511#discussion_r363607715
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 ##
 @@ -60,68 +61,131 @@
  * buffers}.
  */
 public class PTransformTranslation {
-
-  public static final String PAR_DO_TRANSFORM_URN = 
getUrn(StandardPTransforms.Primitives.PAR_DO);
-  public static final String FLATTEN_TRANSFORM_URN = 
getUrn(StandardPTransforms.Primitives.FLATTEN);
-  public static final String GROUP_BY_KEY_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Primitives.GROUP_BY_KEY);
-  public static final String IMPULSE_TRANSFORM_URN = 
getUrn(StandardPTransforms.Primitives.IMPULSE);
-  public static final String ASSIGN_WINDOWS_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Primitives.ASSIGN_WINDOWS);
-  public static final String TEST_STREAM_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Primitives.TEST_STREAM);
-  public static final String MAP_WINDOWS_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Primitives.MAP_WINDOWS);
-
+  // We specifically copy the values here so that they can be used in switch 
case statements
+  // and we validate that the value matches the actual URN in the static block 
below.
+
+  // Primitives
+  public static final String PAR_DO_TRANSFORM_URN = "beam:transform:pardo:v1";
+  public static final String FLATTEN_TRANSFORM_URN = 
"beam:transform:flatten:v1";
+  public static final String GROUP_BY_KEY_TRANSFORM_URN = 
"beam:transform:group_by_key:v1";
+  public static final String IMPULSE_TRANSFORM_URN = 
"beam:transform:impulse:v1";
+  public static final String ASSIGN_WINDOWS_TRANSFORM_URN = 
"beam:transform:window_into:v1";
+  public static final String TEST_STREAM_TRANSFORM_URN = 
"beam:transform:teststream:v1";
+  public static final String MAP_WINDOWS_TRANSFORM_URN = 
"beam:transform:map_windows:v1";
+
+  // DeprecatedPrimitives
   /**
* @deprecated SDKs should move away from creating `Read` transforms and 
migrate to using Impulse
* + SplittableDoFns.
*/
-  @Deprecated
-  public static final String READ_TRANSFORM_URN =
-  getUrn(StandardPTransforms.DeprecatedPrimitives.READ);
+  @Deprecated public static final String READ_TRANSFORM_URN = 
"beam:transform:read:v1";
+
   /**
* @deprecated runners should move away from translating 
`CreatePCollectionView` and treat this as
* part of the translation for a `ParDo` side input.
*/
   @Deprecated
-  public static final String CREATE_VIEW_TRANSFORM_URN =
-  getUrn(StandardPTransforms.DeprecatedPrimitives.CREATE_VIEW);
+  public static final String CREATE_VIEW_TRANSFORM_URN = 
"beam:transform:create_view:v1";
 
-  public static final String COMBINE_PER_KEY_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Composites.COMBINE_PER_KEY);
-  public static final String COMBINE_GLOBALLY_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Composites.COMBINE_GLOBALLY);
-  public static final String COMBINE_GROUPED_VALUES_TRANSFORM_URN =
-  getUrn(CombineComponents.COMBINE_GROUPED_VALUES);
+  // Composites
+  public static final String COMBINE_PER_KEY_TRANSFORM_URN = 
"beam:transform:combine_per_key:v1";
+  public static final String COMBINE_GLOBALLY_TRANSFORM_URN = 
"beam:transform:combine_globally:v1";
+  public static final String RESHUFFLE_URN = "beam:transform:reshuffle:v1";
+  public static final String WRITE_FILES_TRANSFORM_URN = 
"beam:transform:write_files:v1";
+
+  // CombineComponents
   public static final String COMBINE_PER_KEY_PRECOMBINE_TRANSFORM_URN =
-  getUrn(CombineComponents.COMBINE_PER_KEY_PRECOMBINE);
+  "beam:transform:combine_per_key_precombine:v1";
   public static final String COMBINE_PER_KEY_MERGE_ACCUMULATORS_TRANSFORM_URN =
-  getUrn(CombineComponents.COMBINE_PER_KEY_MERGE_ACCUMULATORS);
+  "beam:transform:combine_per_key_merge_accumulators:v1";
   public static final String COMBINE_PER_KEY_EXTRACT_OUTPUTS_TRANSFORM_URN =
-  getUrn(CombineComponents.COMBINE_PER_KEY_EXTRACT_OUTPUTS);
-  public static final String RESHUFFLE_URN = 
getUrn(StandardPTransforms.Composites.RESHUFFLE);
-  public static final String WRITE_FILES_TRANSFORM_URN =
-  getUrn(StandardPTransforms.Composites.WRITE_FILES);
+  "beam:transform:combine_per_key_extract_outputs:v1";
+  public static final String COMBINE_GROUPED_VALUES_TRANSFORM_URN =
+  "beam:transform:combine_grouped_values:v1";
 
   // 

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367303=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367303
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 07/Jan/20 06:03
Start Date: 07/Jan/20 06:03
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571449247
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367303)
Time Spent: 8h 50m  (was: 8h 40m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8593) Define expected behavior of running ZetaSQL query on tables with unsupported field types

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8593?focusedWorklogId=367289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367289
 ]

ASF GitHub Bot logged work on BEAM-8593:


Author: ASF GitHub Bot
Created on: 07/Jan/20 04:57
Start Date: 07/Jan/20 04:57
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #10020: [BEAM-8593] 
Remove testSelectFromTableWithMap from ZetaSQLDialectSpecTest
URL: https://github.com/apache/beam/pull/10020#issuecomment-571434238
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367289)
Remaining Estimate: 0h
Time Spent: 10m

> Define expected behavior of running ZetaSQL query on tables with unsupported 
> field types
> 
>
> Key: BEAM-8593
> URL: https://issues.apache.org/jira/browse/BEAM-8593
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql-zetasql
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> What should be the expected behavior if a user run a ZetaSQL query on a table 
> with a field type (e.g. MAP) that is not supported by ZetaSQL?
> More context: 
> [https://github.com/apache/beam/pull/10020#issuecomment-551368105]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=367275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367275
 ]

ASF GitHub Bot logged work on BEAM-9059:


Author: ASF GitHub Bot
Created on: 07/Jan/20 03:52
Start Date: 07/Jan/20 03:52
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10511: [BEAM-9059] 
Migrate PTransformTranslation to use string constants
URL: https://github.com/apache/beam/pull/10511#issuecomment-571421832
 
 
   @lukecwik sure. I would like to have look at this changes.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367275)
Time Spent: 40m  (was: 0.5h)

> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This allows for the values to be used within switch case statements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=367274=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367274
 ]

ASF GitHub Bot logged work on BEAM-9059:


Author: ASF GitHub Bot
Created on: 07/Jan/20 03:49
Start Date: 07/Jan/20 03:49
Worklog Time Spent: 10m 
  Work Description: sunjincheng121 commented on issue #10511: [BEAM-9059] 
Migrate PTransformTranslation to use string constants
URL: https://github.com/apache/beam/pull/10511#issuecomment-571421418
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367274)
Time Spent: 0.5h  (was: 20m)

> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This allows for the values to be used within switch case statements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9049?focusedWorklogId=367187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367187
 ]

ASF GitHub Bot logged work on BEAM-9049:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:43
Start Date: 07/Jan/20 02:43
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #10499: 
[BEAM-9049] Add opt-in --shutDownOnThrashing flag
URL: https://github.com/apache/beam/pull/10499
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367187)
Time Spent: 1h 40m  (was: 1.5h)

> MemoryMonitor thrashing detection is too aggressive for batch workers
> -
>
> Key: BEAM-9049
> URL: https://issues.apache.org/jira/browse/BEAM-9049
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> In the streaming dataflow worker we've implemented push-back so that we will 
> reduce parallelism when there is memory pressure. Since we cannot do this on 
> the batch worker, it doesn't make sense for our thrashing detection to be so 
> aggressive. We should increase the thresholds used for thrashing detection 
> when running on Batch workloads.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9049?focusedWorklogId=367186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367186
 ]

ASF GitHub Bot logged work on BEAM-9049:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:43
Start Date: 07/Jan/20 02:43
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #10499: [BEAM-9049] Add 
opt-in --shutDownOnThrashing flag
URL: https://github.com/apache/beam/pull/10499#issuecomment-571408052
 
 
   Closing this as I think we're better off making changes just for the batch 
worker.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367186)
Time Spent: 1.5h  (was: 1h 20m)

> MemoryMonitor thrashing detection is too aggressive for batch workers
> -
>
> Key: BEAM-9049
> URL: https://issues.apache.org/jira/browse/BEAM-9049
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> In the streaming dataflow worker we've implemented push-back so that we will 
> reduce parallelism when there is memory pressure. Since we cannot do this on 
> the batch worker, it doesn't make sense for our thrashing detection to be so 
> aggressive. We should increase the thresholds used for thrashing detection 
> when running on Batch workloads.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers

[Brian Hulette (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Hulette updated BEAM-9049:

Description: In the streaming dataflow worker we've implemented push-back 
so that we will reduce parallelism when there is memory pressure. Since we 
cannot do this on the batch worker, it doesn't make sense for our thrashing 
detection to be so aggressive. We should increase the thresholds used for 
thrashing detection when running on Batch workloads.  (was: MemoryMonitor has 
many knobs that are just hard-coded for now. We should make some (or all) of 
these knobs available via pipeline options, so that users have control over 
when workers will be killed due to thrashing. 

https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java#L123)

> MemoryMonitor thrashing detection is too aggressive for batch workers
> -
>
> Key: BEAM-9049
> URL: https://issues.apache.org/jira/browse/BEAM-9049
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In the streaming dataflow worker we've implemented push-back so that we will 
> reduce parallelism when there is memory pressure. Since we cannot do this on 
> the batch worker, it doesn't make sense for our thrashing detection to be so 
> aggressive. We should increase the thresholds used for thrashing detection 
> when running on Batch workloads.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8844) [SQL] Create performance tests for BigQueryTable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8844?focusedWorklogId=367183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367183
 ]

ASF GitHub Bot logged work on BEAM-8844:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:40
Start Date: 07/Jan/20 02:40
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #10226: [BEAM-8844] Add a 
new Jenkins job for SQL perf tests
URL: https://github.com/apache/beam/pull/10226#issuecomment-571407388
 
 
   > This is failing a bunch of tests.
   
   Should be fixed now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367183)
Time Spent: 8h 20m  (was: 8h 10m)

> [SQL] Create performance tests for BigQueryTable
> 
>
> Key: BEAM-8844
> URL: https://issues.apache.org/jira/browse/BEAM-8844
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> They should measure read-time for:
>  * DIRECT_READ w/o push-down
>  * DIRECT_READ w/ push-down
>  * DEFAULT



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8844) [SQL] Create performance tests for BigQueryTable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8844?focusedWorklogId=367182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367182
 ]

ASF GitHub Bot logged work on BEAM-8844:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:40
Start Date: 07/Jan/20 02:40
Worklog Time Spent: 10m 
  Work Description: 11moon11 commented on issue #10226: [BEAM-8844] Add a 
new Jenkins job for SQL perf tests
URL: https://github.com/apache/beam/pull/10226#issuecomment-571407388
 
 
   > This is failing a bunch of tests.
   Should be fixed now.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367182)
Time Spent: 8h 10m  (was: 8h)

> [SQL] Create performance tests for BigQueryTable
> 
>
> Key: BEAM-8844
> URL: https://issues.apache.org/jira/browse/BEAM-8844
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> They should measure read-time for:
>  * DIRECT_READ w/o push-down
>  * DIRECT_READ w/ push-down
>  * DEFAULT



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9049) MemoryMonitor thrashing detection is too aggressive for batch workers

[Brian Hulette (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brian Hulette updated BEAM-9049:

Summary: MemoryMonitor thrashing detection is too aggressive for batch 
workers  (was: Add knobs for dataflow worker MemoryMonitor)

> MemoryMonitor thrashing detection is too aggressive for batch workers
> -
>
> Key: BEAM-9049
> URL: https://issues.apache.org/jira/browse/BEAM-9049
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Fix For: 2.19.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> MemoryMonitor has many knobs that are just hard-coded for now. We should make 
> some (or all) of these knobs available via pipeline options, so that users 
> have control over when workers will be killed due to thrashing. 
> https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/MemoryMonitor.java#L123



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9041) SchemaCoder equals should not rely on from/toRowFunction equality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9041?focusedWorklogId=367178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367178
 ]

ASF GitHub Bot logged work on BEAM-9041:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:36
Start Date: 07/Jan/20 02:36
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #10492: [BEAM-9041, 
BEAM-9042] SchemaCoder equals should not rely on from/toRowFunction equality
URL: https://github.com/apache/beam/pull/10492#issuecomment-571406505
 
 
   Hmm. I am torn on this. I agree with @TheNeuralBit that mostly these 
conversions should just own their own `equals`. But I see how 
`SerializableFunction`, especially lambdas, will usually not implement this.
   
   In the case of the `SchemaCoder` it is often automated, isn't it? And when 
it isn't, it is fair that it is a one-time up front effort to write a slightly 
larger class that has a good `equals`. We could provide a convenient base 
class, I guess, or even now add a `default` implementation in the functional 
interface maybe?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367178)
Time Spent: 1h 10m  (was: 1h)

> SchemaCoder equals should not rely on from/toRowFunction equality
> -
>
> Key: BEAM-9041
> URL: https://issues.apache.org/jira/browse/BEAM-9041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.18.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> SchemaCoder equals implementation relies on SerializableFunction equals 
> method, this is error-prone because users rarely implement the equals method 
> for a SerializableFunction. One alternative would be to rely on bytes 
> equality for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=367172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367172
 ]

ASF GitHub Bot logged work on BEAM-7246:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:19
Start Date: 07/Jan/20 02:19
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add 
Google Spanner IO Read on Python SDK
URL: https://github.com/apache/beam/pull/9606#issuecomment-571402431
 
 
   Thanks. Mostly looks good.
   
   Added few more comments.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367172)
Time Spent: 10h 50m  (was: 10h 40m)

> Create a Spanner IO for Python
> --
>
> Key: BEAM-7246
> URL: https://issues.apache.org/jira/browse/BEAM-7246
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>
> Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only).
> Testing in this work item will be in the form of DirectRunner tests and 
> manual testing.
> Integration and performance tests are a separate work item (not included 
> here).
> See https://beam.apache.org/documentation/io/built-in/. The goal is to add 
> Google Clound Spanner to the Database column for the Python/Batch row.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=367170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367170
 ]

ASF GitHub Bot logged work on BEAM-7246:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:19
Start Date: 07/Jan/20 02:19
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #9606: 
[BEAM-7246] Add Google Spanner IO Read on Python SDK
URL: https://github.com/apache/beam/pull/9606#discussion_r363567042
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/experimental/spannerio_test.py
 ##
 @@ -0,0 +1,271 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+import datetime
+import logging
+import random
+import string
+import unittest
+
+import mock
+
+import apache_beam as beam
+from apache_beam.testing.test_pipeline import TestPipeline
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+
+# Protect against environments where spanner library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
+try:
+  from google.cloud import spanner
+  from apache_beam.io.gcp.experimental.spannerio import (create_transaction,
+ ReadOperation,
+ ReadFromSpanner) # 
pylint: disable=unused-import
+  # disable=unused-import
+except ImportError:
+  spanner = None
+# pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
+
+
+MAX_DB_NAME_LENGTH = 30
+TEST_PROJECT_ID = 'apache-beam-testing'
+TEST_INSTANCE_ID = 'beam-test'
+TEST_DATABASE_PREFIX = 'spanner-testdb-'
+# TEST_TABLE = 'users'
+# TEST_COLUMNS = ['Key', 'Value']
+FAKE_ROWS = [[1, 'Alice'], [2, 'Bob'], [3, 'Carl'], [4, 'Dan'], [5, 'Evan'],
+ [6, 'Floyd']]
+
+
+def _generate_database_name():
+  mask = string.ascii_lowercase + string.digits
+  length = MAX_DB_NAME_LENGTH - 1 - len(TEST_DATABASE_PREFIX)
+  return TEST_DATABASE_PREFIX + ''.join(random.choice(mask) for i in range(
+  length))
+
+
+def _generate_test_data():
+  mask = string.ascii_lowercase + string.digits
+  length = 100
+  return [('users', ['Key', 'Value'], [(x, ''.join(
+  random.choice(mask) for _ in range(length))) for x in range(1, 5)])]
+
+
+@unittest.skipIf(spanner is None, 'GCP dependencies are not installed.')
+@mock.patch('apache_beam.io.gcp.experimental.spannerio.Client')
+@mock.patch('apache_beam.io.gcp.experimental.spannerio.BatchSnapshot')
+class SpannerReadTest(unittest.TestCase):
+
+  def test_read_with_query_batch(self, mock_batch_snapshot_class,
 
 Review comment:
   How about runReadUsingIndex ?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367170)
Time Spent: 10h 40m  (was: 10.5h)

> Create a Spanner IO for Python
> --
>
> Key: BEAM-7246
> URL: https://issues.apache.org/jira/browse/BEAM-7246
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
> Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only).
> Testing in this work item will be in the form of DirectRunner tests and 
> manual testing.
> Integration and performance tests are a separate work item (not included 
> here).
> See https://beam.apache.org/documentation/io/built-in/. The goal is to add 
> Google Clound Spanner to the Database column for the Python/Batch row.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=367169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367169
 ]

ASF GitHub Bot logged work on BEAM-7246:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:18
Start Date: 07/Jan/20 02:18
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #9606: 
[BEAM-7246] Add Google Spanner IO Read on Python SDK
URL: https://github.com/apache/beam/pull/9606#discussion_r363566806
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/experimental/spannerio.py
 ##
 @@ -0,0 +1,565 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Google Cloud Spanner IO
+
+Experimental; no backwards-compatibility guarantees.
+
+This is an experimental module for reading and writing data from Google Cloud
+Spanner. Visit: https://cloud.google.com/spanner for more details.
+
+To read from Cloud Spanner apply ReadFromSpanner transformation. It will
+return a PCollection, where each element represents an individual row returned
+from the read operation. Both Query and Read APIs are supported.
+
+ReadFromSpanner relies on the ReadOperation objects which is exposed by the
+SpannerIO API. ReadOperation holds the immutable data which is responsible to
+execute batch and naive reads on Cloud Spanner. This is done for more
+convenient programming.
+
+ReadFromSpanner reads from Cloud Spanner by providing either an 'sql' param
+in the constructor or 'table' name with 'columns' as list. For example:::
+
+  records = (pipeline
+| ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+sql='Select * from users'))
+
+  records = (pipeline
+| ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+table='users', columns=['id', 'name', 'email']))
+
+You can also perform multiple reads by providing a list of ReadOperations
+to the ReadFromSpanner transform constructor. ReadOperation exposes two static
+methods. Use 'query' to perform sql based reads, 'table' to perform read from
+table name. For example:::
+
+  read_operations = [
+  ReadOperation.table(table='customers', columns=['name',
+  'email']),
+  ReadOperation.table(table='vendors', columns=['name',
+  'email']),
+]
+  all_users = pipeline | ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+read_operations=read_operations)
+
+  ...OR...
+
+  read_operations = [
+  ReadOperation.query(sql='Select name, email from
+  customers'),
+  ReadOperation.query(
+sql='Select * from users where id <= @user_id',
+params={'user_id': 100},
+params_type={'user_id': param_types.INT64}
+  ),
+]
+  # `params_types` are instance of `google.cloud.spanner.param_types`
+  all_users = pipeline | ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+read_operations=read_operations)
+
+For more information, please review the docs on class ReadOperation.
+
+User can also able to provide the ReadOperation in form of PCollection via
+pipeline. For example:::
+
+  users = (pipeline
+   | beam.Create([ReadOperation...])
+   | ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME))
+
+User may also create cloud spanner transaction from the transform called
+`create_transaction` which is available in the SpannerIO API.
+
+The transform is guaranteed to be executed on a consistent snapshot of data,
+utilizing the power of read only transactions. Staleness of data can be
+controlled by providing the `read_timestamp` or `exact_staleness` param values
+in the constructor.
+
+This transform requires root of the pipeline (PBegin) and returns PTransform
+which is passed later to the `ReadFromSpanner` constructor. `ReadFromSpanner`
+pass this transaction PTransform as a singleton side input to the
+`_NaiveSpannerReadDoFn` containing 'session_id' and 'transaction_id'.
+For example:::
+
+  transaction = (pipeline | 

[jira] [Work logged] (BEAM-8496) remove SDF translators in flink streaming transform translator

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8496?focusedWorklogId=367168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367168
 ]

ASF GitHub Bot logged work on BEAM-8496:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:17
Start Date: 07/Jan/20 02:17
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on pull request #9903: [BEAM-8496] 
remove SDF translators from flink translator
URL: https://github.com/apache/beam/pull/9903
 
 
   fix the issue mentioned in the "beam_PostCommit_Java_Nexmark_Flink are 
failing continuously" 
[thread](https://lists.apache.org/thread.html/ef1d7b416a11059d20f0279ac183829eaf32ecbdb85cfda57a1f0027@%3Cdev.beam.apache.org%3E)
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 

[jira] [Updated] (BEAM-9057) Make sure restriction_tracker.deferred_remainder is never called more than once for one

[Boyuan Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang updated BEAM-9057:
---
Summary: Make sure restriction_tracker.deferred_remainder is never called 
more than once for one   (was: Make sure 
restriction_tracker.deferred_remainder is never called more than twice for one 
)

> Make sure restriction_tracker.deferred_remainder is never called more than 
> once for one 
> --
>
> Key: BEAM-9057
> URL: https://issues.apache.org/jira/browse/BEAM-9057
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8376) Add FirestoreIO connector to Java SDK

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8376?focusedWorklogId=367160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367160
 ]

ASF GitHub Bot logged work on BEAM-8376:


Author: ASF GitHub Bot
Created on: 07/Jan/20 02:02
Start Date: 07/Jan/20 02:02
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #10187: [BEAM-8376] 
Initial version of firestore connector JavaSDK
URL: https://github.com/apache/beam/pull/10187#issuecomment-571398831
 
 
   Any updates ?
   
   Thanks.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367160)
Time Spent: 2h 20m  (was: 2h 10m)

> Add FirestoreIO connector to Java SDK
> -
>
> Key: BEAM-8376
> URL: https://issues.apache.org/jira/browse/BEAM-8376
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-gcp
>Reporter: Stefan Djelekar
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Motivation:
> There is no Firestore connector for Java SDK at the moment.
> Having it will enhance the integrations with database options on the Google 
> Cloud Platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=367157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367157
 ]

ASF GitHub Bot logged work on BEAM-7246:


Author: ASF GitHub Bot
Created on: 07/Jan/20 01:58
Start Date: 07/Jan/20 01:58
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #9606: 
[BEAM-7246] Add Google Spanner IO Read on Python SDK
URL: https://github.com/apache/beam/pull/9606#discussion_r363563133
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/spannerio.py
 ##
 @@ -0,0 +1,558 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Google Cloud Spanner IO
+
+This is an experimental module for reading and writing data from Google Cloud
+Spanner. Visit: https://cloud.google.com/spanner for more details.
+
+To read from Cloud Spanner apply _ReadFromSpanner transformation. It will
+return a PCollection, where each element represents an individual row returned
+from the read operation. Both Query and Read APIs are supported.
+
+_ReadFromSpanner relies on the _ReadOperation objects which is exposed by the
+SpannerIO API. _ReadOperation holds the immutable data which is responsible to
+execute batch and naive reads on Cloud Spanner. This is done for more
+convenient programming.
+
+_ReadFromSpanner reads from Cloud Spanner by providing either an 'sql' param
+in the constructor or 'table' name with 'columns' as list. For example:::
+
+  records = (pipeline
+| _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+sql='Select * from users'))
+
+  records = (pipeline
+| _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+table='users', columns=['id', 'name', 'email']))
+
+You can also perform multiple reads by providing a list of _ReadOperations
+to the _ReadFromSpanner transform constructor. _ReadOperation exposes two 
static
+methods. Use 'query' to perform sql based reads, 'table' to perform read from
+table name. For example:::
+
+  read_operations = [
+  _ReadOperation.table('customers', ['name', 'email']),
+  _ReadOperation.table('vendors', ['name', 'email']),
+]
+  all_users = pipeline | _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+read_operations=read_operations)
+
+  ...OR...
+
+  read_operations = [
+  _ReadOperation.query('Select name, email from 
customers'),
+  _ReadOperation.query(
+sql='Select * from users where id <= @user_id',
+params={'user_id': 100},
+params_type={'user_id': param_types.INT64}
+  ),
+]
+  # `params_types` are instance of `google.cloud.spanner_v1.param_types`
+  all_users = pipeline | _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+read_operations=read_operations)
+
+For more information, please review the docs on class _ReadOperation.
+
+User can also able to provide the _ReadOperation in form of PCollection via
+pipeline. For example:::
+
+  users = (pipeline
+   | beam.Create([_ReadOperation...])
+   | _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME))
+
+User may also create cloud spanner transaction from the transform called
+`_create_transaction` which is available in the SpannerIO API.
+
+The transform is guaranteed to be executed on a consistent snapshot of data,
+utilizing the power of read only transactions. Staleness of data can be
+controlled by providing the `read_timestamp` or `exact_staleness` param values
+in the constructor.
+
+This transform requires root of the pipeline (PBegin) and returns the dict
+containing 'session_id' and 'transaction_id'. This `_create_transaction`
+PTransform later passed to the constructor of _ReadFromSpanner. For example:::
+
+  transaction = (pipeline | _create_transaction(TEST_PROJECT_ID,
+  TEST_INSTANCE_ID,
+  DB_NAME))
+
+  users = pipeline | _ReadFromSpanner(PROJECT_ID, INSTANCE_ID, DB_NAME,
+

[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=367148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367148
 ]

ASF GitHub Bot logged work on BEAM-9027:


Author: ASF GitHub Bot
Created on: 07/Jan/20 01:32
Start Date: 07/Jan/20 01:32
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10513: [BEAM-9027] [SQL] 
Fix ZetaSQL Byte Literals
URL: https://github.com/apache/beam/pull/10513#issuecomment-571392353
 
 
   R: @11moon11
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367148)
Time Spent: 2h 20m  (was: 2h 10m)

> [SQL] ZetaSQL unparsing should produce valid result
> ---
>
> Key: BEAM-9027
> URL: https://issues.apache.org/jira/browse/BEAM-9027
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql-zetasql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> * ZetaSQL does not recognize keyword INTERVAL
>  * Calcite cannot unparse RexNode back to bytes literal
>  * Calcite cannot unparse some floating point literals correctly
>  * Calcite cannot unparse some string literals correctly
>  * Calcite cannot unparse types correctly for CAST function



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=367147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367147
 ]

ASF GitHub Bot logged work on BEAM-9027:


Author: ASF GitHub Bot
Created on: 07/Jan/20 01:32
Start Date: 07/Jan/20 01:32
Worklog Time Spent: 10m 
  Work Description: apilloud commented on pull request #10513: [BEAM-9027] 
[SQL] Fix ZetaSQL Byte Literals
URL: https://github.com/apache/beam/pull/10513
 
 
   Fix ZetaSQL Byte Literals affected by CALCITE-3714
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-8780) Improve publish_docker_images.sh

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8780?focusedWorklogId=367140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367140
 ]

ASF GitHub Bot logged work on BEAM-8780:


Author: ASF GitHub Bot
Created on: 07/Jan/20 01:10
Start Date: 07/Jan/20 01:10
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #10512: 
[WIP][BEAM-8780]Improve container release script
URL: https://github.com/apache/beam/pull/10512
 
 
   Improve docker container script: reuse verified rc images instead of 
recreating images. 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8685?focusedWorklogId=367139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367139
 ]

ASF GitHub Bot logged work on BEAM-8685:


Author: ASF GitHub Bot
Created on: 07/Jan/20 01:06
Start Date: 07/Jan/20 01:06
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10508: [BEAM-8685] 
sdks/java: google_auth_version 0.19.0
URL: https://github.com/apache/beam/pull/10508#issuecomment-571385794
 
 
   Run Java PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367139)
Time Spent: 40m  (was: 0.5h)

> Beam Dependency Update Request: 
> com.google.auth:google-auth-library-oauth2-http
> ---
>
> Key: BEAM-8685
> URL: https://issues.apache.org/jira/browse/BEAM-8685
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:27.324449 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:03.844285 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:10:30.864371 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:09:38.646889 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:09:39.967215 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:10.534268 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:08:45.451960 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work started] (BEAM-8780) Improve publish_docker_images.sh

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on BEAM-8780 started by Hannah Jiang.
--
> Improve publish_docker_images.sh
> 
>
> Key: BEAM-8780
> URL: https://issues.apache.org/jira/browse/BEAM-8780
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Mark Liu
>Assignee: Hannah Jiang
>Priority: Major
>
> Few improvement places:
>  * The script builds new image for publishing. Better to reuse existing 
> staged rc images.
>  * This script ask input that’s unrelated to publish (i.e rc #, gpg key...)
>  * This script first run unrelated staging steps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9041) SchemaCoder equals should not rely on from/toRowFunction equality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9041?focusedWorklogId=367138=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367138
 ]

ASF GitHub Bot logged work on BEAM-9041:


Author: ASF GitHub Bot
Created on: 07/Jan/20 00:50
Start Date: 07/Jan/20 00:50
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #10492: [BEAM-9041, 
BEAM-9042] SchemaCoder equals should not rely on from/toRowFunction equality
URL: https://github.com/apache/beam/pull/10492#issuecomment-571381848
 
 
   Sorry revert was a poor choice of words. I just meant a big part of that PR
   was making equals work for the functions produced by
   GetterBasedSchemaProvider so I would like it if they continued to be used.
   For a function without any state I would write the equals function to just
   compare this.getClass().equals(other.getClass()) since any two instances
   will do the same thing.
   
   On Mon, Jan 6, 2020 at 2:00 PM Ismaël Mejía 
   wrote:
   
   > This is not a revert. Previous version did not compare from/toRow
   > functions for equality. Do you have any suggestion on how to compare both
   > functions? It is not really clear to me how to do so in particular for
   > functions with no state.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or unsubscribe
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367138)
Time Spent: 1h  (was: 50m)

> SchemaCoder equals should not rely on from/toRowFunction equality
> -
>
> Key: BEAM-9041
> URL: https://issues.apache.org/jira/browse/BEAM-9041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.18.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> SchemaCoder equals implementation relies on SerializableFunction equals 
> method, this is error-prone because users rarely implement the equals method 
> for a SerializableFunction. One alternative would be to rely on bytes 
> equality for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=367126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367126
 ]

ASF GitHub Bot logged work on BEAM-9059:


Author: ASF GitHub Bot
Created on: 07/Jan/20 00:23
Start Date: 07/Jan/20 00:23
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #10511: [BEAM-9059] Migrate 
PTransformTranslation to use string constants
URL: https://github.com/apache/beam/pull/10511#issuecomment-571375067
 
 
   R: @sunjincheng121 can you review this change for me?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367126)
Time Spent: 20m  (was: 10m)

> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This allows for the values to be used within switch case statements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9059) Migrate PTransformTranslation to use string constants

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?focusedWorklogId=367124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367124
 ]

ASF GitHub Bot logged work on BEAM-9059:


Author: ASF GitHub Bot
Created on: 07/Jan/20 00:19
Start Date: 07/Jan/20 00:19
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10511: [BEAM-9059] 
Migrate PTransformTranslation to use string constants
URL: https://github.com/apache/beam/pull/10511
 
 
   This allows for us to use them in switch/case statements
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 

[jira] [Updated] (BEAM-9059) Migrate PTransformTranslation to use string constants

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-9059:

Status: Open  (was: Triage Needed)

> Migrate PTransformTranslation to use string constants
> -
>
> Key: BEAM-9059
> URL: https://issues.apache.org/jira/browse/BEAM-9059
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Trivial
>
> This allows for the values to be used within switch case statements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9059) Migrate PTransformTranslation to use string constants

[Luke Cwik (Jira)]

Luke Cwik created BEAM-9059:
---

 Summary: Migrate PTransformTranslation to use string constants
 Key: BEAM-9059
 URL: https://issues.apache.org/jira/browse/BEAM-9059
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Luke Cwik
Assignee: Luke Cwik


This allows for the values to be used within switch case statements.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367105
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:49
Start Date: 06/Jan/20 23:49
Worklog Time Spent: 10m 
  Work Description: apilloud commented on pull request #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367105)
Time Spent: 3h 20m  (was: 3h 10m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?focusedWorklogId=367102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367102
 ]

ASF GitHub Bot logged work on BEAM-9058:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:39
Start Date: 06/Jan/20 23:39
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #10510: [BEAM-9058] 
Reenable line-too-long linter check.
URL: https://github.com/apache/beam/pull/10510#issuecomment-571363535
 
 
   @chadrik np. Please LGTM once you think this is good to go :)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367102)
Time Spent: 40m  (was: 0.5h)

> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> {noformat}
> > Task :sdks:python:test-suites:tox:py37:lintPy37
>  
>  * Module apache_beam.testing.test_stream_it_test
>  apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
>  * Module apache_beam.io.filesystems
>  apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
>  apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
>  * Module apache_beam.runners.portability.fn_api_runner
>  apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
>  * Module apache_beam.transforms.core
>  apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
>  * Module setup
>  setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
>  {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8685?focusedWorklogId=367101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367101
 ]

ASF GitHub Bot logged work on BEAM-8685:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:38
Start Date: 06/Jan/20 23:38
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10508: [BEAM-8685] 
sdks/java: google_auth_version 0.19.0
URL: https://github.com/apache/beam/pull/10508#issuecomment-571363344
 
 
   retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367101)
Time Spent: 0.5h  (was: 20m)

> Beam Dependency Update Request: 
> com.google.auth:google-auth-library-oauth2-http
> ---
>
> Key: BEAM-8685
> URL: https://issues.apache.org/jira/browse/BEAM-8685
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:27.324449 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:03.844285 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:10:30.864371 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:09:38.646889 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:09:39.967215 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:10.534268 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:08:45.451960 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?focusedWorklogId=367098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367098
 ]

ASF GitHub Bot logged work on BEAM-9058:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:35
Start Date: 06/Jan/20 23:35
Worklog Time Spent: 10m 
  Work Description: chadrik commented on issue #10510: [BEAM-9058] Reenable 
line-too-long linter check.
URL: https://github.com/apache/beam/pull/10510#issuecomment-571362692
 
 
   d'oh!  sorry about that.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367098)
Time Spent: 0.5h  (was: 20m)

> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> {noformat}
> > Task :sdks:python:test-suites:tox:py37:lintPy37
>  
>  * Module apache_beam.testing.test_stream_it_test
>  apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
>  * Module apache_beam.io.filesystems
>  apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
>  apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
>  * Module apache_beam.runners.portability.fn_api_runner
>  apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
>  * Module apache_beam.transforms.core
>  apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
>  * Module setup
>  setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
>  {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9013) Multi-output TestStream breaks the DataflowRunner

[Ahmet Altay (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-9013:
--
Fix Version/s: 2.18.0

> Multi-output TestStream breaks the DataflowRunner
> -
>
> Key: BEAM-9013
> URL: https://issues.apache.org/jira/browse/BEAM-9013
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.17.0
>Reporter: Sam Rohde
>Assignee: Sam Rohde
>Priority: Major
> Fix For: 2.17.0, 2.18.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-9058:
--
Description: 
Without getting into a question whether 80 is a reasonable limit, what are the 
reasons to treat  type annotations differently than other parts of Python code 
(including comments), which have a 80 character limit?  

 

Places that need to be fixed (PR coming):

```
{noformat}

> Task :sdks:python:test-suites:tox:py37:lintPy37
 
 * Module apache_beam.testing.test_stream_it_test
 apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long (106/80) 
(line-too-long)
 * Module apache_beam.io.filesystems
 apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
(line-too-long)
 apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
(line-too-long)
 * Module apache_beam.runners.portability.fn_api_runner
 apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
(114/80) (line-too-long)
 * Module apache_beam.transforms.core
 apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
(line-too-long)
 apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
(line-too-long)
 apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
(line-too-long)
 * Module setup
 setup.py:141:0: C0301: Line too long (81/80) (line-too-long)


 {noformat}

  was:
Without getting into a question whether 80 is a reasonable limit, what are the 
reasons to treat  type annotations differently than other parts of Python code 
(including comments), which have a 80 character limit?  

 

Places that need to be fixed (PR coming):

```

> Task :sdks:python:test-suites:tox:py37:lintPy37
* Module apache_beam.testing.test_stream_it_test
apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long (106/80) 
(line-too-long)
* Module apache_beam.io.filesystems
apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
(line-too-long)
apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
(line-too-long)
* Module apache_beam.runners.portability.fn_api_runner
apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
(114/80) (line-too-long)
* Module apache_beam.transforms.core
apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
(line-too-long)
apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
(line-too-long)
apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
(line-too-long)
* Module setup
setup.py:141:0: C0301: Line too long (81/80) (line-too-long)



```


> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> {noformat}
> > Task :sdks:python:test-suites:tox:py37:lintPy37
>  
>  * Module apache_beam.testing.test_stream_it_test
>  apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
>  * Module apache_beam.io.filesystems
>  apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
>  apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
>  * Module apache_beam.runners.portability.fn_api_runner
>  apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
>  * Module apache_beam.transforms.core
>  apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
>  apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
>  * Module setup
>  setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
>  {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?focusedWorklogId=367097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367097
 ]

ASF GitHub Bot logged work on BEAM-9058:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:28
Start Date: 06/Jan/20 23:28
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #10510: [BEAM-9058] 
Reenable line-too-long linter check.
URL: https://github.com/apache/beam/pull/10510#issuecomment-571361008
 
 
   R: @chadrik @udim 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367097)
Time Spent: 20m  (was: 10m)

> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> > Task :sdks:python:test-suites:tox:py37:lintPy37
> * Module apache_beam.testing.test_stream_it_test
> apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
> * Module apache_beam.io.filesystems
> apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
> apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
> * Module apache_beam.runners.portability.fn_api_runner
> apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
> * Module apache_beam.transforms.core
> apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
> * Module setup
> setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?focusedWorklogId=367096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367096
 ]

ASF GitHub Bot logged work on BEAM-9058:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:28
Start Date: 06/Jan/20 23:28
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10510: [BEAM-9058] 
Reenable line-too-long linter check.
URL: https://github.com/apache/beam/pull/10510
 
 
   This PR fixes the regular expression to exclude type annotations from linter 
line-too-long check.
   We may also want to reconsider whether type annotations should be excluded 
from 80 character limit, see BEAM-9058. 
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[![Build
 

[jira] [Work logged] (BEAM-8933) BigQuery IO should support read/write in Arrow format

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8933?focusedWorklogId=367095=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367095
 ]

ASF GitHub Bot logged work on BEAM-8933:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:27
Start Date: 06/Jan/20 23:27
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #10384: [BEAM-8933] 
Utilities for converting Arrow schemas and reading Arrow batches as Rows
URL: https://github.com/apache/beam/pull/10384#issuecomment-571360566
 
 
   R: @reuvenlax 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367095)
Time Spent: 4.5h  (was: 4h 20m)

> BigQuery IO should support read/write in Arrow format
> -
>
> Key: BEAM-8933
> URL: https://issues.apache.org/jira/browse/BEAM-8933
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> As of right now BigQuery uses Avro format for reading and writing.
> We should add a config to BigQueryIO to specify which format to use: Arrow or 
> Avro (with Avro as default).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367093
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:25
Start Date: 06/Jan/20 23:25
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571360024
 
 
   Run SQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367093)
Time Spent: 3h  (was: 2h 50m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367092
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:25
Start Date: 06/Jan/20 23:25
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571360001
 
 
   Run RAT PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367092)
Time Spent: 2h 50m  (was: 2h 40m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367094=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367094
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:25
Start Date: 06/Jan/20 23:25
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571360052
 
 
   Run Spotless PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367094)
Time Spent: 3h 10m  (was: 3h)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7746) Add type hints to python code

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7746?focusedWorklogId=367089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367089
 ]

ASF GitHub Bot logged work on BEAM-7746:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:24
Start Date: 06/Jan/20 23:24
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #9915: [BEAM-7746] 
Add python type hints (part 1)
URL: https://github.com/apache/beam/pull/9915#discussion_r363529954
 
 

 ##
 File path: sdks/python/.pylintrc
 ##
 @@ -178,6 +179,7 @@ indent-after-paren=4
 ignore-long-lines=(?x)
   (^\s*(import|from)\s
|^\s*(\#\ )??$
+   |# type:
 
 Review comment:
   This line disabled line-too-long errors across the board: 
https://issues.apache.org/jira/browse/BEAM-9058.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367089)
Time Spent: 36h 50m  (was: 36h 40m)

> Add type hints to python code
> -
>
> Key: BEAM-7746
> URL: https://issues.apache.org/jira/browse/BEAM-7746
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: Chad Dombrova
>Assignee: Chad Dombrova
>Priority: Major
>  Time Spent: 36h 50m
>  Remaining Estimate: 0h
>
> As a developer of the beam source code, I would like the code to use pep484 
> type hints so that I can clearly see what types are required, get completion 
> in my IDE, and enforce code correctness via a static analyzer like mypy.
> This may be considered a precursor to BEAM-7060
> Work has been started here:  [https://github.com/apache/beam/pull/9056]
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367090
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:24
Start Date: 06/Jan/20 23:24
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571359887
 
 
   Run precommits
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367090)
Time Spent: 2.5h  (was: 2h 20m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367091
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:24
Start Date: 06/Jan/20 23:24
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571359938
 
 
   Run JavaBeamZetaSQL PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367091)
Time Spent: 2h 40m  (was: 2.5h)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-9058:
--
Status: Open  (was: Triage Needed)

> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> > Task :sdks:python:test-suites:tox:py37:lintPy37
> * Module apache_beam.testing.test_stream_it_test
> apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
> * Module apache_beam.io.filesystems
> apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
> apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
> * Module apache_beam.runners.portability.fn_api_runner
> apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
> * Module apache_beam.transforms.core
> apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
> * Module setup
> setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-9058:
-

Assignee: Chad Dombrova

> Line-too-long Python lint checks are no longer working.
> ---
>
> Key: BEAM-9058
> URL: https://issues.apache.org/jira/browse/BEAM-9058
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Chad Dombrova
>Priority: Major
>
> Without getting into a question whether 80 is a reasonable limit, what are 
> the reasons to treat  type annotations differently than other parts of Python 
> code (including comments), which have a 80 character limit?  
>  
> Places that need to be fixed (PR coming):
> ```
> > Task :sdks:python:test-suites:tox:py37:lintPy37
> * Module apache_beam.testing.test_stream_it_test
> apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long 
> (106/80) (line-too-long)
> * Module apache_beam.io.filesystems
> apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
> (line-too-long)
> apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
> (line-too-long)
> * Module apache_beam.runners.portability.fn_api_runner
> apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
> (114/80) (line-too-long)
> * Module apache_beam.transforms.core
> apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
> (line-too-long)
> apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
> (line-too-long)
> * Module setup
> setup.py:141:0: C0301: Line too long (81/80) (line-too-long)
> 
> ```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9058) Line-too-long Python lint checks are no longer working.

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-9058:
-

 Summary: Line-too-long Python lint checks are no longer working.
 Key: BEAM-9058
 URL: https://issues.apache.org/jira/browse/BEAM-9058
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


Without getting into a question whether 80 is a reasonable limit, what are the 
reasons to treat  type annotations differently than other parts of Python code 
(including comments), which have a 80 character limit?  

 

Places that need to be fixed (PR coming):

```

> Task :sdks:python:test-suites:tox:py37:lintPy37
* Module apache_beam.testing.test_stream_it_test
apache_beam/testing/test_stream_it_test.py:50:0: C0301: Line too long (106/80) 
(line-too-long)
* Module apache_beam.io.filesystems
apache_beam/io/filesystems.py:99:0: C0301: Line too long (101/80) 
(line-too-long)
apache_beam/io/filesystems.py:100:0: C0301: Line too long (111/80) 
(line-too-long)
* Module apache_beam.runners.portability.fn_api_runner
apache_beam/runners/portability/fn_api_runner.py:115:0: C0301: Line too long 
(114/80) (line-too-long)
* Module apache_beam.transforms.core
apache_beam/transforms/core.py:1307:0: C0301: Line too long (116/80) 
(line-too-long)
apache_beam/transforms/core.py:2271:0: C0301: Line too long (95/80) 
(line-too-long)
apache_beam/transforms/core.py:2272:0: C0301: Line too long (90/80) 
(line-too-long)
* Module setup
setup.py:141:0: C0301: Line too long (81/80) (line-too-long)



```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367088
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:22
Start Date: 06/Jan/20 23:22
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571359333
 
 
   Run precommits
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367088)
Time Spent: 2h 20m  (was: 2h 10m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-9057) Make sure restriction_tracker.deferred_remainder is never called more than twice for one

[Boyuan Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyuan Zhang reassigned BEAM-9057:
--

Assignee: Boyuan Zhang

> Make sure restriction_tracker.deferred_remainder is never called more than 
> twice for one 
> ---
>
> Key: BEAM-9057
> URL: https://issues.apache.org/jira/browse/BEAM-9057
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Boyuan Zhang
>Assignee: Boyuan Zhang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9057) Make sure restriction_tracker.deferred_remainder is never called more than twice for one

[Boyuan Zhang (Jira)]

Boyuan Zhang created BEAM-9057:
--

 Summary: Make sure restriction_tracker.deferred_remainder is never 
called more than twice for one 
 Key: BEAM-9057
 URL: https://issues.apache.org/jira/browse/BEAM-9057
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-harness
Reporter: Boyuan Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367085
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:15
Start Date: 06/Jan/20 23:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571357595
 
 
   run precommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367085)
Time Spent: 2h 10m  (was: 2h)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8794) Projects should be handled by an IOPushDownRule before applying AggregateProjectMergeRule

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8794?focusedWorklogId=367084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367084
 ]

ASF GitHub Bot logged work on BEAM-8794:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:15
Start Date: 06/Jan/20 23:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10180: [BEAM-8794] 
Conditional aggregate project merge
URL: https://github.com/apache/beam/pull/10180#issuecomment-571357541
 
 
   Run sql postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367084)
Time Spent: 2h  (was: 1h 50m)

> Projects should be handled by an IOPushDownRule before applying 
> AggregateProjectMergeRule
> -
>
> Key: BEAM-8794
> URL: https://issues.apache.org/jira/browse/BEAM-8794
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> It is more efficient to push-down projected fields at an IO level (vs merging 
> with an Aggregate), when supported.
> When running queries like:
> {code:java}
> select SUM(score) as total_score from  group by name{code}
> Projects get merged with an aggregate, as a result Calc (after an 
> IOSourceRel) projects all fields and BeamIOPushDown rule does know what 
> fields can be dropped, thus not dropping any.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8844) [SQL] Create performance tests for BigQueryTable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8844?focusedWorklogId=367083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367083
 ]

ASF GitHub Bot logged work on BEAM-8844:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:14
Start Date: 06/Jan/20 23:14
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10226: [BEAM-8844] Add a 
new Jenkins job for SQL perf tests
URL: https://github.com/apache/beam/pull/10226#issuecomment-571357333
 
 
   This is failing a bunch of tests.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367083)
Time Spent: 8h  (was: 7h 50m)

> [SQL] Create performance tests for BigQueryTable
> 
>
> Key: BEAM-8844
> URL: https://issues.apache.org/jira/browse/BEAM-8844
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> They should measure read-time for:
>  * DIRECT_READ w/o push-down
>  * DIRECT_READ w/ push-down
>  * DEFAULT



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=367082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367082
 ]

ASF GitHub Bot logged work on BEAM-9027:


Author: ASF GitHub Bot
Created on: 06/Jan/20 23:13
Start Date: 06/Jan/20 23:13
Worklog Time Spent: 10m 
  Work Description: apilloud commented on pull request #10455: [BEAM-9027] 
[SQL] Fix ZetaSql unparsing bugs
URL: https://github.com/apache/beam/pull/10455
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367082)
Time Spent: 2h  (was: 1h 50m)

> [SQL] ZetaSQL unparsing should produce valid result
> ---
>
> Key: BEAM-9027
> URL: https://issues.apache.org/jira/browse/BEAM-9027
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql-zetasql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> * ZetaSQL does not recognize keyword INTERVAL
>  * Calcite cannot unparse RexNode back to bytes literal
>  * Calcite cannot unparse some floating point literals correctly
>  * Calcite cannot unparse some string literals correctly
>  * Calcite cannot unparse types correctly for CAST function



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-9056) Staging artifacts from environment

[Heejong Lee (Jira)]

Heejong Lee created BEAM-9056:
-

 Summary: Staging artifacts from environment
 Key: BEAM-9056
 URL: https://issues.apache.org/jira/browse/BEAM-9056
 Project: Beam
  Issue Type: Improvement
  Components: java-fn-execution
Reporter: Heejong Lee
Assignee: Heejong Lee


staging artifacts from artifact information embedded in environment proto.

detail: 
https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-9056) Staging artifacts from environment

[Heejong Lee (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Heejong Lee updated BEAM-9056:
--
Status: Open  (was: Triage Needed)

> Staging artifacts from environment
> --
>
> Key: BEAM-9056
> URL: https://issues.apache.org/jira/browse/BEAM-9056
> Project: Beam
>  Issue Type: Improvement
>  Components: java-fn-execution
>Reporter: Heejong Lee
>Assignee: Heejong Lee
>Priority: Major
>
> staging artifacts from artifact information embedded in environment proto.
> detail: 
> https://docs.google.com/document/d/1L7MJcfyy9mg2Ahfw5XPhUeBe-dyvAPMOYOiFA1-kAog



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (BEAM-7997) Create a single docker container for a pipeline instead of creating one for each worker.

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang closed BEAM-7997.
--
Fix Version/s: 2.15.0
   Resolution: Not A Problem

> Create a single docker container for a pipeline instead of creating one for 
> each worker.
> 
>
> Key: BEAM-7997
> URL: https://issues.apache.org/jira/browse/BEAM-7997
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
>  Labels: portability
> Fix For: 2.15.0
>
>
> With --direct_nun_workers option, current implementation creates a container 
> for each worker. We should change this behavior to creating a container for a 
> pipeline and handles multi workers inside of the container.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8780) Improve publish_docker_images.sh

[Mikhail Gryzykhin (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009217#comment-17009217
 ] 

Mikhail Gryzykhin commented on BEAM-8780:
-

Good sources
[https://dille.name/blog/2018/09/20/how-to-tag-docker-images-without-pulling-them/]
[https://stackoverflow.com/questions/48363572/how-to-add-tag-alias-on-docker-hub-without-pulling-an-image]

 

Rebuilding image is incorrect since version of compiler might change.

> Improve publish_docker_images.sh
> 
>
> Key: BEAM-8780
> URL: https://issues.apache.org/jira/browse/BEAM-8780
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Mark Liu
>Assignee: Hannah Jiang
>Priority: Major
>
> Few improvement places:
>  * The script builds new image for publishing. Better to reuse existing 
> staged rc images.
>  * This script ask input that’s unrelated to publish (i.e rc #, gpg key...)
>  * This script first run unrelated staging steps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Closed] (BEAM-8160) Add instructions about how to set FnApi multi-threads/processes

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang closed BEAM-8160.
--

> Add instructions about how to set FnApi multi-threads/processes
> ---
>
> Key: BEAM-8160
> URL: https://issues.apache.org/jira/browse/BEAM-8160
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add instructions to Beam site or Beam wiki for easy discovery.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-8160) Add instructions about how to set FnApi multi-threads/processes

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang resolved BEAM-8160.

Resolution: Fixed

> Add instructions about how to set FnApi multi-threads/processes
> ---
>
> Key: BEAM-8160
> URL: https://issues.apache.org/jira/browse/BEAM-8160
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Add instructions to Beam site or Beam wiki for easy discovery.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367020
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 06/Jan/20 22:06
Start Date: 06/Jan/20 22:06
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571336283
 
 
   Run Dataflow ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367020)
Time Spent: 8h 40m  (was: 8.5h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=367018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367018
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 06/Jan/20 22:06
Start Date: 06/Jan/20 22:06
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571336209
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367018)
Time Spent: 8.5h  (was: 8h 20m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9041) SchemaCoder equals should not rely on from/toRowFunction equality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9041?focusedWorklogId=367017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367017
 ]

ASF GitHub Bot logged work on BEAM-9041:


Author: ASF GitHub Bot
Created on: 06/Jan/20 22:00
Start Date: 06/Jan/20 22:00
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #10492: [BEAM-9041, 
BEAM-9042] SchemaCoder equals should not rely on from/toRowFunction equality
URL: https://github.com/apache/beam/pull/10492#issuecomment-571333837
 
 
   This is not a revert. Previous version did not compare from/toRow functions 
for equality. Do you have any suggestion on how to compare both functions? It 
is not really clear to me how to do so in particular for functions with no 
state.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367017)
Time Spent: 50m  (was: 40m)

> SchemaCoder equals should not rely on from/toRowFunction equality
> -
>
> Key: BEAM-9041
> URL: https://issues.apache.org/jira/browse/BEAM-9041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.18.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> SchemaCoder equals implementation relies on SerializableFunction equals 
> method, this is error-prone because users rarely implement the equals method 
> for a SerializableFunction. One alternative would be to rely on bytes 
> equality for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9041) SchemaCoder equals should not rely on from/toRowFunction equality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9041?focusedWorklogId=367015=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367015
 ]

ASF GitHub Bot logged work on BEAM-9041:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:47
Start Date: 06/Jan/20 21:47
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10492: [BEAM-9041, 
BEAM-9042] SchemaCoder equals should not rely on from/toRowFunction equality
URL: https://github.com/apache/beam/pull/10492#discussion_r363497491
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
 ##
 @@ -431,7 +431,20 @@ public static GenericRecord toGenericRecord(
*/
   public static SerializableFunction 
getRowToGenericRecordFunction(
   @Nullable org.apache.avro.Schema avroSchema) {
-return g -> toGenericRecord(g, avroSchema);
+return new RowToGenericRecordFn(avroSchema);
+  }
+
+  private static class RowToGenericRecordFn implements 
SerializableFunction {
+private final transient org.apache.avro.Schema avroSchema;
+
+RowToGenericRecordFn(@Nullable org.apache.avro.Schema avroSchema) {
+  this.avroSchema = avroSchema;
+}
+
+@Override
+public GenericRecord apply(Row input) {
+  return toGenericRecord(input, avroSchema);
+}
 
 Review comment:
   This is a second different issue about capture of Avro schema on 
serialization (the key change is the transient part) so not really related to 
equals. As explained above I put both together because I use equality to 
validate the roundtrip of serialization/deserialization. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367015)
Time Spent: 40m  (was: 0.5h)

> SchemaCoder equals should not rely on from/toRowFunction equality
> -
>
> Key: BEAM-9041
> URL: https://issues.apache.org/jira/browse/BEAM-9041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.18.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> SchemaCoder equals implementation relies on SerializableFunction equals 
> method, this is error-prone because users rarely implement the equals method 
> for a SerializableFunction. One alternative would be to rely on bytes 
> equality for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-9041) SchemaCoder equals should not rely on from/toRowFunction equality

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9041?focusedWorklogId=367013=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367013
 ]

ASF GitHub Bot logged work on BEAM-9041:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:44
Start Date: 06/Jan/20 21:44
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #10492: [BEAM-9041, 
BEAM-9042] SchemaCoder equals should not rely on from/toRowFunction equality
URL: https://github.com/apache/beam/pull/10492#discussion_r363497491
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
 ##
 @@ -431,7 +431,20 @@ public static GenericRecord toGenericRecord(
*/
   public static SerializableFunction 
getRowToGenericRecordFunction(
   @Nullable org.apache.avro.Schema avroSchema) {
-return g -> toGenericRecord(g, avroSchema);
+return new RowToGenericRecordFn(avroSchema);
+  }
+
+  private static class RowToGenericRecordFn implements 
SerializableFunction {
+private final transient org.apache.avro.Schema avroSchema;
+
+RowToGenericRecordFn(@Nullable org.apache.avro.Schema avroSchema) {
+  this.avroSchema = avroSchema;
+}
+
+@Override
+public GenericRecord apply(Row input) {
+  return toGenericRecord(input, avroSchema);
+}
 
 Review comment:
   This is a second different issue about capture of Avro schema on 
serialization so not really related to equals. As explained above I put both 
together because I use equality to validate the roundtrip of 
serialization/deserialization.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367013)
Time Spent: 0.5h  (was: 20m)

> SchemaCoder equals should not rely on from/toRowFunction equality
> -
>
> Key: BEAM-9041
> URL: https://issues.apache.org/jira/browse/BEAM-9041
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
> Fix For: 2.18.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> SchemaCoder equals implementation relies on SerializableFunction equals 
> method, this is error-prone because users rarely implement the equals method 
> for a SerializableFunction. One alternative would be to rely on bytes 
> equality for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8676) Beam Dependency Update Request: com.google.api:gax-grpc

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8676?focusedWorklogId=367008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367008
 ]

ASF GitHub Bot logged work on BEAM-8676:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:38
Start Date: 06/Jan/20 21:38
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10438: [BEAM-8676] 
sdks/java: gax and grpc upgrades
URL: https://github.com/apache/beam/pull/10438#issuecomment-571323951
 
 
   https://gist.github.com/suztomo/9af333b78b157d6a829d59ac5353e01d
   
   ```
   Linkage Error on com.google.auth:google-auth-library-oauth2-http:0.12.0 x 
com.google.api:gax:1.52.0
   
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountCredentials's method getQuotaProjectId() 
is not found (type: SYMBOL_NOT_FOUND)
 com.google.api.gax.core.GoogleCredentialsProvider 
(com.google.api:gax:1.52.0)
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials$Builder's method 
setQuotaProjectId(String arg1) is not found (type: SYMBOL_NOT_FOUND)
 com.google.api.gax.core.GoogleCredentialsProvider 
(com.google.api:gax:1.52.0)
   ```
   
   This PR needs https://github.com/apache/beam/pull/10508 first.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367008)
Time Spent: 10h 40m  (was: 10.5h)

> Beam Dependency Update Request: com.google.api:gax-grpc
> ---
>
> Key: BEAM-8676
> URL: https://issues.apache.org/jira/browse/BEAM-8676
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:38:32.410774 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:03:23.809273 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:08:16.165687 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:07:17.894174 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.51.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:07:26.564886 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:02:53.791811 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:06:28.934655 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  

[jira] [Work logged] (BEAM-8676) Beam Dependency Update Request: com.google.api:gax-grpc

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8676?focusedWorklogId=367005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367005
 ]

ASF GitHub Bot logged work on BEAM-8676:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:33
Start Date: 06/Jan/20 21:33
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10438: [BEAM-8676] 
sdks/java: gax and grpc upgrades
URL: https://github.com/apache/beam/pull/10438#issuecomment-571323951
 
 
   ```
   Linkage Error on com.google.auth:google-auth-library-oauth2-http:0.12.0 x 
com.google.api:gax:1.52.0
   
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountCredentials's method getQuotaProjectId() 
is not found (type: SYMBOL_NOT_FOUND)
 com.google.api.gax.core.GoogleCredentialsProvider 
(com.google.api:gax:1.52.0)
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials$Builder's method 
setQuotaProjectId(String arg1) is not found (type: SYMBOL_NOT_FOUND)
 com.google.api.gax.core.GoogleCredentialsProvider 
(com.google.api:gax:1.52.0)
   ```
   
   This PR needs https://github.com/apache/beam/pull/10508 first.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367005)
Time Spent: 10.5h  (was: 10h 20m)

> Beam Dependency Update Request: com.google.api:gax-grpc
> ---
>
> Key: BEAM-8676
> URL: https://issues.apache.org/jira/browse/BEAM-8676
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:38:32.410774 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:03:23.809273 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:08:16.165687 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:07:17.894174 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.51.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:07:26.564886 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:02:53.791811 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:06:28.934655 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> 

[jira] [Work logged] (BEAM-8676) Beam Dependency Update Request: com.google.api:gax-grpc

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8676?focusedWorklogId=367004=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367004
 ]

ASF GitHub Bot logged work on BEAM-8676:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:32
Start Date: 06/Jan/20 21:32
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10438: [BEAM-8676] 
sdks/java: gax and grpc upgrades
URL: https://github.com/apache/beam/pull/10438#issuecomment-571323951
 
 
   ```
   Linkage Error on com.google.auth:google-auth-library-oauth2-http:0.12.0 x 
com.google.api:gax:1.52.0
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountCredentials's method getQuotaProjectId() 
is not found (type: SYMBOL_NOT_FOUND)
   com.google.api.gax.core.GoogleCredentialsProvider (com.google.api:gax:1.52.0)
   (com.google.auth:google-auth-library-oauth2-http:0.12.0) 
com.google.auth.oauth2.ServiceAccountJwtAccessCredentials$Builder's method 
setQuotaProjectId(String arg1) is not found (type: SYMBOL_NOT_FOUND)
   com.google.api.gax.core.GoogleCredentialsProvider (com.google.api:gax:1.52.0)
   ```
   
   This PR needs https://github.com/apache/beam/pull/10508 first.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367004)
Time Spent: 10h 20m  (was: 10h 10m)

> Beam Dependency Update Request: com.google.api:gax-grpc
> ---
>
> Key: BEAM-8676
> URL: https://issues.apache.org/jira/browse/BEAM-8676
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:38:32.410774 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:03:23.809273 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:08:16.165687 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.50.1 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:07:17.894174 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.51.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:07:26.564886 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:02:53.791811 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:06:28.934655 
> -
> Please consider upgrading the dependency com.google.api:gax-grpc. 
> The current version is 1.38.0. The latest version is 1.52.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> 

[jira] [Work logged] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8685?focusedWorklogId=367000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-367000
 ]

ASF GitHub Bot logged work on BEAM-8685:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:32
Start Date: 06/Jan/20 21:32
Worklog Time Spent: 10m 
  Work Description: suztomo commented on issue #10508: [BEAM-8685] 
sdks/java: google_auth_version 0.19.0
URL: https://github.com/apache/beam/pull/10508#issuecomment-571323812
 
 
   R: @lukecwik, @kennknowles 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 367000)
Time Spent: 20m  (was: 10m)

> Beam Dependency Update Request: 
> com.google.auth:google-auth-library-oauth2-http
> ---
>
> Key: BEAM-8685
> URL: https://issues.apache.org/jira/browse/BEAM-8685
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
>  - 2019-11-15 19:39:27.324449 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:03.844285 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:10:30.864371 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:09:38.646889 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:09:39.967215 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:10.534268 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:08:45.451960 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8559) Run Dataflow Nexmark suites with Java 11

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8559?focusedWorklogId=366966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366966
 ]

ASF GitHub Bot logged work on BEAM-8559:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:12
Start Date: 06/Jan/20 21:12
Worklog Time Spent: 10m 
  Work Description: stale[bot] commented on issue #9994: [BEAM-8559] Run 
Nexmark Dataflow suites with Java 11
URL: https://github.com/apache/beam/pull/9994#issuecomment-571316635
 
 
   This pull request has been marked as stale due to 60 days of inactivity. It 
will be closed in 1 week if no further activity occurs. If you think that’s 
incorrect or this pull request requires a review, please simply write any 
comment. If closed, you can revive the PR at any time and @mention a reviewer 
or discuss it on the d...@beam.apache.org list. Thank you for your 
contributions.
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366966)
Time Spent: 50m  (was: 40m)

> Run Dataflow Nexmark suites with Java 11
> 
>
> Key: BEAM-8559
> URL: https://issues.apache.org/jira/browse/BEAM-8559
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing-nexmark
>Reporter: Lukasz Gajowy
>Assignee: Michal Walenia
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This task is similar to https://issues.apache.org/jira/browse/BEAM-6936.
> The goal is to run Nexmark suites with Java 11 but compile with java 8 to 
> verify compatibility. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=366953=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366953
 ]

ASF GitHub Bot logged work on BEAM-8932:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:02
Start Date: 06/Jan/20 21:02
Worklog Time Spent: 10m 
  Work Description: dpcollins-google commented on issue #10474: [BEAM-8932] 
[BEAM-9036] Revert reverted commit to use PubsubMessage as the canonical type 
in beam client
URL: https://github.com/apache/beam/pull/10474#issuecomment-571312941
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366953)
Time Spent: 7.5h  (was: 7h 20m)

> Expose complete Cloud Pub/Sub messages through PubsubIO API
> ---
>
> Key: BEAM-8932
> URL: https://issues.apache.org/jira/browse/BEAM-8932
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Daniel Collins
>Assignee: Daniel Collins
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> The PubsubIO API only exposes a subset of the fields in the underlying 
> PubsubMessage protocol buffer. To accomodate future feature changes as well 
> as for greater compatability with code using the Cloud Pub/Sub apis, a method 
> to read and write these protocol messages should be exposed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=366952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366952
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 06/Jan/20 21:01
Start Date: 06/Jan/20 21:01
Worklog Time Spent: 10m 
  Work Description: rehmanmuradali commented on pull request #10316: 
[BEAM-6857] Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316#discussion_r363480976
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WindmillTimerInternals.java
 ##
 @@ -94,8 +94,13 @@ public void setTimer(TimerData timerKey) {
 
   @Override
   public void setTimer(
-  StateNamespace namespace, String timerId, Instant timestamp, TimeDomain 
timeDomain) {
-timers.put(timerId, namespace, TimerData.of(timerId, namespace, timestamp, 
timeDomain));
+  StateNamespace namespace,
+  String timerId,
+  String timerFamilyId,
+  Instant timestamp,
+  TimeDomain timeDomain) {
+timers.put(
+timerId, namespace, TimerData.of(timerId, timerFamilyId, namespace, 
timestamp, timeDomain));
 
 Review comment:
   @reuvenlax  can you please guide us about how can we achieve this?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366952)
Time Spent: 8h 20m  (was: 8h 10m)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8932) Expose complete Cloud Pub/Sub messages through PubsubIO API

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8932?focusedWorklogId=366949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366949
 ]

ASF GitHub Bot logged work on BEAM-8932:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:59
Start Date: 06/Jan/20 20:59
Worklog Time Spent: 10m 
  Work Description: dpcollins-google commented on issue #10474: [BEAM-8932] 
[BEAM-9036] Revert reverted commit to use PubsubMessage as the canonical type 
in beam client
URL: https://github.com/apache/beam/pull/10474#issuecomment-571312036
 
 
   Run PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366949)
Time Spent: 7h 20m  (was: 7h 10m)

> Expose complete Cloud Pub/Sub messages through PubsubIO API
> ---
>
> Key: BEAM-8932
> URL: https://issues.apache.org/jira/browse/BEAM-8932
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Daniel Collins
>Assignee: Daniel Collins
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> The PubsubIO API only exposes a subset of the fields in the underlying 
> PubsubMessage protocol buffer. To accomodate future feature changes as well 
> as for greater compatability with code using the Cloud Pub/Sub apis, a method 
> to read and write these protocol messages should be exposed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-6857) Support dynamic timers

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=366936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366936
 ]

ASF GitHub Bot logged work on BEAM-6857:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:47
Start Date: 06/Jan/20 20:47
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #10506: [BEAM-6857] 
OnTimer/SetTimer Signature Updates
URL: https://github.com/apache/beam/pull/10506#issuecomment-571307692
 
 
   retest this please
   
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366936)
Time Spent: 8h 10m  (was: 8h)

> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified 
> in the DoFn. The user must provide a separate callback method per timer. For 
> example:
>  
> {code:java}
> DoFn()
> {   
>   @TimerId("timer1") 
>   private final TimerSpec timer1 = TimerSpecs.timer(...);   
>   @TimerId("timer2") 
>   private final TimerSpec timer2 = TimerSpecs.timer(...);                 
>   .. set timers in processElement    
>   @OnTimer("timer1") 
>   public void onTimer1() { .}
>   @OnTimer("timer2") 
>   public void onTimer2() {}
> }
> {code}
>  
> However there are many cases where the user does not know the set of timers 
> statically when writing their code. This happens when the timer tag should be 
> based on the data. It also happens when writing a DSL on top of Beam, where 
> the DSL author has to create DoFns but does not know statically which timers 
> their users will want to set (e.g. Scio).
>  
> The goal is to support dynamic timers. Something as follows;
>  
> {code:java}
> DoFn() 
> {
>   @TimerId("timer") 
>   private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
>   @ProcessElement process(@TimerId("timer") DynamicTimer timer)
>   {
>        timer.set("tag1'", ts);       
>timer.set("tag2", ts);     
>   }
>   @OnTimer("timer") 
>   public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8685?focusedWorklogId=366934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366934
 ]

ASF GitHub Bot logged work on BEAM-8685:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:37
Start Date: 06/Jan/20 20:37
Worklog Time Spent: 10m 
  Work Description: suztomo commented on pull request #10508: [BEAM-8685] 
google_auth_version 0.19.0
URL: https://github.com/apache/beam/pull/10508
 
 
   Upgrading the following two libraries to 0.19.0.
   
   ```
   google_auth_library_credentials : 
"com.google.auth:google-auth-library-credentials:$google_auth_version",
   google_auth_library_oauth2_http : 
"com.google.auth:google-auth-library-oauth2-http:$google_auth_version",
   ```
   
   # Linkage Check
   
   ## sdks-java-extensions-google-cloud-platform-core
   - before:
   - after: https://gist.github.com/suztomo/be1388f48fb13c75813992cab9400067
   
   ## beam-runners-google-cloud-dataflow-java
   - 
   - after: https://gist.github.com/suztomo/2c0c264f55e44ae6ced36cf6cc4a1114
   
   ## https://gist.github.com/suztomo/66c4494daab25365f0407a156f2a311d
   - before:
   - after: https://gist.github.com/suztomo/66c4494daab25365f0407a156f2a311d
   
   ## beam-examples-java
   - before: 
   - after: https://gist.github.com/suztomo/64f9ae4aac13c9c1add0a0c73ac17b17
   
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more 
tips on [how to make review process 
smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 

[jira] [Work logged] (BEAM-8951) Stop using nose in load tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8951?focusedWorklogId=366924=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366924
 ]

ASF GitHub Bot logged work on BEAM-8951:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:09
Start Date: 06/Jan/20 20:09
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10435: [BEAM-8951] 
Stop using nose in load tests
URL: https://github.com/apache/beam/pull/10435#discussion_r363461573
 
 

 ##
 File path: sdks/python/apache_beam/testing/load_tests/load_test.py
 ##
 @@ -17,19 +17,50 @@
 from __future__ import absolute_import
 
 import json
-import logging
-import unittest
 
 from apache_beam.metrics import MetricsFilter
 from apache_beam.testing.load_tests.load_test_metrics_utils import 
MetricsReader
 from apache_beam.testing.test_pipeline import TestPipeline
 
 
-class LoadTest(unittest.TestCase):
-  def parseTestPipelineOptions(self, options=None):
+class LoadTest(object):
+  def __init__(self):
+self.pipeline = TestPipeline(is_integration_test=True)
 
 Review comment:
   If we proceed with this change, do we still need to use TestPipeline as 
opposed to regular pipeline?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366924)
Time Spent: 3.5h  (was: 3h 20m)

> Stop using nose in load tests
> -
>
> Key: BEAM-8951
> URL: https://issues.apache.org/jira/browse/BEAM-8951
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> The community is considering moving away from nose to pytest: 
> https://issues.apache.org/jira/browse/BEAM-3713. We should change the way of 
> running Python load tests: instead of being subclasses of 
> `unittest.TestCase`, they could be plain Python scripts, just like wordcount 
> examples. This will bring one additional benefit: _LOAD_TEST_ENABLED_ guard 
> will be no longer needed and could be safely removed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8951) Stop using nose in load tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8951?focusedWorklogId=366922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366922
 ]

ASF GitHub Bot logged work on BEAM-8951:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:05
Start Date: 06/Jan/20 20:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10435: [BEAM-8951] 
Stop using nose in load tests
URL: https://github.com/apache/beam/pull/10435#discussion_r363452643
 
 

 ##
 File path: 
sdks/python/apache_beam/testing/load_tests/streaming/group_by_key_streaming_test.py
 ##
 @@ -62,12 +56,11 @@ def _add_argparse_args(cls, parser):
 parser.add_argument('--test-pipeline-options')
 
 
-@unittest.skipIf(not load_test_enabled, 'Enabled only for phase triggering.')
 class GroupByKeyStreamingTest(LoadTest):
   ID_LABEL = 'id'
 
-  def setUp(self):
-super(GroupByKeyStreamingTest, self).setUp()
+  def __init__(self):
+super(GroupByKeyStreamingTest, self).__init__()
 
 Review comment:
   Does running these tests on Dataflow runner require --save_main_session? 
Heads-up that there is https://issues.apache.org/jira/browse/BEAM-6158 which 
may affect the execution on Python 3. Please check that we can run these tests 
on Python 3 on Dataflow (you can check on any test that is supported by 
Dataflow Runner).
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366922)
Time Spent: 3h 20m  (was: 3h 10m)

> Stop using nose in load tests
> -
>
> Key: BEAM-8951
> URL: https://issues.apache.org/jira/browse/BEAM-8951
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The community is considering moving away from nose to pytest: 
> https://issues.apache.org/jira/browse/BEAM-3713. We should change the way of 
> running Python load tests: instead of being subclasses of 
> `unittest.TestCase`, they could be plain Python scripts, just like wordcount 
> examples. This will bring one additional benefit: _LOAD_TEST_ENABLED_ guard 
> will be no longer needed and could be safely removed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8951) Stop using nose in load tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8951?focusedWorklogId=366923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366923
 ]

ASF GitHub Bot logged work on BEAM-8951:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:05
Start Date: 06/Jan/20 20:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10435: [BEAM-8951] 
Stop using nose in load tests
URL: https://github.com/apache/beam/pull/10435#discussion_r363434067
 
 

 ##
 File path: sdks/python/apache_beam/testing/load_tests/combine_test.py
 ##
 @@ -59,62 +55,10 @@
 --input_options=\'
   {"num_records": 1,
   "key_size": 1,
-  "value_size":1,
-  "bundle_size_distribution_type": "const",
-  "bundle_size_distribution_param": 1,
-  "force_initial_num_bundles": 1}\'
---runner=DirectRunner
---fanout=1
---top_count=1000' \
+  "value_size": 1}\'
 
 Review comment:
   The gradle command is not copy-pasteable. Python command is. Did not check 
commands in other files.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366923)

> Stop using nose in load tests
> -
>
> Key: BEAM-8951
> URL: https://issues.apache.org/jira/browse/BEAM-8951
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The community is considering moving away from nose to pytest: 
> https://issues.apache.org/jira/browse/BEAM-3713. We should change the way of 
> running Python load tests: instead of being subclasses of 
> `unittest.TestCase`, they could be plain Python scripts, just like wordcount 
> examples. This will bring one additional benefit: _LOAD_TEST_ENABLED_ guard 
> will be no longer needed and could be safely removed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8951) Stop using nose in load tests

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8951?focusedWorklogId=366921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366921
 ]

ASF GitHub Bot logged work on BEAM-8951:


Author: ASF GitHub Bot
Created on: 06/Jan/20 20:05
Start Date: 06/Jan/20 20:05
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #10435: [BEAM-8951] 
Stop using nose in load tests
URL: https://github.com/apache/beam/pull/10435#discussion_r363449250
 
 

 ##
 File path: sdks/python/apache_beam/testing/load_tests/load_test.py
 ##
 @@ -17,19 +17,50 @@
 from __future__ import absolute_import
 
 import json
-import logging
-import unittest
 
 from apache_beam.metrics import MetricsFilter
 from apache_beam.testing.load_tests.load_test_metrics_utils import 
MetricsReader
 from apache_beam.testing.test_pipeline import TestPipeline
 
 
-class LoadTest(unittest.TestCase):
-  def parseTestPipelineOptions(self, options=None):
+class LoadTest(object):
+  def __init__(self):
+self.pipeline = TestPipeline(is_integration_test=True)
+self.input_options = json.loads(self.pipeline.get_option('input_options')
+or '{}')
+self.project_id = self.pipeline.get_option('project')
+self.metrics_namespace = self.pipeline.get_option('metrics_table')
+self._metrics_monitor = MetricsReader(
+publish_to_bq=self.pipeline.get_option('publish_to_big_query') ==
+'true',
 
 Review comment:
   Let's strengthen this to check for possible misspellings. `publish_to_bq = 
bool(['false', 
'true'].index(self.pipeline.get_option('publish_to_big_query').lower()))`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366921)
Time Spent: 3h 20m  (was: 3h 10m)

> Stop using nose in load tests
> -
>
> Key: BEAM-8951
> URL: https://issues.apache.org/jira/browse/BEAM-8951
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The community is considering moving away from nose to pytest: 
> https://issues.apache.org/jira/browse/BEAM-3713. We should change the way of 
> running Python load tests: instead of being subclasses of 
> `unittest.TestCase`, they could be plain Python scripts, just like wordcount 
> examples. This will bring one additional benefit: _LOAD_TEST_ENABLED_ guard 
> will be no longer needed and could be safely removed



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366910
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363450074
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
+   * @param requestObserver The outbound request observer this client uses to 
send new status
+   * requests to its corresponding SDK Harness.
+   * @return {@link WorkerStatusClient}
+   */
+  public static WorkerStatusClient forRequestObserver(
+  String workerId, StreamObserver requestObserver) {
+return new WorkerStatusClient(workerId, requestObserver);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness. A random id will
+   * be used to specify the request_id field.
+   *
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  public CompletableFuture getWorkerStatus() {
+WorkerStatusRequest request =
+WorkerStatusRequest.newBuilder().setId(idGenerator.getId()).build();
+return getWorkerStatus(request);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness with request.
+   *
+   * @param request WorkerStatusRequest to be sent to SDK Harness.
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  CompletableFuture getWorkerStatus(WorkerStatusRequest 
request) {
+CompletableFuture future = new CompletableFuture<>();
+this.responseQueue.put(request.getId(), future);
+this.requestReceiver.onNext(request);
+return future;
+  }
+
+  /**
+   * Set up a deregister call back function for cleaning up connected client 
cache.
+   *
+   * @param deregisterCallback Consumer that takes worker id as param for 
deregister itself from
+   * connected client cache when close is called.
+   */
+  public void setDeregisterCallback(Consumer deregisterCallback) {
+this.deregisterCallback = 

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366909=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366909
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363447657
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
 
 Review comment:
   ```suggestion
  * Create new status api client with SDK harness worker id and request 
observer.
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366909)
Time Spent: 10h 40m  (was: 10.5h)

> Implement FnService for status api in Dataflow runner
> -
>
> Key: BEAM-8624
> URL: https://issues.apache.org/jira/browse/BEAM-8624
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366911
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363450298
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
+   * @param requestObserver The outbound request observer this client uses to 
send new status
+   * requests to its corresponding SDK Harness.
+   * @return {@link WorkerStatusClient}
+   */
+  public static WorkerStatusClient forRequestObserver(
+  String workerId, StreamObserver requestObserver) {
+return new WorkerStatusClient(workerId, requestObserver);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness. A random id will
+   * be used to specify the request_id field.
+   *
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  public CompletableFuture getWorkerStatus() {
+WorkerStatusRequest request =
+WorkerStatusRequest.newBuilder().setId(idGenerator.getId()).build();
+return getWorkerStatus(request);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness with request.
+   *
+   * @param request WorkerStatusRequest to be sent to SDK Harness.
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  CompletableFuture getWorkerStatus(WorkerStatusRequest 
request) {
+CompletableFuture future = new CompletableFuture<>();
+this.responseQueue.put(request.getId(), future);
+this.requestReceiver.onNext(request);
+return future;
+  }
+
+  /**
+   * Set up a deregister call back function for cleaning up connected client 
cache.
+   *
+   * @param deregisterCallback Consumer that takes worker id as param for 
deregister itself from
+   * connected client cache when close is called.
+   */
+  public void setDeregisterCallback(Consumer deregisterCallback) {
+this.deregisterCallback = 

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366914
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r36345
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
+   * @param requestObserver The outbound request observer this client uses to 
send new status
+   * requests to its corresponding SDK Harness.
+   * @return {@link WorkerStatusClient}
+   */
+  public static WorkerStatusClient forRequestObserver(
+  String workerId, StreamObserver requestObserver) {
+return new WorkerStatusClient(workerId, requestObserver);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness. A random id will
+   * be used to specify the request_id field.
+   *
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  public CompletableFuture getWorkerStatus() {
+WorkerStatusRequest request =
+WorkerStatusRequest.newBuilder().setId(idGenerator.getId()).build();
+return getWorkerStatus(request);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness with request.
+   *
+   * @param request WorkerStatusRequest to be sent to SDK Harness.
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  CompletableFuture getWorkerStatus(WorkerStatusRequest 
request) {
+CompletableFuture future = new CompletableFuture<>();
+this.responseQueue.put(request.getId(), future);
+this.requestReceiver.onNext(request);
+return future;
+  }
+
+  /**
+   * Set up a deregister call back function for cleaning up connected client 
cache.
+   *
+   * @param deregisterCallback Consumer that takes worker id as param for 
deregister itself from
+   * connected client cache when close is called.
+   */
+  public void setDeregisterCallback(Consumer deregisterCallback) {
+this.deregisterCallback = 

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366915
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363445208
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366902
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363442048
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366916=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366916
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363447088
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366912
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363452319
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
 
 Review comment:
   `responseQueue` -> `pendingResponses`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366912)
Time Spent: 10h 50m  (was: 10h 40m)

> Implement FnService for status api in Dataflow runner
> -
>
> Key: BEAM-8624
> URL: https://issues.apache.org/jira/browse/BEAM-8624
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366907
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363444737
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366903
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363444574
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366905
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363447389
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
 
 Review comment:
   ```suggestion
* harness.
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366905)
Time Spent: 10h 10m  (was: 10h)

> Implement FnService for status api in Dataflow runner
> -
>
> Key: BEAM-8624
> URL: https://issues.apache.org/jira/browse/BEAM-8624
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366906
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363444348
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366913
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363447614
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
 
 Review comment:
   ```suggestion
  * @param workerId SDK harness worker id.
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366913)
Time Spent: 11h  (was: 10h 50m)

> Implement FnService for status api in Dataflow runner
> -
>
> Key: BEAM-8624
> URL: https://issues.apache.org/jira/browse/BEAM-8624
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Yichi Zhang
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 11h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8685) Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http

[Tomo Suzuki (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17009129#comment-17009129
 ] 

Tomo Suzuki commented on BEAM-8685:
---

Affected modules:

{noformat}
suztomo@suxtomo24:~/beam6$ find . -name 'build.gradle' |xargs grep -ir 
library.java.google_auth_library_credentials
./sdks/java/io/google-cloud-platform/build.gradle:  compile 
library.java.google_auth_library_credentials
./sdks/java/extensions/google-cloud-platform-core/build.gradle:  compile 
library.java.google_auth_library_credentials
./runners/google-cloud-dataflow-java/build.gradle:  compile 
library.java.google_auth_library_credentials
./examples/kotlin/build.gradle:  compile 
library.java.google_auth_library_credentials
./examples/java/build.gradle:  compile 
library.java.google_auth_library_credentials
suztomo@suxtomo24:~/beam6$ find . -name 'build.gradle' |xargs grep -ir 
library.java.google_auth_library_credentials^C
suztomo@suxtomo24:~/beam6$ 
suztomo@suxtomo24:~/beam6$ find . -name 'build.gradle' |xargs grep -ir 
library.java.google_auth_library_oauth2_http
./sdks/java/io/google-cloud-platform/build.gradle:  compile 
library.java.google_auth_library_oauth2_http
./sdks/java/extensions/google-cloud-platform-core/build.gradle:  compile 
library.java.google_auth_library_oauth2_http
./sdks/java/extensions/sql/datacatalog/build.gradle:  provided 
library.java.google_auth_library_oauth2_http
./runners/google-cloud-dataflow-java/build.gradle:  compile 
library.java.google_auth_library_oauth2_http
./examples/kotlin/build.gradle:  compile 
library.java.google_auth_library_oauth2_http
./examples/java/build.gradle:  compile 
library.java.google_auth_library_oauth2_http
{noformat}



> Beam Dependency Update Request: 
> com.google.auth:google-auth-library-oauth2-http
> ---
>
> Key: BEAM-8685
> URL: https://issues.apache.org/jira/browse/BEAM-8685
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Tomo Suzuki
>Priority: Major
>
>  - 2019-11-15 19:39:27.324449 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-11-19 21:05:03.844285 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-02 12:10:30.864371 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-09 12:09:38.646889 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.18.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-23 12:09:39.967215 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2019-12-30 14:05:10.534268 
> -
> Please consider upgrading the dependency 
> com.google.auth:google-auth-library-oauth2-http. 
> The current version is 0.12.0. The latest version is 0.19.0 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2020-01-06 12:08:45.451960 
> -
> Please consider upgrading the dependency 
> 

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366917
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363452483
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/WorkerStatusClient.java
 ##
 @@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.function.Consumer;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Client for handling requests and responses over Fn Worker Status Api 
between runner and SDK
+ * Harness.
+ */
+class WorkerStatusClient implements Closeable {
+
+  public static final Logger LOG = 
LoggerFactory.getLogger(WorkerStatusClient.class);
+  private final IdGenerator idGenerator = IdGenerators.incrementingLongs();
+  private final StreamObserver requestReceiver;
+  private final Map> 
responseQueue =
+  new ConcurrentHashMap<>();
+  private final String workerId;
+  private Consumer deregisterCallback;
+  private AtomicBoolean isClosed = new AtomicBoolean(false);
+
+  private WorkerStatusClient(String workerId, 
StreamObserver requestReceiver) {
+this.requestReceiver = 
SynchronizedStreamObserver.wrapping(requestReceiver);
+this.workerId = workerId;
+  }
+
+  /**
+   * Create new status api client with SDK Harness worker id and request 
observer.
+   *
+   * @param workerId SDK Harness worker id.
+   * @param requestObserver The outbound request observer this client uses to 
send new status
+   * requests to its corresponding SDK Harness.
+   * @return {@link WorkerStatusClient}
+   */
+  public static WorkerStatusClient forRequestObserver(
+  String workerId, StreamObserver requestObserver) {
+return new WorkerStatusClient(workerId, requestObserver);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness. A random id will
+   * be used to specify the request_id field.
+   *
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  public CompletableFuture getWorkerStatus() {
+WorkerStatusRequest request =
+WorkerStatusRequest.newBuilder().setId(idGenerator.getId()).build();
+return getWorkerStatus(request);
+  }
+
+  /**
+   * Get the latest sdk worker status from the client's corresponding SDK 
Harness with request.
+   *
+   * @param request WorkerStatusRequest to be sent to SDK Harness.
+   * @return {@link CompletableFuture} of the SDK Harness status response.
+   */
+  CompletableFuture getWorkerStatus(WorkerStatusRequest 
request) {
+CompletableFuture future = new CompletableFuture<>();
 
 Review comment:
   You have a concurrency race between `getWorkerStatus` adding to the 
`responseQueue` and `close()`
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

 

[jira] [Work logged] (BEAM-8624) Implement FnService for status api in Dataflow runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8624?focusedWorklogId=366904=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366904
 ]

ASF GitHub Bot logged work on BEAM-8624:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #10115: [BEAM-8624] 
Implement Worker Status FnService in Dataflow runner
URL: https://github.com/apache/beam/pull/10115#discussion_r363444678
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/status/BeamWorkerStatusGrpcService.java
 ##
 @@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.status;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentSkipListMap;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.WorkerStatusResponse;
+import 
org.apache.beam.model.fnexecution.v1.BeamFnWorkerStatusGrpc.BeamFnWorkerStatusImplBase;
+import org.apache.beam.model.pipeline.v1.Endpoints.ApiServiceDescriptor;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.HeaderAccessor;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.StreamObserver;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Strings;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A Fn Status service which can collect run-time status information from SDK 
Harnesses for
+ * debugging purpose.
+ */
+public class BeamWorkerStatusGrpcService extends BeamFnWorkerStatusImplBase 
implements FnService {
+
+  static final String DEFAULT_ERROR_RESPONSE = "Error getting status from SDK 
harness";
+
+  private static final Logger LOG = 
LoggerFactory.getLogger(BeamWorkerStatusGrpcService.class);
+  private static final long DEFAULT_CLIENT_CONNECTION_WAIT_TIME_SECONDS = 5;
+  private final HeaderAccessor headerAccessor;
+  private final Map> 
connectedClient =
+  new ConcurrentHashMap<>();
+
+  private BeamWorkerStatusGrpcService(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+this.headerAccessor = headerAccessor;
+LOG.info("Launched Beam Fn Status service at {}", apiServiceDescriptor);
+  }
+
+  /**
+   * Create new instance of {@link BeamWorkerStatusGrpcService}.
+   *
+   * @param apiServiceDescriptor describes the configuration for the endpoint 
the server will
+   * expose.
+   * @param headerAccessor headerAccessor gRPC header accessor used to obtain 
SDK harness worker id.
+   * @return {@link BeamWorkerStatusGrpcService}
+   */
+  public static BeamWorkerStatusGrpcService create(
+  ApiServiceDescriptor apiServiceDescriptor, HeaderAccessor 
headerAccessor) {
+return new BeamWorkerStatusGrpcService(apiServiceDescriptor, 
headerAccessor);
+  }
+
+  @Override
+  public void close() throws Exception {
+for (CompletableFuture clientFuture : 
connectedClient.values()) {
+  if (clientFuture.isDone()) {
+clientFuture.get().close();
+  }
+}
+connectedClient.clear();
+  }
+
+  @Override
+  public StreamObserver workerStatus(
+  StreamObserver requestObserver) {
+String workerId = headerAccessor.getSdkWorkerId();
+LOG.info("Beam Fn Status client connected with id {}", workerId);
+
+WorkerStatusClient fnApiStatusClient =
+WorkerStatusClient.forRequestObserver(workerId, requestObserver);
+fnApiStatusClient.setDeregisterCallback(this.connectedClient::remove);
+

[jira] [Work logged] (BEAM-9027) [SQL] ZetaSQL unparsing should produce valid result

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-9027?focusedWorklogId=366900=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366900
 ]

ASF GitHub Bot logged work on BEAM-9027:


Author: ASF GitHub Bot
Created on: 06/Jan/20 19:51
Start Date: 06/Jan/20 19:51
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #10455: [BEAM-9027] [SQL] 
Fix ZetaSql unparsing bugs
URL: https://github.com/apache/beam/pull/10455#issuecomment-571287258
 
 
   Run precommits
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 366900)
Time Spent: 1h 40m  (was: 1.5h)

> [SQL] ZetaSQL unparsing should produce valid result
> ---
>
> Key: BEAM-9027
> URL: https://issues.apache.org/jira/browse/BEAM-9027
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql-zetasql
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> * ZetaSQL does not recognize keyword INTERVAL
>  * Calcite cannot unparse RexNode back to bytes literal
>  * Calcite cannot unparse some floating point literals correctly
>  * Calcite cannot unparse some string literals correctly
>  * Calcite cannot unparse types correctly for CAST function



--
This message was sent by Atlassian Jira
(v8.3.4#803005)