[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311123
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 12/Sep/19 02:27
Start Date: 12/Sep/19 02:27
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on pull request #9491: 
[BEAM-5539] Upgrade google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311123)
Time Spent: 4h 10m  (was: 4h)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> -
> Please consider upgrading the 

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311122
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 12/Sep/19 02:26
Start Date: 12/Sep/19 02:26
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #9491: [BEAM-5539] 
Upgrade google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530637983
 
 
   Thanks. Merging.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311122)
Time Spent: 4h  (was: 3h 50m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> -
> 

[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=32=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-32
 ]

ASF GitHub Bot logged work on BEAM-8217:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:43
Start Date: 12/Sep/19 01:43
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #9551: [BEAM-8217] Encode 
str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9551#issuecomment-530629951
 
 
   LGTM, thank you.
   
   On Wed, Sep 11, 2019, 6:11 PM Ankur  wrote:
   
   > *@angoenka* commented on this pull request.
   > --
   >
   > In sdks/python/apache_beam/runners/worker/sdk_worker_main.py
   > :
   >
   > > @@ -74,7 +74,7 @@ def do_GET(self):  # pylint: disable=invalid-name
   >  self.end_headers()
   >
   >  for line in StatusServer.get_thread_dump():
   > -  self.wfile.write(line)
   > +  self.wfile.write(str.encode(line, 'UTF-8'))
   >
   >
   >1. Done
   >2. Done
   >3. checked with 3.5
   >4. TODO
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 32)
Time Spent: 20m  (was: 10m)

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311107
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:22
Start Date: 12/Sep/19 01:22
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530625707
 
 
   run python 2 postcommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311107)
Time Spent: 3h 50m  (was: 3h 40m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> -
>

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311105
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:21
Start Date: 12/Sep/19 01:21
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530625337
 
 
   Run Portable_Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311105)
Time Spent: 3.5h  (was: 3h 20m)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-26 12:09:57.098788 
> -
>  

[jira] [Work logged] (BEAM-5539) Beam Dependency Update Request: google-cloud-pubsub

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5539?focusedWorklogId=311106=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311106
 ]

ASF GitHub Bot logged work on BEAM-5539:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:21
Start Date: 12/Sep/19 01:21
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9491: [BEAM-5539] Upgrade 
google-cloud-pubsub and bigtable packages.
URL: https://github.com/apache/beam/pull/9491#issuecomment-530625559
 
 
   I hacked bigtableio_it.py locally to fix pickling issues and ran it on 
direct and Dataflow runners successfully.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311106)
Time Spent: 3h 40m  (was: 3.5h)

> Beam Dependency Update Request: google-cloud-pubsub
> ---
>
> Key: BEAM-5539
> URL: https://issues.apache.org/jira/browse/BEAM-5539
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies, io-py-gcp
>Reporter: Beam JIRA Bot
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
>  - 2018-10-01 19:17:59.633423 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-08 12:11:22.339342 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-15 12:10:17.795498 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.26.0. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-22 12:10:36.369849 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-10-29 12:12:47.703482 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-05 12:11:03.025934 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-12 12:11:07.165089 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
>  - 2018-11-19 12:11:33.666549 
> -
> Please consider upgrading the dependency google-cloud-pubsub. 
> The current version is 0.35.4. The latest version is 0.38.0 
> cc: [~markflyhigh], 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache.org/contribute/dependencies/]for more information. 
> Do Not Modify The Description Above. 
> 

[jira] [Work logged] (BEAM-8098) BigQueryIO needs documentation on how data types in BigQuery and in Beam SDK correspond

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8098?focusedWorklogId=311104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311104
 ]

ASF GitHub Bot logged work on BEAM-8098:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:19
Start Date: 12/Sep/19 01:19
Worklog Time Spent: 10m 
  Work Description: robinyqiu commented on pull request #9555: [BEAM-8098] 
Improve documentation on BigQueryIO
URL: https://github.com/apache/beam/pull/9555
 
 
   While working on https://github.com/apache/beam/pull/9144, I realized there 
is a gap in BigQueryIO documentation on mapping between data types defined in 
BigQuery and in Beam SDK.
   
   For example, if a user reads a BYTES field from BigQuery into Beam, it will 
be represented as java.nio.ByteBuffer type in Beam Java SDK. The user will need 
to do an explicit type cast to ByteBuffer in order to use the data, but there 
is no easy way the user can know which type they should cast to, unless digging 
into BigQueryIO's implementation (Java - Avro - BigQuery).
   
   r: @pabloem 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 

[jira] [Work logged] (BEAM-7911) test_corrupted_file test flaky

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7911?focusedWorklogId=311102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311102
 ]

ASF GitHub Bot logged work on BEAM-7911:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:15
Start Date: 12/Sep/19 01:15
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #9554: [BEAM-7911] Fix 
flakiness in avroio_test.py
URL: https://github.com/apache/beam/pull/9554#issuecomment-530624340
 
 
   > > This LGTM.
   > > If the condition was never true, why the test was flaky? Should not it 
always fail?
   > 
   > It was intentionally corrupting data by replacing the last byte. It was 
failing to corrupt the data in 1 out of 256 cases, where the last byte was 
already 'B'.
   
   Ack. Thank you for the fix. Much cleaner.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311102)
Time Spent: 40m  (was: 0.5h)

> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24 
> 11:22:24 
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24 
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
>  line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync 
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute 
> 'exception'
> 11:22:24  >> begin captured logging << 
> 
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern: 
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging << 
> -
> 11:22:24 
> 11:22:24 
> --
> 11:22:24 XML: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24 
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24 
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
>  setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary 
> 
> 11:22:24 ERROR:   py36: commands failed
> 11:22:24 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7911) test_corrupted_file test flaky

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7911?focusedWorklogId=311101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311101
 ]

ASF GitHub Bot logged work on BEAM-7911:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:13
Start Date: 12/Sep/19 01:13
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9554: [BEAM-7911] Fix 
flakiness in avroio_test.py
URL: https://github.com/apache/beam/pull/9554#issuecomment-530623759
 
 
   > This LGTM.
   > 
   > If the condition was never true, why the test was flaky? Should not it 
always fail?
   
   It was intentionally corrupting data by replacing the last byte. It was 
failing to corrupt the data in 1 out of 256 cases, where the last byte was 
already 'B'.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311101)
Time Spent: 0.5h  (was: 20m)

> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24 
> 11:22:24 
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24 
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
>  line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync 
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute 
> 'exception'
> 11:22:24  >> begin captured logging << 
> 
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern: 
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging << 
> -
> 11:22:24 
> 11:22:24 
> --
> 11:22:24 XML: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24 
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24 
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
>  setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary 
> 
> 11:22:24 ERROR:   py36: commands failed
> 11:22:24 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8217) py3 sdk harness status server does not print thread stack

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8217?focusedWorklogId=311100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311100
 ]

ASF GitHub Bot logged work on BEAM-8217:


Author: ASF GitHub Bot
Created on: 12/Sep/19 01:11
Start Date: 12/Sep/19 01:11
Worklog Time Spent: 10m 
  Work Description: angoenka commented on pull request #9551: [BEAM-8217] 
Encode str to bytes py3 in sdk harness status server
URL: https://github.com/apache/beam/pull/9551#discussion_r323521094
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py
 ##
 @@ -74,7 +74,7 @@ def do_GET(self):  # pylint: disable=invalid-name
 self.end_headers()
 
 for line in StatusServer.get_thread_dump():
-  self.wfile.write(line)
+  self.wfile.write(str.encode(line, 'UTF-8'))
 
 Review comment:
   1. Done
   2. Done
   3. checked with 3.5
   4. TODO
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311100)
Remaining Estimate: 0h
Time Spent: 10m

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7730) Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7730?focusedWorklogId=311092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311092
 ]

ASF GitHub Bot logged work on BEAM-7730:


Author: ASF GitHub Bot
Created on: 12/Sep/19 00:14
Start Date: 12/Sep/19 00:14
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #9296: [BEAM-7730] Introduce 
Flink 1.9 Runner
URL: https://github.com/apache/beam/pull/9296#issuecomment-530613323
 
 
   @dmvk Let us know when you have addressed the comments. Happy to take 
another look then.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311092)
Time Spent: 4.5h  (was: 4h 20m)

> Add Flink 1.9 build target and Make FlinkRunner compatible with Flink 1.9
> -
>
> Key: BEAM-7730
> URL: https://issues.apache.org/jira/browse/BEAM-7730
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: sunjincheng
>Assignee: David Moravek
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> Apache Flink 1.9 will coming and it's better to add Flink 1.9 build target 
> and make Flink Runner compatible with Flink 1.9.
> I will add the brief changes after the Flink 1.9.0 released. 
> And I appreciate it if you can leave your suggestions or comments!



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8156) Finish migration to standard Python typing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8156?focusedWorklogId=311091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311091
 ]

ASF GitHub Bot logged work on BEAM-8156:


Author: ASF GitHub Bot
Created on: 12/Sep/19 00:02
Start Date: 12/Sep/19 00:02
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9509: [BEAM-8156] Add 
convert_to_typing_type
URL: https://github.com/apache/beam/pull/9509
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311091)
Remaining Estimate: 502h 20m  (was: 502.5h)
Time Spent: 1h 40m  (was: 1.5h)

> Finish migration to standard Python typing
> --
>
> Key: BEAM-8156
> URL: https://issues.apache.org/jira/browse/BEAM-8156
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Udi Meiri
>Priority: Major
>   Original Estimate: 504h
>  Time Spent: 1h 40m
>  Remaining Estimate: 502h 20m
>
> We should migrate all Python uses of types to the standard typing module, and 
> make the typehints.* ones aliases of the Python ones. 
>  
> There are three places where we use custom typehints behavior: 
> (1) is_compatible_with
> (2) bind_type_variables/match_type_variables
> (3) trivial type inference. 
>  
> I would propose that each of these be adapted to a (internal) public 
> interface that accepts and returns standard typing types, and internally 
> converts to our (nowhere else exposed) typehints types, performs the logic, 
> and converts back. Each of these in turn can then be updated, as needed and 
> orthogonally, to operate on the typing types natively (possibly via deference 
> to a third-party library). 
>  
> I think coder inference could be easily adopted to use typing types directly, 
> but it may be a fourth place where we do internal conversion first. Another 
> gotcha is special care may need to be taken if we ever need to pickle these 
> types (which IIRC may have issues). 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8218) Implement Apache PulsarIO

[Alex Van Boxel (Jira)]

Alex Van Boxel created BEAM-8218:


 Summary: Implement Apache PulsarIO
 Key: BEAM-8218
 URL: https://issues.apache.org/jira/browse/BEAM-8218
 Project: Beam
  Issue Type: Task
  Components: io-ideas
Reporter: Alex Van Boxel
Assignee: Alex Van Boxel


Apache Pulsar is starting to gain popularity. Having a native Beam PulsarIO 
could be beneficial.

[https://pulsar.apache.org/|https://pulsar.apache.org/en/]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7911) test_corrupted_file test flaky

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7911?focusedWorklogId=311090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311090
 ]

ASF GitHub Bot logged work on BEAM-7911:


Author: ASF GitHub Bot
Created on: 11/Sep/19 23:57
Start Date: 11/Sep/19 23:57
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9554: [BEAM-7911] Fix 
flakiness in avroio_test.py
URL: https://github.com/apache/beam/pull/9554#issuecomment-530609712
 
 
   R: @aaltay 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311090)
Time Spent: 20m  (was: 10m)

> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24 
> 11:22:24 
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24 
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
>  line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync 
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute 
> 'exception'
> 11:22:24  >> begin captured logging << 
> 
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern: 
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging << 
> -
> 11:22:24 
> 11:22:24 
> --
> 11:22:24 XML: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24 
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24 
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
>  setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary 
> 
> 11:22:24 ERROR:   py36: commands failed
> 11:22:24 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7911) test_corrupted_file test flaky

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7911?focusedWorklogId=311089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311089
 ]

ASF GitHub Bot logged work on BEAM-7911:


Author: ASF GitHub Bot
Created on: 11/Sep/19 23:55
Start Date: 11/Sep/19 23:55
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9554: [BEAM-7911] Fix 
flakiness in avroio_test.py
URL: https://github.com/apache/beam/pull/9554
 
 
   - Comparison `data[last_char_index] == b'B'` was never true in Py3 (LHS
 is int, RHS is bytes).
   - Fix assertion on exception text.
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 

[jira] [Updated] (BEAM-8217) py3 sdk harness status server does not print thread stack

[Ankur Goenka (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-8217:
---
Priority: Blocker  (was: Major)

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Blocker
> Fix For: 2.16.0
>
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8217) py3 sdk harness status server does not print thread stack

[Ankur Goenka (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928098#comment-16928098
 ] 

Ankur Goenka commented on BEAM-8217:


cc: [~tvalentyn] [~markflyhigh]

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.16.0
>
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8217) py3 sdk harness status server does not print thread stack

[Ankur Goenka (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-8217:
---
Fix Version/s: 2.16.0

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.16.0
>
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8217) py3 sdk harness status server does not print thread stack

[Ankur Goenka (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928097#comment-16928097
 ] 

Ankur Goenka commented on BEAM-8217:


Fix PR https://github.com/apache/beam/pull/9551

> py3 sdk harness status server does not print thread stack
> -
>
> Key: BEAM-8217
> URL: https://issues.apache.org/jira/browse/BEAM-8217
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Priority: Major
> Fix For: 2.16.0
>
>
> The calls are failing with 
> 
> Exception happened during processing of request from ('127.0.0.1', 56106)
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/socketserver.py", line 320, in 
> _handle_request_noblock
> self.process_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
> self.finish_request(request, client_address)
>   File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
> self.handle()
>   File "/usr/lib/python3.6/http/server.py", line 418, in handle
> self.handle_one_request()
>   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
> method()
>   File "a.py", line 70, in do_GET
> self.wfile.write(line)
>   File "/usr/lib/python3.6/socketserver.py", line 803, in write
> self._sock.sendall(b)
> TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8217) py3 sdk harness status server does not print thread stack

[Ankur Goenka (Jira)]

Ankur Goenka created BEAM-8217:
--

 Summary: py3 sdk harness status server does not print thread stack
 Key: BEAM-8217
 URL: https://issues.apache.org/jira/browse/BEAM-8217
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-harness
Reporter: Ankur Goenka


The calls are failing with 


Exception happened during processing of request from ('127.0.0.1', 56106)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 320, in 
_handle_request_noblock
self.process_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 351, in process_request
self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 364, in finish_request
self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
self.handle()
  File "/usr/lib/python3.6/http/server.py", line 418, in handle
self.handle_one_request()
  File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
method()
  File "a.py", line 70, in do_GET
self.wfile.write(line)
  File "/usr/lib/python3.6/socketserver.py", line 803, in write
self._sock.sendall(b)
TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8196?focusedWorklogId=311084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311084
 ]

ASF GitHub Bot logged work on BEAM-8196:


Author: ASF GitHub Bot
Created on: 11/Sep/19 23:39
Start Date: 11/Sep/19 23:39
Worklog Time Spent: 10m 
  Work Description: udim commented on pull request #9547: [BEAM-8196] Add 
wait_until_finish_duration
URL: https://github.com/apache/beam/pull/9547
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311084)
Time Spent: 40m  (was: 0.5h)

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5820) Vendor Calcite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311082
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 11/Sep/19 23:31
Start Date: 11/Sep/19 23:31
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-530604564
 
 
   I only reviewed the shading changes and not its usage. I would be fine with 
a rebase and update as all the vendoring of Calcite was done in a different PR.
   
   Please also update the PR title/description and link to JIRA.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311082)
Time Spent: 8h  (was: 7h 50m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 8h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8165) Change image name and add images to release process

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8165:
---
Description: 
change name to apachebeam/\{lang} \{ver}_sdk

Images are published to [https://hub.docker.com/u/apachebeam].

  was:
change name to apachebeam/ \{lang} \{ver}_sdk

Images are published to [https://hub.docker.com/u/apachebeam].


> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang} \{ver}_sdk
> Images are published to [https://hub.docker.com/u/apachebeam].



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8165) Change image name and add images to release process

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8165:
---
Description: 
change name to apachebeam/ \{lang} \{ver}_sdk

Images are published to [https://hub.docker.com/u/apachebeam].

  was:
change name to apachebeam/\{lang}{ver}_sdk

Images are published to [https://hub.docker.com/u/apachebeam].


> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> change name to apachebeam/ \{lang} \{ver}_sdk
> Images are published to [https://hub.docker.com/u/apachebeam].



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8165) Change image name and add images to release process

[Hannah Jiang (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hannah Jiang updated BEAM-8165:
---
Description: 
change name to apachebeam/\{lang}{ver}_sdk

Images are published to [https://hub.docker.com/u/apachebeam].

  was:change name to apachebeam/\{lang}{ver}_sdk


> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk
> Images are published to [https://hub.docker.com/u/apachebeam].



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928090#comment-16928090
 ] 

Valentyn Tymofieiev commented on BEAM-5878:
---

Per conversation offline with Mark, removing the 2.16.0 label to follow Beam 
release procedures, we can add the label back once the issue is fixed.

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5878:
--
Fix Version/s: (was: 2.16.0)

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7305) Add first version of Hazelcast Jet Runner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7305?focusedWorklogId=311081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311081
 ]

ASF GitHub Bot logged work on BEAM-7305:


Author: ASF GitHub Bot
Created on: 11/Sep/19 23:10
Start Date: 11/Sep/19 23:10
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #9471: [BEAM-7305] 
Improve Jet Runner related documentation
URL: https://github.com/apache/beam/pull/9471
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311081)
Time Spent: 11h 40m  (was: 11.5h)

> Add first version of Hazelcast Jet Runner
> -
>
> Key: BEAM-7305
> URL: https://issues.apache.org/jira/browse/BEAM-7305
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-jet
>Reporter: Maximilian Michels
>Assignee: Jozsef Bartok
>Priority: Major
> Fix For: 2.14.0
>
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8216) GCS IO fails with uninformative 'Broken pipe' errors while attempting to write to a GCS bucket without proper permissions.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928073#comment-16928073
 ] 

Valentyn Tymofieiev commented on BEAM-8216:
---

Over to [~chamikara] to triage.

> GCS IO fails with uninformative 'Broken pipe' errors while attempting to 
> write to a GCS bucket without proper permissions.
> --
>
> Key: BEAM-8216
> URL: https://issues.apache.org/jira/browse/BEAM-8216
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Major
>
> Obvserved while executing a wordcount IT pipeline:
> {noformat}
>  ./gradlew :sdks:python:test-suites:dataflow:py36:integrationTest \
> -Dtests=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
> -Dattr=IT 
> -DpipelineOptions="--project=some_project_different_from_apache_beam_testing \
> --staging_location=gs://some_bucket/ \
> --temp_location=gs://some_bucket/ \
> --input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.* 
> \
> --output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output  \
> --expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710 \
> --num_workers=10 \
> --autoscaling_algorithm=NONE \
> --runner=TestDataflowRunner \
> --sdk_location=/full/path/to/beam/sdks/python/dist/apache-beam-2.16.0.dev0.tar.gz"
>  \
> --info  
> {noformat}
> gs://temp-storage-for-end-to-end-tests/py-it-cloud/output lives in a 
> different project than was running the pipeline.
> This caused a bunch of Broken pipe errors. Console logs:
> {noformat}
> root: INFO: 2019-09-11T19:06:23.055Z: JOB_MESSAGE_BASIC: Finished operation 
> read/Read+split+pair_with_one+group/Reify+group/Write
> root: INFO: 2019-09-11T19:06:23.157Z: JOB_MESSAGE_BASIC: Executing operation 
> group/Close
> root: INFO: 2019-09-11T19:06:23.208Z: JOB_MESSAGE_BASIC: Finished operation 
> group/Close
> root: INFO: 2019-09-11T19:06:23.263Z: JOB_MESSAGE_BASIC: Executing operation 
> group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
> root: INFO: 2019-09-11T19:06:25.571Z: JOB_MESSAGE_ERROR: Traceback (most 
> recent call last):
>   File "apache_beam/runners/common.py", line 782, in 
> apache_beam.runners.common.DoFnRunner.process
>   File "apache_beam/runners/common.py", line 594, in 
> apache_beam.runners.common.PerWindowInvoker.invoke_process
>   File "apache_beam/runners/common.py", line 666, in 
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
>   File "/usr/local/lib/python3.6/site-packages/apache_beam/io/iobase.py", 
> line 1042, in process
> self.writer.write(element)
>   File 
> "/usr/local/lib/python3.6/site-packages/apache_beam/io/filebasedsink.py", 
> line 393, in write
> self.sink.write_record(self.temp_handle, value)
>   File 
> "/usr/local/lib/python3.6/site-packages/apache_beam/io/filebasedsink.py", 
> line 137, in write_record
> self.write_encoded_record(file_handle, self.coder.encode(value))
>   File "/usr/local/lib/python3.6/site-packages/apache_beam/io/textio.py", 
> line 407, in write_encoded_record
> file_handle.write(encoded_value)
>   File 
> "/usr/local/lib/python3.6/site-packages/apache_beam/io/filesystemio.py", line 
> 202, in write
> self._uploader.put(b)
>   File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/gcsio.py", 
> line 594, in put
> self._conn.send_bytes(data.tobytes())
>   File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 200, in 
> send_bytes
> self._send_bytes(m[offset:offset + size])
>   File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 397, in 
> _send_bytes
> self._send(header)
>   File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 368, in 
> _send
> n = write(self._handle, buf)
> BrokenPipeError: [Errno 32] Broken pipe
> During handling of the above exception, another exception occurred:
> Traceback (most recent call last):
>   File 
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line 
> 649, in do_work
> ...
> root: INFO: 2019-09-11T19:06:33.027Z: JOB_MESSAGE_DEBUG: Executing failure 
> step failure25
> root: INFO: 2019-09-11T19:06:33.066Z: JOB_MESSAGE_ERROR: Workflow failed. 
> Causes: 
> S08:group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
>  failed., The job failed because a work item has failed 4 

[jira] [Created] (BEAM-8216) GCS IO fails with uninformative 'Broken pipe' errors while attempting to write to a GCS bucket without proper permissions.

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-8216:
-

 Summary: GCS IO fails with uninformative 'Broken pipe' errors 
while attempting to write to a GCS bucket without proper permissions.
 Key: BEAM-8216
 URL: https://issues.apache.org/jira/browse/BEAM-8216
 Project: Beam
  Issue Type: Bug
  Components: io-py-gcp
Reporter: Valentyn Tymofieiev
Assignee: Chamikara Jayalath


Obvserved while executing a wordcount IT pipeline:
{noformat}
 ./gradlew :sdks:python:test-suites:dataflow:py36:integrationTest \
-Dtests=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
-Dattr=IT 
-DpipelineOptions="--project=some_project_different_from_apache_beam_testing \
--staging_location=gs://some_bucket/ \
--temp_location=gs://some_bucket/ \
--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.* \
--output=gs://temp-storage-for-end-to-end-tests/py-it-cloud/output  \
--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710 \
--num_workers=10 \
--autoscaling_algorithm=NONE \
--runner=TestDataflowRunner \
--sdk_location=/full/path/to/beam/sdks/python/dist/apache-beam-2.16.0.dev0.tar.gz"
 \
--info  
{noformat}
gs://temp-storage-for-end-to-end-tests/py-it-cloud/output lives in a different 
project than was running the pipeline.

This caused a bunch of Broken pipe errors. Console logs:
{noformat}
root: INFO: 2019-09-11T19:06:23.055Z: JOB_MESSAGE_BASIC: Finished operation 
read/Read+split+pair_with_one+group/Reify+group/Write
root: INFO: 2019-09-11T19:06:23.157Z: JOB_MESSAGE_BASIC: Executing operation 
group/Close
root: INFO: 2019-09-11T19:06:23.208Z: JOB_MESSAGE_BASIC: Finished operation 
group/Close
root: INFO: 2019-09-11T19:06:23.263Z: JOB_MESSAGE_BASIC: Executing operation 
group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
root: INFO: 2019-09-11T19:06:25.571Z: JOB_MESSAGE_ERROR: Traceback (most recent 
call last):
  File "apache_beam/runners/common.py", line 782, in 
apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 594, in 
apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 666, in 
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "/usr/local/lib/python3.6/site-packages/apache_beam/io/iobase.py", line 
1042, in process
self.writer.write(element)
  File 
"/usr/local/lib/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 
393, in write
self.sink.write_record(self.temp_handle, value)
  File 
"/usr/local/lib/python3.6/site-packages/apache_beam/io/filebasedsink.py", line 
137, in write_record
self.write_encoded_record(file_handle, self.coder.encode(value))
  File "/usr/local/lib/python3.6/site-packages/apache_beam/io/textio.py", line 
407, in write_encoded_record
file_handle.write(encoded_value)
  File "/usr/local/lib/python3.6/site-packages/apache_beam/io/filesystemio.py", 
line 202, in write
self._uploader.put(b)
  File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/gcsio.py", 
line 594, in put
self._conn.send_bytes(data.tobytes())
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 200, in 
send_bytes
self._send_bytes(m[offset:offset + size])
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 397, in 
_send_bytes
self._send(header)
  File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 368, in 
_send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", 
line 649, in do_work
...

root: INFO: 2019-09-11T19:06:33.027Z: JOB_MESSAGE_DEBUG: Executing failure step 
failure25
root: INFO: 2019-09-11T19:06:33.066Z: JOB_MESSAGE_ERROR: Workflow failed. 
Causes: 
S08:group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write
 failed., The job failed because a work item has failed 4 times. Look in 
previous log entries for the cause of each one of the 4 failures. For more 
information, see https://cloud.google.com/dataflow/docs/guides/common-errors. 
The work item was attempted on these workers:
  beamapp-valentyn-09111855-0955-pj3z-harness-5g6h
  Root cause: Work item failed.,
  beamapp-valentyn-09111855-0955-pj3z-harness-6ccc
  Root cause: Work item failed.,
  beamapp-valentyn-09111855-0955-pj3z-harness-45pp
  Root cause: Work item failed.,
  

[jira] [Commented] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928063#comment-16928063
 ] 

Valentyn Tymofieiev commented on BEAM-8198:
---

It's a bit hard working with PKB, due to issues like: 
https://issues.apache.org/jira/browse/BEAM-8215. It may be easier to run the 
benchmark directly and measure walltime:

{noformat}
time ./gradlew :sdks:python:test-suites:dataflow:py36:integrationTest \
-Dtests=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it \
-Dattr=IT -DpipelineOptions="--project=some_project \
--staging_location=gs://some_bucket/ \
--temp_location=gs://some_bucket/ \
--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.* \
--output=gs://some_bucket/output  \
--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710 \
--num_workers=10 \
--autoscaling_algorithm=NONE \
--runner=TestDataflowRunner \
--sdk_location=/full/path/to/apache-beam-2.16.0.dev0.tar.gz" \
--info  
{noformat}

> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8157) [Flink Runner 1.9] State requests return wrong state in timers when encoded key is length-prefixed

[Mark Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-8157:
---
Fix Version/s: (was: 2.16.0)
   2.17.0

> [Flink Runner 1.9] State requests return wrong state in timers when encoded 
> key is length-prefixed
> --
>
> Key: BEAM-8157
> URL: https://issues.apache.org/jira/browse/BEAM-8157
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.13.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.17.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The Flink runner requires the internal key to be encoded without a length 
> prefix (OUTER context). The user state request handler exposes a serialized 
> version of the key to the Runner. This key is encoded with the NESTED context 
> which may add a length prefix. We need to convert it to OUTER context to 
> match the Flink runner's key encoding.
> So far this has not caused the Flink Runner to behave incorrectly. However, 
> with the upcoming support for Flink 1.9, the state backend will not accept 
> requests for keys not part of any key group/partition of the operator. This 
> is very likely to happen with the encoding not being consistent.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8157) [Flink Runner 1.9] State requests return wrong state in timers when encoded key is length-prefixed

[Mark Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928062#comment-16928062
 ] 

Mark Liu commented on BEAM-8157:


[~mxm] 2.16 release branch is cut. I'll move it to 2.17. If you really want to 
have this in 2.16, please let me know and we can cherrypick your change if it 
fits the release time frame.

> [Flink Runner 1.9] State requests return wrong state in timers when encoded 
> key is length-prefixed
> --
>
> Key: BEAM-8157
> URL: https://issues.apache.org/jira/browse/BEAM-8157
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.13.0
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The Flink runner requires the internal key to be encoded without a length 
> prefix (OUTER context). The user state request handler exposes a serialized 
> version of the key to the Runner. This key is encoded with the NESTED context 
> which may add a length prefix. We need to convert it to OUTER context to 
> match the Flink runner's key encoding.
> So far this has not caused the Flink Runner to behave incorrectly. However, 
> with the upcoming support for Flink 1.9, the state backend will not accept 
> requests for keys not part of any key group/partition of the operator. This 
> is very likely to happen with the encoding not being consistent.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7945) Allow runner to configure "semi_persist_dir" which is used in the SDK harness

[Mark Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928057#comment-16928057
 ] 

Mark Liu commented on BEAM-7945:


2.16 release branch is cut. I'll move it to 2.17. If you really want to have 
this in 2.16, please let me know and we can cherrypick your change if it fits 
the release time frame.

> Allow runner to configure "semi_persist_dir" which is used in the SDK harness
> -
>
> Key: BEAM-7945
> URL: https://issues.apache.org/jira/browse/BEAM-7945
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution, sdk-go, sdk-java-core, sdk-py-core
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Currently "semi_persist_dir" is not configurable. This may become a problem 
> in certain scenarios. For example, the default value of "semi_persist_dir" is 
> "/tmp" 
> ([https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L48])
>  in Python SDK harness. When the environment type is "PROCESS", the disk of 
> "/tmp" may be filled up and unexpected issues will occur in production 
> environment. We should provide a way to configure "semi_persist_dir" in 
> EnvironmentFactory at the runner side. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-7945) Allow runner to configure "semi_persist_dir" which is used in the SDK harness

[Mark Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-7945:
---
Fix Version/s: (was: 2.16.0)
   2.17.0

> Allow runner to configure "semi_persist_dir" which is used in the SDK harness
> -
>
> Key: BEAM-7945
> URL: https://issues.apache.org/jira/browse/BEAM-7945
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution, sdk-go, sdk-java-core, sdk-py-core
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.17.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Currently "semi_persist_dir" is not configurable. This may become a problem 
> in certain scenarios. For example, the default value of "semi_persist_dir" is 
> "/tmp" 
> ([https://github.com/apache/beam/blob/master/sdks/python/container/boot.go#L48])
>  in Python SDK harness. When the environment type is "PROCESS", the disk of 
> "/tmp" may be filled up and unexpected issues will occur in production 
> environment. We should provide a way to configure "semi_persist_dir" in 
> EnvironmentFactory at the runner side. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder

[Mark Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928056#comment-16928056
 ] 

Mark Liu commented on BEAM-7951:


2.16 release branch is cut. I'll move this jira to 2.17. If you really want to 
have this in 2.16, please let me know and we can cherrypick your change if it 
fits the release time frame.

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.16.0
>
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8215) Wordcount 1GB Python PKB benchmarks sometimes fail without uninformative error

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-8215:
--
Summary: Wordcount 1GB Python PKB benchmarks sometimes fail without 
uninformative error  (was: Wordcount 1GB Python PKB benchmarks sometimes fail 
without an informative error)

> Wordcount 1GB Python PKB benchmarks sometimes fail without uninformative error
> --
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Mark Liu
>Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread 
> beam_integration_benchmark(1/1) ERRORError during benchmark 
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with 
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another 
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-8215) Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-8215:
--
Summary: Wordcount 1GB Python PKB benchmarks sometimes fail with 
uninformative error  (was: Wordcount 1GB Python PKB benchmarks sometimes fail 
without uninformative error)

> Wordcount 1GB Python PKB benchmarks sometimes fail with uninformative error
> ---
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Mark Liu
>Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread 
> beam_integration_benchmark(1/1) ERRORError during benchmark 
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with 
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another 
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-7951) Allow runner to configure customization WindowedValue coder such as ValueOnlyWindowedValueCoder

[Mark Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-7951:
---
Fix Version/s: (was: 2.16.0)
   2.17.0

> Allow runner to configure customization WindowedValue coder such as 
> ValueOnlyWindowedValueCoder
> ---
>
> Key: BEAM-7951
> URL: https://issues.apache.org/jira/browse/BEAM-7951
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.17.0
>
>
> The coder of WindowedValue cannot be configured and it’s always 
> FullWindowedValueCoder. We don't need to serialize the timestamp, window and 
> pane properties in Flink and so it will be better to make the coder 
> configurable (i.e. allowing to use ValueOnlyWindowedValueCoder)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=311072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311072
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 22:30
Start Date: 11/Sep/19 22:30
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9510: [BEAM-8105] 
update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#issuecomment-530591243
 
 
   @soyrice I addressed your comments, PTAL.
   @markflyhigh , your comments were addressed at the same commit. PTAL.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311072)
Time Spent: 6h 40m  (was: 6.5h)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928053#comment-16928053
 ] 

Valentyn Tymofieiev commented on BEAM-5878:
---

Per comment above, we expect 2.16.0 to be the fix version, once this issue is 
resolved.  

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
> Fix For: 2.16.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Valentyn Tymofieiev (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5878:
--
Fix Version/s: (was: 2.17.0)
   2.16.0

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
> Fix For: 2.16.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Mark Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928050#comment-16928050
 ] 

Mark Liu commented on BEAM-5878:


release-2.16.0 branch is cut. Move it to 2.17

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
> Fix For: 2.16.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-5878) Support DoFns with Keyword-only arguments in Python 3.

[Mark Liu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-5878:
---
Fix Version/s: (was: 2.16.0)
   2.17.0

> Support DoFns with Keyword-only arguments in Python 3.
> --
>
> Key: BEAM-5878
> URL: https://issues.apache.org/jira/browse/BEAM-5878
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: yoshiki obata
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>
> Python 3.0 [adds a possibility|https://www.python.org/dev/peps/pep-3102/] to 
> define functions with keyword-only arguments. 
> Currently Beam does not handle them correctly. [~ruoyu] pointed out [one 
> place|https://github.com/apache/beam/blob/a56ce43109c97c739fa08adca45528c41e3c925c/sdks/python/apache_beam/typehints/decorators.py#L118]
>  in our codebase that we should fix: in Python in 3.0 inspect.getargspec() 
> will fail on functions with keyword-only arguments, but a new method 
> [inspect.getfullargspec()|https://docs.python.org/3/library/inspect.html#inspect.getfullargspec]
>  supports them.
> There may be implications for our (best-effort) type-hints machinery.
> We should also add a Py3-only unit tests that covers DoFn's with keyword-only 
> arguments once Beam Python 3 tests are in a good shape.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8215) Wordcount 1GB Python PKB benchmarks sometimes fail without an informative error

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928049#comment-16928049
 ] 

Valentyn Tymofieiev commented on BEAM-8215:
---

cc: [~manishab]

> Wordcount 1GB Python PKB benchmarks sometimes fail without an informative 
> error
> ---
>
> Key: BEAM-8215
> URL: https://issues.apache.org/jira/browse/BEAM-8215
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Valentyn Tymofieiev
>Assignee: Mark Liu
>Priority: Major
>
> Example:
> https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console
> {noformat}
> 12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread 
> beam_integration_benchmark(1/1) ERRORError during benchmark 
> beam_integration_benchmark
> 12:09:27 Traceback (most recent call last):
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 841, in RunBenchmark
> 12:09:27 DoRunPhase(spec, collector, detailed_timer)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
>  line 687, in DoRunPhase
> 12:09:27 samples = spec.BenchmarkRun(spec)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
>  line 160, in Run
> 12:09:27 job_type=job_type)
> 12:09:27   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
>  line 91, in SubmitJob
> 12:09:27 assert retcode == 0, "Integration Test Failed."
> 12:09:27 AssertionError: Integration Test Failed.
> {noformat}
> It seems like Job submission failed, but there are no details. I talked with 
> [~markflyhigh], and sounds like we plan to stop using PKB in favor of another 
> framework.
> Assigning to Mark for now to triage follow up or reassign as appropriate.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-7274) Protobuf Beam Schema support

[Alex Van Boxel (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Van Boxel updated BEAM-7274:
-
Fix Version/s: (was: 2.16.0)
   2.17.0

Moved to 2.17.0

> Protobuf Beam Schema support
> 
>
> Key: BEAM-7274
> URL: https://issues.apache.org/jira/browse/BEAM-7274
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>
> Add support for the new Beam Schema to the Protobuf extension.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7611) Python BigTableIO IT is not running in any test suites

[Chamikara Jayalath (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928044#comment-16928044
 ] 

Chamikara Jayalath commented on BEAM-7611:
--

I believe [~udim] is trying to run a pipeline with BigTable sink using new 
dependencies in [https://github.com/apache/beam/pull/9491].

 

If that works, we can remove this Jira out of Beam 2.16.0 blocking status but 
fixing test coverage for BigTable sink is still critical IMO.

> Python BigTableIO IT is not running in any test suites
> --
>
> Key: BEAM-7611
> URL: https://issues.apache.org/jira/browse/BEAM-7611
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-gcp, testing
>Reporter: Chamikara Jayalath
>Assignee: Solomon Duskis
>Priority: Blocker
> Fix For: 2.16.0
>
>
> We added an integration test here: [https://github.com/apache/beam/pull/7367]
>  
> But this currently does not get picked up by any test suites (and get skipped 
> by some due to missing dependencies) hence BigTable sink is largely untested.
>  
> First attempt to enable it: [https://github.com/apache/beam/pull/8886]
>  
> Solomon assigning to you since I cannot find Juan's (PR author) Jira ID.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Updated] (BEAM-5967) ProtoCoder doesn't support DynamicMessage

[Alex Van Boxel (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Van Boxel updated BEAM-5967:
-
Fix Version/s: (was: 2.16.0)
   2.17.0

Moved to 2.17.0

> ProtoCoder doesn't support DynamicMessage
> -
>
> Key: BEAM-5967
> URL: https://issues.apache.org/jira/browse/BEAM-5967
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 2.8.0
>Reporter: Alex Van Boxel
>Assignee: Alex Van Boxel
>Priority: Major
> Fix For: 2.17.0
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> The ProtoCoder does make some assumptions about static messages being 
> available. The DynamicMessage doesn't have some of them, mainly because the 
> proto schema is defined at runtime and not at compile time.
> Does it make sense to make a special coder for DynamicMessage or build it 
> into the normal ProtoCoder.
> Here is an example of the assumtion being made in the current Codec:
> {code:java}
> try {
>   @SuppressWarnings("unchecked")
>   T protoMessageInstance = (T) 
> protoMessageClass.getMethod("getDefaultInstance").invoke(null);
>   @SuppressWarnings("unchecked")
>   Parser tParser = (Parser) protoMessageInstance.getParserForType();
>   memoizedParser = tParser;
> } catch (IllegalAccessException | InvocationTargetException | 
> NoSuchMethodException e) {
>   throw new IllegalArgumentException(e);
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8215) Wordcount 1GB Python PKB benchmarks sometimes fail without an informative error

[Valentyn Tymofieiev (Jira)]

Valentyn Tymofieiev created BEAM-8215:
-

 Summary: Wordcount 1GB Python PKB benchmarks sometimes fail 
without an informative error
 Key: BEAM-8215
 URL: https://issues.apache.org/jira/browse/BEAM-8215
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Valentyn Tymofieiev
Assignee: Mark Liu


Example:
https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py36/452/console

{noformat}
12:09:27 2019-09-11 19:09:27,655 a47400ce MainThread 
beam_integration_benchmark(1/1) ERRORError during benchmark 
beam_integration_benchmark
12:09:27 Traceback (most recent call last):
12:09:27   File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
 line 841, in RunBenchmark
12:09:27 DoRunPhase(spec, collector, detailed_timer)
12:09:27   File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/pkb.py",
 line 687, in DoRunPhase
12:09:27 samples = spec.BenchmarkRun(spec)
12:09:27   File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/linux_benchmarks/beam_integration_benchmark.py",
 line 160, in Run
12:09:27 job_type=job_type)
12:09:27   File 
"/home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_WordCountIT_Py36/PerfKitBenchmarker/perfkitbenchmarker/providers/gcp/gcp_dpb_dataflow.py",
 line 91, in SubmitJob
12:09:27 assert retcode == 0, "Integration Test Failed."
12:09:27 AssertionError: Integration Test Failed.
{noformat}

It seems like Job submission failed, but there are no details. I talked with 
[~markflyhigh], and sounds like we plan to stop using PKB in favor of another 
framework.

Assigning to Mark for now to triage follow up or reassign as appropriate.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928020#comment-16928020
 ] 

Valentyn Tymofieiev commented on BEAM-8198:
---

Looking at Jenkins jobs for Wordcount 1 GB benchmark 
(https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37), we can 
do the following to reproduce these runs.

1) Clone PKB and install PKB dependencies in a virtual environment with Python 
2.7. It looks like we run perfkit benchmarker in Python 2.7 environment, but 
the benchmarks pipeline is triggered via gradle and can use other runtime.

git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
pip install -r ./PerfKitBenchmarker/requirements.txt

2) Clone Beam SDK and build SDK tarball against desired commit

3) Configure the parameters to the benchmark:


{noformat}
PROJECT=my_gcp_project
PKB_DIR=/path/to/PerfKitBenchmarker
PKB_BQ_TABLE=bq_dataset_to_save_results.wordcount_py36_beam216_pkb_results
BEAM_LOCATION=/path/to/clone/of/beam
BEAM_TARBALL=$BEAM_LOCATION/sdks/python/dist/apache-beam-2.16.0.dev0.tar.gz
TEMP_LOCATION=gs://some/temp/location/

{noformat}

4) Run the benchmark:


{noformat}
bash -c "python $PKB_DIR/pkb.py \
--project=${PROJECT} --dpb_log_level=INFO --bigquery_table=${PKB_BQ_TABLE} \
--k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=/tmp \
--beam_location=${BEAM_LOCATION} --official=true --dpb_service_zone=fake_zone 
--beam_sdk=python \
--benchmarks=beam_integration_benchmark \
--beam_it_class=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it
 \
--beam_it_module=:sdks:python:test-suites:dataflow:py36 \
--beam_prebuilt=true --beam_python_sdk_location=${BEAM_TARBALL} \
--beam_runner=TestDataflowRunner --beam_it_timeout=12000 \
'--beam_it_args=--project=${PROJECT},\
--staging_location=${TEMP_LOCATION},\
--temp_location=${TEMP_LOCATION},\
--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.*,\
--output=${TEMP_LOCATION}temp-storage-for-end-to-end-tests/py-it-cloud/output,\
--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710,\
--num_workers=10,--autoscaling_algorithm=NONE'"
{noformat}



> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version to 2.16.0 to understand the cause, hopefully before the 
> vote.
> cc: [~altay] [~thw] [~markflyhigh]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Comment Edited] (BEAM-8198) Investigate possible performance regression of Wordcount 1GB batch benchmark on Py3.

[Valentyn Tymofieiev (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928020#comment-16928020
 ] 

Valentyn Tymofieiev edited comment on BEAM-8198 at 9/11/19 9:37 PM:


Looking at Jenkins jobs for Wordcount 1 GB benchmark 
(https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37), we can 
do the following to reproduce these runs.

1) Clone PKB and install PKB dependencies in a virtual environment with Python 
2.7. It looks like we run perfkit benchmarker in Python 2.7 environment, but 
the benchmarks pipeline is triggered via gradle and can use other runtime.


{noformat}
git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
pip install -r ./PerfKitBenchmarker/requirements.txt

{noformat}

2) Clone Beam SDK and build SDK tarball against desired commit

3) Configure the parameters to the benchmark:


{noformat}
PROJECT=my_gcp_project
PKB_DIR=/path/to/PerfKitBenchmarker
PKB_BQ_TABLE=bq_dataset_to_save_results.wordcount_py36_beam216_pkb_results
BEAM_LOCATION=/path/to/clone/of/beam
BEAM_TARBALL=$BEAM_LOCATION/sdks/python/dist/apache-beam-2.16.0.dev0.tar.gz
TEMP_LOCATION=gs://some/temp/location/

{noformat}

4) Run the benchmark:


{noformat}
bash -c "python $PKB_DIR/pkb.py \
--project=${PROJECT} --dpb_log_level=INFO --bigquery_table=${PKB_BQ_TABLE} \
--k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=/tmp \
--beam_location=${BEAM_LOCATION} --official=true --dpb_service_zone=fake_zone 
--beam_sdk=python \
--benchmarks=beam_integration_benchmark \
--beam_it_class=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it
 \
--beam_it_module=:sdks:python:test-suites:dataflow:py36 \
--beam_prebuilt=true --beam_python_sdk_location=${BEAM_TARBALL} \
--beam_runner=TestDataflowRunner --beam_it_timeout=12000 \
'--beam_it_args=--project=${PROJECT},\
--staging_location=${TEMP_LOCATION},\
--temp_location=${TEMP_LOCATION},\
--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.*,\
--output=${TEMP_LOCATION}temp-storage-for-end-to-end-tests/py-it-cloud/output,\
--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710,\
--num_workers=10,--autoscaling_algorithm=NONE'"
{noformat}




was (Author: tvalentyn):
Looking at Jenkins jobs for Wordcount 1 GB benchmark 
(https://builds.apache.org/job/beam_PerformanceTests_WordCountIT_Py37), we can 
do the following to reproduce these runs.

1) Clone PKB and install PKB dependencies in a virtual environment with Python 
2.7. It looks like we run perfkit benchmarker in Python 2.7 environment, but 
the benchmarks pipeline is triggered via gradle and can use other runtime.

git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
pip install -r ./PerfKitBenchmarker/requirements.txt

2) Clone Beam SDK and build SDK tarball against desired commit

3) Configure the parameters to the benchmark:


{noformat}
PROJECT=my_gcp_project
PKB_DIR=/path/to/PerfKitBenchmarker
PKB_BQ_TABLE=bq_dataset_to_save_results.wordcount_py36_beam216_pkb_results
BEAM_LOCATION=/path/to/clone/of/beam
BEAM_TARBALL=$BEAM_LOCATION/sdks/python/dist/apache-beam-2.16.0.dev0.tar.gz
TEMP_LOCATION=gs://some/temp/location/

{noformat}

4) Run the benchmark:


{noformat}
bash -c "python $PKB_DIR/pkb.py \
--project=${PROJECT} --dpb_log_level=INFO --bigquery_table=${PKB_BQ_TABLE} \
--k8s_get_retry_count=36 --k8s_get_wait_interval=10 --temp_dir=/tmp \
--beam_location=${BEAM_LOCATION} --official=true --dpb_service_zone=fake_zone 
--beam_sdk=python \
--benchmarks=beam_integration_benchmark \
--beam_it_class=apache_beam.examples.wordcount_it_test:WordCountIT.test_wordcount_it
 \
--beam_it_module=:sdks:python:test-suites:dataflow:py36 \
--beam_prebuilt=true --beam_python_sdk_location=${BEAM_TARBALL} \
--beam_runner=TestDataflowRunner --beam_it_timeout=12000 \
'--beam_it_args=--project=${PROJECT},\
--staging_location=${TEMP_LOCATION},\
--temp_location=${TEMP_LOCATION},\
--input=gs://apache-beam-samples/input_small_files/ascii_sort_1MB_input.*,\
--output=${TEMP_LOCATION}temp-storage-for-end-to-end-tests/py-it-cloud/output,\
--expect_checksum=ea0ca2e5ee4ea5f218790f28d0b9fe7d09d8d710,\
--num_workers=10,--autoscaling_algorithm=NONE'"
{noformat}



> Investigate possible performance regression of Wordcount 1GB batch benchmark 
> on Py3.
> 
>
> Key: BEAM-8198
> URL: https://issues.apache.org/jira/browse/BEAM-8198
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.16.0
>
>
> context: 
> https://lists.apache.org/thread.html/51e000f16481451c207c00ac5e881aa4a46fa020922eddffd00ad527@%3Cdev.beam.apache.org%3E
> Setting fix version 

[jira] [Created] (BEAM-8214) Change portability examples to DOCKER mode

[Kyle Weaver (Jira)]

Kyle Weaver created BEAM-8214:
-

 Summary: Change portability examples to DOCKER mode
 Key: BEAM-8214
 URL: https://issues.apache.org/jira/browse/BEAM-8214
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Kyle Weaver
Assignee: Kyle Weaver


Right now [https://beam.apache.org/roadmap/portability/] uses loopback mode, 
which should be replaced by docker. Also fix the SDK harness gradle invocation 
in Spark.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=311024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311024
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 21:24
Start Date: 11/Sep/19 21:24
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9549: 
[BEAM-8165] add _sdk to container name
URL: https://github.com/apache/beam/pull/9549
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311024)
Time Spent: 5h 10m  (was: 5h)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8208) [website] Python container build invocation fails

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8208?focusedWorklogId=311022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311022
 ]

ASF GitHub Bot logged work on BEAM-8208:


Author: ASF GitHub Bot
Created on: 11/Sep/19 21:22
Start Date: 11/Sep/19 21:22
Worklog Time Spent: 10m 
  Work Description: ibzib commented on pull request #9545: [BEAM-8208] 
[website] update Python container build invocation
URL: https://github.com/apache/beam/pull/9545
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311022)
Time Spent: 0.5h  (was: 20m)

> [website] Python container build invocation fails
> -
>
> Key: BEAM-8208
> URL: https://issues.apache.org/jira/browse/BEAM-8208
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The command `./gradlew :sdks:python:container:docker` mentioned on the 
> portability, Flink, and Spark pages causes an error (and an obscure one at 
> that): 
> Execution failed for task ':sdks:python:container:docker'.
> > name is a required docker configuration item.
> We need to include the python version in the container invocation now.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5820) Vendor Calcite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=311006=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-311006
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 11/Sep/19 21:04
Start Date: 11/Sep/19 21:04
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-530565186
 
 
   Sorry @vectorijk, as your PR is big(which is a great refactor to move to 
vendor calcite), it's easy to have conflicts by code changes on java imports. 
   
   You could wait until PR reviewers have a consensus, then do a final rebase 
and pass test to make PR be merged.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 311006)
Time Spent: 7h 50m  (was: 7h 40m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-5820) Vendor Calcite

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=310996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310996
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:52
Start Date: 11/Sep/19 20:52
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-530560947
 
 
   @apilloud @lukecwik @iemejia @kennknowles 
   
   As this PR has passed tests before(although it has conflicts now but that's 
can be solved quickly), if you don't have other comments, I will merge this PR.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310996)
Time Spent: 7h 40m  (was: 7.5h)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-7911) test_corrupted_file test flaky

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-7911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927983#comment-16927983
 ] 

Udi Meiri commented on BEAM-7911:
-

I figured it out. It's a python 3 issue. type(data[last_char_index]) is str in 
py2 vs int in py3, so it'll never match b'B' in py3.

> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24 
> 11:22:24 
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24 
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
>  line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync 
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute 
> 'exception'
> 11:22:24  >> begin captured logging << 
> 
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern: 
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging << 
> -
> 11:22:24 
> 11:22:24 
> --
> 11:22:24 XML: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24 
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24 
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
>  setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary 
> 
> 11:22:24 ERROR:   py36: commands failed
> 11:22:24 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Assigned] (BEAM-7911) test_corrupted_file test flaky

[Udi Meiri (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-7911:
---

Assignee: Udi Meiri  (was: Heejong Lee)

> test_corrupted_file test flaky
> --
>
> Key: BEAM-7911
> URL: https://issues.apache.org/jira/browse/BEAM-7911
> Project: Beam
>  Issue Type: Bug
>  Components: io-py-avro, test-failures
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Major
>
> Looks like a flake:
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7924/consoleFull
> 11:22:24 
> 11:22:24 
> ==
> 11:22:24 ERROR: test_corrupted_file (apache_beam.io.avroio_test.TestFastAvro)
> 11:22:24 
> --
> 11:22:24 Traceback (most recent call last):
> 11:22:24   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/avroio_test.py",
>  line 380, in test_corrupted_file
> 11:22:24 self.assertEqual(0, exn.exception.message.find('Unexpected sync 
> marker'))
> 11:22:24 AttributeError: '_AssertRaisesContext' object has no attribute 
> 'exception'
> 11:22:24  >> begin captured logging << 
> 
> 11:22:24 apache_beam.io.filesystem: DEBUG: translate_pattern: 
> '/tmp/tmpckgw8mk6' -> '\\/tmp\\/tmpckgw8mk6'
> 11:22:24 - >> end captured logging << 
> -
> 11:22:24 
> 11:22:24 
> --
> 11:22:24 XML: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/nosetests.xml
> 11:22:24 
> --
> 11:22:24 Ran 2455 tests in 1297.267s
> 11:22:24 
> 11:22:24 FAILED (SKIP=546, errors=1)
> 11:22:24 ERROR: InvocationError for command 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36/py36/bin/python
>  setup.py nosetests (exited with code 1)
> 11:22:24 py36 run-test-post: commands[0] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh
> 11:22:24 ___ summary 
> 
> 11:22:24 ERROR:   py36: commands failed
> 11:22:24 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8213) Run and report python tox tasks separately within Jenkins

[Chad Dombrova (Jira)]

Chad Dombrova created BEAM-8213:
---

 Summary: Run and report python tox tasks separately within Jenkins
 Key: BEAM-8213
 URL: https://issues.apache.org/jira/browse/BEAM-8213
 Project: Beam
  Issue Type: Improvement
  Components: build-system
Reporter: Chad Dombrova


As a python developer, the speed and comprehensibility of the jenkins PreCommit 
job could be greatly improved.

Here are some of the problems

- when a lint job fails, it's not reported in the Test results, so even though 
the job failed, I see "Test Result (no failures)"
- I have to wait for over an hour to discover the lint failed, which takes 
about a minute to run
- The logs are a jumbled mess of all the different tasks running on top of each 
other
- The test results give no indication of whether they come from python 27, 35, 
etc.  I click on Test results, then the test module, then the test class, then 
I see 4 tests named the same thing.  I assume that the first is python 2.7, the 
second is 3.5 and so on.   This makes it very difficult to discover problems, 
and deduce that they may have something to do with python version mismatches.

I believe the solution to this is to split up the single monolithic python 
PreCommit job into sub-jobs (possibly using a pipeline with steps).  This would 
give us the following benefits:

- sub-job results should become available as they finish, so lint results 
should be available very early on
- sub-job results will be reported separately, so it will be clear when an 
error is related to a particular python version
- sub-jobs without reports, like docs and lint, will have their own failure 
status and logs, so it will be more obvious when they fail what went wrong.

I'm happy to help out once I get some feedback on the desired way forward.




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310983=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310983
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:33
Start Date: 11/Sep/19 20:33
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on pull request #9446: 
[BEAM-8111] Enable CloudObjectsTest$DefaultCoders
URL: https://github.com/apache/beam/pull/9446#discussion_r323445919
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
 ##
 @@ -100,4 +99,47 @@ public boolean consistentWithEquals() {
   public String toString() {
 return "SchemaCoder: " + rowCoder.toString();
   }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null || getClass() != o.getClass()) {
+  return false;
+}
+SchemaCoder that = (SchemaCoder) o;
+return rowCoder.equals(that.rowCoder)
+&& toRowFunction.equals(that.toRowFunction)
+&& fromRowFunction.equals(that.fromRowFunction);
 
 Review comment:
   Yeah - I discussed this offline a bit with @kennknowles and he convinced me 
that it was better to have an equals function that might have some false 
negatives (if the toRowFunction and fromRowFunction don't have a good equals), 
rather than one that could have false positives (like if we rely on just 
checking the schema and typeDescriptor, and assume that the toRow/fromRow are 
the same).
   
   I managed to make the CloudObjectsTest work by adding RowIdentity with an 
equals() function here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310983)
Time Spent: 3h  (was: 2h 50m)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310982=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310982
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:33
Start Date: 11/Sep/19 20:33
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9549: [BEAM-8165] add 
_sdk to container name
URL: https://github.com/apache/beam/pull/9549#issuecomment-530553927
 
 
   Run Python Spark ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310982)
Time Spent: 5h  (was: 4h 50m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310978
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:29
Start Date: 11/Sep/19 20:29
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310978)
Time Spent: 2h  (was: 1h 50m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310977
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:29
Start Date: 11/Sep/19 20:29
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530552521
 
 
   Got enough lgtm. I'll merge this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310977)
Time Spent: 1h 50m  (was: 1h 40m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8184) Allow asynchronous execution

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8184?focusedWorklogId=310975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310975
 ]

ASF GitHub Bot logged work on BEAM-8184:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:22
Start Date: 11/Sep/19 20:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #9530: [BEAM-8184] Add 
--async flag to dataflow runner
URL: https://github.com/apache/beam/pull/9530#issuecomment-530549669
 
 
   Note that in the Beam Python and Beam Java SDKs, the default behavior is to 
not block and instead return a PipelineResult object that can be queried for 
job status, metrics and also contains a method that blocks till the pipeline is 
done.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310975)
Time Spent: 1h 40m  (was: 1.5h)

> Allow asynchronous execution
> 
>
> Key: BEAM-8184
> URL: https://issues.apache.org/jira/browse/BEAM-8184
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: 2.15.0
>Reporter: Jack Whelpton
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> When developing streaming pipelines, it would be useful to have a means of 
> deploying a pipeline and exiting, without blocking on completion.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8105) Add container publishing instruction to release manual

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8105?focusedWorklogId=310973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310973
 ]

ASF GitHub Bot logged work on BEAM-8105:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:21
Start Date: 11/Sep/19 20:21
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9510: 
[BEAM-8105] update release guide with docker images
URL: https://github.com/apache/beam/pull/9510#discussion_r323440065
 
 

 ##
 File path: website/src/contribute/release-guide.md
 ##
 @@ -821,7 +892,8 @@ Start the review-and-vote thread on the dev@ mailing list. 
Here’s an email tem
 [7] https://github.com/apache/beam-site/pull/...
 [8] https://github.com/apache/beam/pull/...
 [9] 
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=...
-
+[10] https://hub.docker.com/u/apachebeam
 
 Review comment:
   The line you change is part of an email template (start from line 861). We 
put links at bottom of the email since they're referenced in above content. Do 
you want to reference this docker link in email? Otherwise, we can remove it.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310973)
Time Spent: 6.5h  (was: 6h 20m)

> Add container publishing instruction to release manual
> --
>
> Key: BEAM-8105
> URL: https://issues.apache.org/jira/browse/BEAM-8105
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310970
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:13
Start Date: 11/Sep/19 20:13
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530546521
 
 
   Technically I think Flink should not be relying on comparing coders. Beam
   has never required Coder authors to implement a Java equals method, and
   other runners don't do such Coder comparisons on flatten. However for now
   we can disable the test on Flink.
   
   On Wed, Sep 11, 2019 at 1:07 PM Brian Hulette 
   wrote:
   
   > +1 LGTM. I think @reuvenlax  is right that
   > #9446 's addition of
   > SchemaCoder.equals may fix this, but it's not fully baked yet. I think we
   > should just skip until we can get the equality check right and un-skip
   > these.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310970)
Time Spent: 1h 40m  (was: 1.5h)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8204) Newly added Java ValidatesRunner tests failed on ApexRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8204?focusedWorklogId=310965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310965
 ]

ASF GitHub Bot logged work on BEAM-8204:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:07
Start Date: 11/Sep/19 20:07
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9541: 
[BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
URL: https://github.com/apache/beam/pull/9541#issuecomment-530544356
 
 
   +1 LGTM. I think @reuvenlax is right that #9446's addition of 
SchemaCoder.equals may fix this, but it's not fully baked yet. I think we 
should just skip until we can get the equality check right and un-skip these.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310965)
Time Spent: 1.5h  (was: 1h 20m)

> Newly added Java ValidatesRunner tests failed on ApexRunner
> ---
>
> Key: BEAM-8204
> URL: https://issues.apache.org/jira/browse/BEAM-8204
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, test-failures
>Reporter: Yueyang Qiu
>Assignee: Brian Hulette
>Priority: Major
>  Labels: currently-failing
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Jenkins link:
> [https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/testReport/]
>  
> Initial investigation:
> [https://github.com/apache/beam/pull/9454] and 
> [https://github.com/apache/beam/pull/9372] added new ValidatesRunner tests. 
> They have been tested on Dataflow runner, but are failing on Apex runner.
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (BEAM-4948) Beam Dependency Update Request: com.google.guava

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-4948.
-
Fix Version/s: 2.15.0
 Assignee: Luke Cwik
   Resolution: Fixed

> Beam Dependency Update Request: com.google.guava
> 
>
> Key: BEAM-4948
> URL: https://issues.apache.org/jira/browse/BEAM-4948
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.15.0
>
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:28:03.628639
> Please review and upgrade the com.google.guava to the latest version 
> None 
>  
> cc: 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310964
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 20:04
Start Date: 11/Sep/19 20:04
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on pull request #9446: [BEAM-8111] 
Enable CloudObjectsTest$DefaultCoders
URL: https://github.com/apache/beam/pull/9446#discussion_r323433725
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaCoder.java
 ##
 @@ -100,4 +99,47 @@ public boolean consistentWithEquals() {
   public String toString() {
 return "SchemaCoder: " + rowCoder.toString();
   }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null || getClass() != o.getClass()) {
+  return false;
+}
+SchemaCoder that = (SchemaCoder) o;
+return rowCoder.equals(that.rowCoder)
+&& toRowFunction.equals(that.toRowFunction)
+&& fromRowFunction.equals(that.fromRowFunction);
 
 Review comment:
   Doesn't this just revert to object equality comparison on the to/from 
functions?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310964)
Time Spent: 2h 50m  (was: 2h 40m)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310962
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:59
Start Date: 11/Sep/19 19:59
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #9550: Revert 
"[BEAM-8111] Add ValidatesRunner test to AvroSchemaTest"
URL: https://github.com/apache/beam/pull/9550#issuecomment-530541628
 
 
   It should be Flink. Thank you Reuven!
   
   Robin already had https://github.com/apache/beam/pull/9541 to skip this test 
and test is done. I think we can continue with 
https://github.com/apache/beam/pull/9541 and ignore this rollback.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310962)
Time Spent: 2h 40m  (was: 2.5h)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310959
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:51
Start Date: 11/Sep/19 19:51
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9549: [BEAM-8165] add 
_sdk to container name
URL: https://github.com/apache/beam/pull/9549#issuecomment-530538752
 
 
   All tests cleared!
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310959)
Time Spent: 4h 50m  (was: 4h 40m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8111) SchemaCoder broken on DataflowRunner

[Brian Hulette (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927943#comment-16927943
 ] 

Brian Hulette commented on BEAM-8111:
-

I am OOO but I'm just at ApacheCon, so I have access to chat etc

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8111) SchemaCoder broken on DataflowRunner

[Brian Hulette (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927941#comment-16927941
 ] 

Brian Hulette commented on BEAM-8111:
-

I think this Jira can still be considered resolved without #9454. The only 
reason it's tied to this ticket is because I added that test specifically to 
avoid a failure like this one in the future.

Ideally we would continue to run this test on Dataflow, because a failure there 
would be a blocker. Is it possible to just skip it on the runners where it's 
failing instead of removing entirely?

FYI, [~reuvenlax] and I suspect the failures on other runners may be caused by 
BEAM-8146. I think that issue is fine to punt until after 2.16 is cut.

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310946
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:15
Start Date: 11/Sep/19 19:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #9550: Revert "[BEAM-8111] 
Add ValidatesRunner test to AvroSchemaTest"
URL: https://github.com/apache/beam/pull/9550#issuecomment-530525184
 
 
   Also I suspect that https://github.com/apache/beam/pull/9446 will fix this.
   
   On Wed, Sep 11, 2019 at 12:05 PM Reuven Lax  wrote:
   
   > Also it appears that Flink relies on all the Coders being exactly the
   > same, which is far more restrictive than other runners. Is there another
   > way for Flink to do this - maybe comparing the TypeDescriptor instead of
   > the Coder?
   >
   > On Wed, Sep 11, 2019 at 11:54 AM Reuven Lax  wrote:
   >
   >> FYI this appears to be Flink ValidatesRunner, not Dataflow?
   >>
   >> On Wed, Sep 11, 2019 at 11:30 AM Mark Liu 
   >> wrote:
   >>
   >>> @markflyhigh  requested your review on:
   >>> #9550  Revert "[BEAM-8111]
   >>> Add ValidatesRunner test to AvroSchemaTest".
   >>>
   >>> —
   >>> You are receiving this because your review was requested.
   >>> Reply to this email directly, view it on GitHub
   >>> 
,
   >>> or mute the thread
   >>> 

   >>> .
   >>>
   >>
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310946)
Time Spent: 2.5h  (was: 2h 20m)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (BEAM-7967) Execute portable Flink application jar

[Kyle Weaver (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Weaver resolved BEAM-7967.
---
Fix Version/s: 2.16.0
   Resolution: Fixed

> Execute portable Flink application jar
> --
>
> Key: BEAM-7967
> URL: https://issues.apache.org/jira/browse/BEAM-7967
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Labels: portability-flink
> Fix For: 2.16.0
>
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> [https://docs.google.com/document/d/1kj_9JWxGWOmSGeZ5hbLVDXSTv-zBrx4kQRqOq85RYD4/edit#heading=h.oes73844vmhl]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7967) Execute portable Flink application jar

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7967?focusedWorklogId=310945=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310945
 ]

ASF GitHub Bot logged work on BEAM-7967:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:12
Start Date: 11/Sep/19 19:12
Worklog Time Spent: 10m 
  Work Description: ibzib commented on pull request #9408:  [BEAM-7967] 
Execute portable Flink application jar
URL: https://github.com/apache/beam/pull/9408
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310945)
Time Spent: 8h 50m  (was: 8h 40m)

> Execute portable Flink application jar
> --
>
> Key: BEAM-7967
> URL: https://issues.apache.org/jira/browse/BEAM-7967
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Labels: portability-flink
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> [https://docs.google.com/document/d/1kj_9JWxGWOmSGeZ5hbLVDXSTv-zBrx4kQRqOq85RYD4/edit#heading=h.oes73844vmhl]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310940
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:05
Start Date: 11/Sep/19 19:05
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #9550: Revert "[BEAM-8111] 
Add ValidatesRunner test to AvroSchemaTest"
URL: https://github.com/apache/beam/pull/9550#issuecomment-530521285
 
 
   Also it appears that Flink relies on all the Coders being exactly the same,
   which is far more restrictive than other runners. Is there another way for
   Flink to do this - maybe comparing the TypeDescriptor instead of the Coder?
   
   On Wed, Sep 11, 2019 at 11:54 AM Reuven Lax  wrote:
   
   > FYI this appears to be Flink ValidatesRunner, not Dataflow?
   >
   > On Wed, Sep 11, 2019 at 11:30 AM Mark Liu 
   > wrote:
   >
   >> @markflyhigh  requested your review on:
   >> #9550  Revert "[BEAM-8111] Add
   >> ValidatesRunner test to AvroSchemaTest".
   >>
   >> —
   >> You are receiving this because your review was requested.
   >> Reply to this email directly, view it on GitHub
   >> 
,
   >> or mute the thread
   >> 

   >> .
   >>
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310940)
Time Spent: 2h 20m  (was: 2h 10m)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8131) Provide Kubernetes setup with Prometheus

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8131?focusedWorklogId=310941=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310941
 ]

ASF GitHub Bot logged work on BEAM-8131:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:05
Start Date: 11/Sep/19 19:05
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on pull request #9482: [BEAM-8131] 
Provide Kubernetes setup for Prometheus
URL: https://github.com/apache/beam/pull/9482#discussion_r323409365
 
 

 ##
 File path: .test-infra/metrics/README.md
 ##
 @@ -84,23 +95,31 @@ docker-compose build
 
 # Spinup docker-compose related containers.
 docker-compose up
 
 Review comment:
   I believe it is reasonable to have a script that deploys full snapshot. The 
whole metrics part is rather small setup, so it is beneficial to deploy a whole 
stack with single command.
   We can keeps separate script for each part, but then we should add a single 
deployment script on top of those.
   
   I don't see it a blocker for PR as of now, since we just deploy new services 
and not connect everything together.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310941)
Time Spent: 2h 10m  (was: 2h)

> Provide Kubernetes setup with Prometheus
> 
>
> Key: BEAM-8131
> URL: https://issues.apache.org/jira/browse/BEAM-8131
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8131) Provide Kubernetes setup with Prometheus

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8131?focusedWorklogId=310938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310938
 ]

ASF GitHub Bot logged work on BEAM-8131:


Author: ASF GitHub Bot
Created on: 11/Sep/19 19:04
Start Date: 11/Sep/19 19:04
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on pull request #9482: [BEAM-8131] 
Provide Kubernetes setup for Prometheus
URL: https://github.com/apache/beam/pull/9482#discussion_r323409365
 
 

 ##
 File path: .test-infra/metrics/README.md
 ##
 @@ -84,23 +95,31 @@ docker-compose build
 
 # Spinup docker-compose related containers.
 docker-compose up
 
 Review comment:
   I believe it is reasonable to have a script that deploys full snapshot. The 
whole metrics part is rather small setup, so it is beneficial to deploy a whole 
stack with single command.
   We can keeps separate script for each part, but then we should add a single 
deployment script on top of those.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310938)
Time Spent: 2h  (was: 1h 50m)

> Provide Kubernetes setup with Prometheus
> 
>
> Key: BEAM-8131
> URL: https://issues.apache.org/jira/browse/BEAM-8131
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310933
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:54
Start Date: 11/Sep/19 18:54
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #9550: Revert "[BEAM-8111] 
Add ValidatesRunner test to AvroSchemaTest"
URL: https://github.com/apache/beam/pull/9550#issuecomment-530517191
 
 
   FYI this appears to be Flink ValidatesRunner, not Dataflow?
   
   On Wed, Sep 11, 2019 at 11:30 AM Mark Liu  wrote:
   
   > @markflyhigh  requested your review on:
   > #9550  Revert "[BEAM-8111] Add
   > ValidatesRunner test to AvroSchemaTest".
   >
   > —
   > You are receiving this because your review was requested.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310933)
Time Spent: 2h 10m  (was: 2h)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8111) SchemaCoder broken on DataflowRunner

[Mark Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927895#comment-16927895
 ] 

Mark Liu commented on BEAM-8111:


ValidatesRunner tests added in https://github.com/apache/beam/pull/9454 
consistently failed on Jenkins. Rollback 
https://github.com/apache/beam/pull/9550 is out to unblock 2.16 release 
validation.

However, is this jira resolved or not? Should it become a blocker for 2.16?

BTW, Brian is ooo this week. +cc: [~kenn]

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8212) StatefulParDoFn creates GC timers for every record

[Akshay Iyangar (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927893#comment-16927893
 ] 

Akshay Iyangar commented on BEAM-8212:
--

[~NathanHowell] - Adding nate to add something that i may have missed in the 
description.
[~randomsamples] 

> StatefulParDoFn creates GC timers for every record 
> ---
>
> Key: BEAM-8212
> URL: https://issues.apache.org/jira/browse/BEAM-8212
> Project: Beam
>  Issue Type: Bug
>  Components: beam-community
>Reporter: Akshay Iyangar
>Assignee: Aizhamal Nurmamat kyzy
>Priority: Major
>
> Hi 
> So currently the StatefulParDoFn create timers for all the records.
> [https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java#L211]
> This becomes a problem if you are using GlobalWindows for streaming where 
> these timers get created and never get closed since the window will never 
> close.
> This is a problem especially if your memory bound in rocksDB where these 
> timers take up potential space and sloe the pipelines considerably.
> Was wondering that if the pipeline runs in global windows we should avoid 
> adding timers to it at all?
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8212) StatefulParDoFn creates GC timers for every record

[Akshay Iyangar (Jira)]

Akshay Iyangar created BEAM-8212:


 Summary: StatefulParDoFn creates GC timers for every record 
 Key: BEAM-8212
 URL: https://issues.apache.org/jira/browse/BEAM-8212
 Project: Beam
  Issue Type: Bug
  Components: beam-community
Reporter: Akshay Iyangar
Assignee: Aizhamal Nurmamat kyzy


Hi 

So currently the StatefulParDoFn create timers for all the records.
[https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java#L211]

This becomes a problem if you are using GlobalWindows for streaming where these 
timers get created and never get closed since the window will never close.

This is a problem especially if your memory bound in rocksDB where these timers 
take up potential space and sloe the pipelines considerably.

Was wondering that if the pipeline runs in global windows we should avoid 
adding timers to it at all?

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8111) SchemaCoder broken on DataflowRunner

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8111?focusedWorklogId=310918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310918
 ]

ASF GitHub Bot logged work on BEAM-8111:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:30
Start Date: 11/Sep/19 18:30
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on pull request #9550: Revert 
"[BEAM-8111] Add ValidatesRunner test to AvroSchemaTest"
URL: https://github.com/apache/beam/pull/9550
 
 
   Reverts apache/beam#9454 due to AvroSchemaTest.testAvroPipelineGroupBy 
consistently failed in Dataflow ValidatesRunner test suite.
   
   
[This](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5178/)
 is the first failure instance. You can go to console output or Gradle scan for 
error details.
   
   +R: @reuvenlax 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310918)
Time Spent: 2h  (was: 1h 50m)

> SchemaCoder broken on DataflowRunner
> 
>
> Key: BEAM-8111
> URL: https://issues.apache.org/jira/browse/BEAM-8111
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> https://github.com/apache/beam/commit/e65c176a9f34e45d408281e1101a2ae54cef0f6c
>  broke SchemaCoder on Dataflow. When translating a schema that uses logical 
> types from a cloud object dataflow encounters a runtime error.
> This means any pipelines that use SqlTransform or schema transforms will fail 
> on Dataflow in 2.15.0



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (BEAM-6777) SDK Harness Resilience

[Yueyang Qiu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu closed BEAM-6777.
-
Fix Version/s: 2.14.0
   Resolution: Fixed

> SDK Harness Resilience
> --
>
> Key: BEAM-6777
> URL: https://issues.apache.org/jira/browse/BEAM-6777
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Sam Rohde
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.14.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> If the Python SDK Harness crashes in any way (user code exception, OOM, etc) 
> the job will hang and waste resources. The fix is to add a daemon in the SDK 
> Harness and Runner Harness to communicate with Dataflow to restart the VM 
> when stuckness is detected.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[Yueyang Qiu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu closed BEAM-7013.
-
Resolution: Fixed

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 34h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Closed] (BEAM-8126) HllCount needs to handle null sketches properly

[Yueyang Qiu (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yueyang Qiu closed BEAM-8126.
-
Fix Version/s: 2.16.0
   Resolution: Fixed

> HllCount needs to handle null sketches properly
> ---
>
> Key: BEAM-8126
> URL: https://issues.apache.org/jira/browse/BEAM-8126
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Minor
> Fix For: 2.16.0
>
>
> BigQuery uses `null` to represent result of aggregations of empty input. We 
> need to make sure `null` sketches are handled properly in Beam and users have 
> a smooth end-to-end experience using BigQuery's `HLL_COUNT` in conjunction 
> with Beam's `HllCount`.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[Robert Bradshaw (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927886#comment-16927886
 ] 

Robert Bradshaw commented on BEAM-8196:
---

Yes, I think we could make this an error if we're sure end users won't have 
issues (i.e. all the files are already there with the right timestamps). 

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310913
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:24
Start Date: 11/Sep/19 18:24
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9549: [BEAM-8165] add 
_sdk to container name
URL: https://github.com/apache/beam/pull/9549#issuecomment-530505301
 
 
   Run Go Spark ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310913)
Time Spent: 4h 40m  (was: 4.5h)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310912
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:24
Start Date: 11/Sep/19 18:24
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9549: [BEAM-8165] add 
_sdk to container name
URL: https://github.com/apache/beam/pull/9549#issuecomment-530505198
 
 
   Run Go Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310912)
Time Spent: 4.5h  (was: 4h 20m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-7013) A new count distinct transform based on BigQuery compatible HyperLogLog++ implementation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7013?focusedWorklogId=310910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310910
 ]

ASF GitHub Bot logged work on BEAM-7013:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:23
Start Date: 11/Sep/19 18:23
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on pull request #9519: [BEAM-7013] 
Use a 0-length byte array to represent empty sketch in HllCount
URL: https://github.com/apache/beam/pull/9519
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310910)
Time Spent: 34h 10m  (was: 34h)

> A new count distinct transform based on BigQuery compatible HyperLogLog++ 
> implementation
> 
>
> Key: BEAM-7013
> URL: https://issues.apache.org/jira/browse/BEAM-7013
> Project: Beam
>  Issue Type: New Feature
>  Components: extensions-java-sketching, sdk-java-core
>Reporter: Yueyang Qiu
>Assignee: Yueyang Qiu
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 34h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310908
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:21
Start Date: 11/Sep/19 18:21
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on issue #9549: [BEAM-8165] add 
_sdk to container name
URL: https://github.com/apache/beam/pull/9549#issuecomment-530503645
 
 
   Run Go PostCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310908)
Time Spent: 4h 20m  (was: 4h 10m)

> Change image name and add images to release process
> ---
>
> Key: BEAM-8165
> URL: https://issues.apache.org/jira/browse/BEAM-8165
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go, sdk-java-harness, sdk-py-harness
>Reporter: Hannah Jiang
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> change name to apachebeam/\{lang}{ver}_sdk



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8165) Change image name and add images to release process

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8165?focusedWorklogId=310906=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310906
 ]

ASF GitHub Bot logged work on BEAM-8165:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:20
Start Date: 11/Sep/19 18:20
Worklog Time Spent: 10m 
  Work Description: Hannah-Jiang commented on pull request #9549: 
[BEAM-8165] add _sdk to container name
URL: https://github.com/apache/beam/pull/9549
 
 
   R: @markflyhigh 
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)
 | --- | 

[jira] [Work logged] (BEAM-7947) Improves the interfaces of classes such as FnDataService, BundleProcessor, ActiveBundle, etc to change the parameter type from WindowedValue to T

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7947?focusedWorklogId=310903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310903
 ]

ASF GitHub Bot logged work on BEAM-7947:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:17
Start Date: 11/Sep/19 18:17
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #9496: [BEAM-7947] 
Improves the interfaces of classes such as FnDataService,…
URL: https://github.com/apache/beam/pull/9496
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310903)
Time Spent: 0.5h  (was: 20m)

> Improves the interfaces of classes such as FnDataService, BundleProcessor, 
> ActiveBundle, etc to change the parameter type from WindowedValue to T
> 
>
> Key: BEAM-7947
> URL: https://issues.apache.org/jira/browse/BEAM-7947
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Both `Coder>` and `FnDataReceiver>` use 
> `WindowedValue` as the data structure that both sides of Runner and SDK 
> Harness know each other. Control Plane/Data Plane/State Plane/Logging is a 
> highly abstraction, such as Control Plane and Logging, these are common 
> requirements for all multi-language platforms. For example, the Flink 
> community is also discussing how to support Python UDF, as well as how to 
> deal with docker environment. how to data transfer, how to state access, how 
> to logging etc. If Beam can further abstract these service interfaces, i.e., 
> interface definitions are compatible with multiple engines, and finally 
> provided to other projects in the form of class libraries, it definitely will 
> help other platforms that want to support multiple languages. Here I am to 
> throw out a minnow to catch a whale, take the FnDataService#receive interface 
> as an example, and turn `WindowedValue` into `T` so that other platforms 
> can be extended arbitrarily, as follows:
> {code}
>  InboundDataClient receive(LogicalEndpoint inputLocation, Coder coder, 
> FnDataReceiver> listener);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Resolved] (BEAM-7947) Improves the interfaces of classes such as FnDataService, BundleProcessor, ActiveBundle, etc to change the parameter type from WindowedValue to T

[Luke Cwik (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-7947.
-
Resolution: Fixed

> Improves the interfaces of classes such as FnDataService, BundleProcessor, 
> ActiveBundle, etc to change the parameter type from WindowedValue to T
> 
>
> Key: BEAM-7947
> URL: https://issues.apache.org/jira/browse/BEAM-7947
> Project: Beam
>  Issue Type: Sub-task
>  Components: java-fn-execution
>Reporter: sunjincheng
>Assignee: sunjincheng
>Priority: Major
> Fix For: 2.16.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Both `Coder>` and `FnDataReceiver>` use 
> `WindowedValue` as the data structure that both sides of Runner and SDK 
> Harness know each other. Control Plane/Data Plane/State Plane/Logging is a 
> highly abstraction, such as Control Plane and Logging, these are common 
> requirements for all multi-language platforms. For example, the Flink 
> community is also discussing how to support Python UDF, as well as how to 
> deal with docker environment. how to data transfer, how to state access, how 
> to logging etc. If Beam can further abstract these service interfaces, i.e., 
> interface definitions are compatible with multiple engines, and finally 
> provided to other projects in the form of class libraries, it definitely will 
> help other platforms that want to support multiple languages. Here I am to 
> throw out a minnow to catch a whale, take the FnDataService#receive interface 
> as an example, and turn `WindowedValue` into `T` so that other platforms 
> can be extended arbitrarily, as follows:
> {code}
>  InboundDataClient receive(LogicalEndpoint inputLocation, Coder coder, 
> FnDataReceiver> listener);
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[Ahmet Altay (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927877#comment-16927877
 ] 

Ahmet Altay commented on BEAM-8196:
---

Should we make issues with generating proto files an error here: 
https://github.com/apache/beam/blob/master/sdks/python/setup.py#L177

/cc [~robertwb]

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[Udi Meiri (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927874#comment-16927874
 ] 

Udi Meiri commented on BEAM-8196:
-

I did download the tar file and it did contain the file.
Also opened https://issues.apache.org/jira/browse/BEAM-8211 for setting a 
default timeout.

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (BEAM-8211) Set a default wait_until_finish_duration?

[Udi Meiri (Jira)]

Udi Meiri created BEAM-8211:
---

 Summary: Set a default wait_until_finish_duration?
 Key: BEAM-8211
 URL: https://issues.apache.org/jira/browse/BEAM-8211
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Udi Meiri


This would benefit python ITs.
If this value is not set, tests with workers failing to start up might take a 
full hour, causing post-commits to time out.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Commented] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[Ahmet Altay (Jira)]

[ 
https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927873#comment-16927873
 ] 

Ahmet Altay commented on BEAM-8196:
---

Setting `wait_until_finish_duration` sounds good.

For the failing job, do you have a way to check what was staged as the sdk 
tarball and does it contain the endpoints_pb2 file or not? I suspect we have an 
issue with creating the tarball.

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Work logged] (BEAM-8196) Python 3.5 post commit timed out at 100 minutes

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/BEAM-8196?focusedWorklogId=310896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-310896
 ]

ASF GitHub Bot logged work on BEAM-8196:


Author: ASF GitHub Bot
Created on: 11/Sep/19 18:06
Start Date: 11/Sep/19 18:06
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #9547: [BEAM-8196] Add 
wait_until_finish_duration
URL: https://github.com/apache/beam/pull/9547#issuecomment-530497694
 
 
   R: @aaltay 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 310896)
Time Spent: 0.5h  (was: 20m)

> Python 3.5 post commit timed out at 100 minutes
> ---
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ahmet Altay
>Assignee: Udi Meiri
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the 
> timeout? We can also look into why this postcommit was slow. A later post 
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/) 
> completed in 66 minutes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)