date:20190923



 [ 
https://issues.apache.org/jira/browse/BEAM-1296?focusedWorklogId=316503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316503
 ]

ASF GitHub Bot logged work on BEAM-1296:


Author: ASF GitHub Bot
Created on: 23/Sep/19 07:49
Start Date: 23/Sep/19 07:49
Worklog Time Spent: 10m 
  Work Description: angulartist commented on issue #9633: [BEAM-1296] 
Providing a small dataset for "Apache Beam Mobile Gaming …
URL: https://github.com/apache/beam/pull/9633#issuecomment-533993501
 
 
retest this please
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316503)
Time Spent: 20m  (was: 10m)

> Providing a small dataset for "Apache Beam Mobile Gaming Pipeline Examples"
> ---
>
> Key: BEAM-1296
> URL: https://issues.apache.org/jira/browse/BEAM-1296
> Project: Beam
>  Issue Type: Wish
>  Components: examples-java
>Reporter: Keiji Yoshida
>Priority: Trivial
>  Labels: ccoss2019, newbie, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A dataset "gs://apache-beam-samples/game/gaming_data*.csv" for "Apache Beam 
> Mobile Gaming Pipeline Examples" is so huge (about 12 GB) and it takes long 
> time to download the dataset. It might pose difficulties to Apache Beam 
> beginners who want to try "Apache Beam Mobile Gaming Pipeline Examples" 
> quickly.
> How about providing a small dataset (say less than 1 GB) for this examples?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5820) Vendor Calcite



 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=316512&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316512
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 23/Sep/19 08:07
Start Date: 23/Sep/19 08:07
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-533998611
 
 
   Run Spark Runner Nexmark Tests
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316512)
Time Spent: 11.5h  (was: 11h 20m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5820) Vendor Calcite



 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=316513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316513
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 23/Sep/19 08:07
Start Date: 23/Sep/19 08:07
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-533998690
 
 
   Run Dataflow Runner Nexmark Tests
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316513)
Time Spent: 11h 40m  (was: 11.5h)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 11h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5820) Vendor Calcite



 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=316511&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316511
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 23/Sep/19 08:07
Start Date: 23/Sep/19 08:07
Worklog Time Spent: 10m 
  Work Description: vectorijk commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-533998571
 
 
   Run Direct Runner Nexmark Tests
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316511)
Time Spent: 11h 20m  (was: 11h 10m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7660) Create ParDo Python Load Test Jenkins Job [Flink]



 [ 
https://issues.apache.org/jira/browse/BEAM-7660?focusedWorklogId=316594&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316594
 ]

ASF GitHub Bot logged work on BEAM-7660:


Author: ASF GitHub Bot
Created on: 23/Sep/19 10:27
Start Date: 23/Sep/19 10:27
Worklog Time Spent: 10m 
  Work Description: kamilwu commented on issue #9449: [BEAM-7660] Create 
Python ParDo load test job on Flink
URL: https://github.com/apache/beam/pull/9449#issuecomment-534042649
 
 
   @lgajowy I replied to your comments, please take a look once again.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316594)
Time Spent: 4h 10m  (was: 4h)

> Create ParDo Python Load Test Jenkins Job [Flink]
> -
>
> Key: BEAM-7660
> URL: https://issues.apache.org/jira/browse/BEAM-7660
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316595
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 10:29
Start Date: 23/Sep/19 10:29
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #9614: [BEAM-8025] Avoid 
race condition in cluster cleanup and introduce cluster creation retrial under 
load.
URL: https://github.com/apache/beam/pull/9614#issuecomment-534043066
 
 
@aromanenko-dev after a discussion with the Achilles author (Datastax 
evangelist) it appears that this error is due to load when the cluster is 
created. So I added a commit to retry cluster creation. Can you please take a 
look ?

 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316595)
Time Spent: 2h  (was: 1h 50m)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-7370) Beam Dependency Update Request: Sphinx

[
https://issues.apache.org/jira/browse/BEAM-7370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7370:

Description:

- 2019-05-20 16:38:07.937770 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:32:27.855338 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:02:59.052884 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-01 12:04:13.113613 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-08 12:03:15.091005 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-15 12:03:09.406918 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-22 12:03:31.157859 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:05:13.023604 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:03:03.242767 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:04:01.647619 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.1.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:02:52.342008 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:44.918642 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:28.319578 -

Please consider upgrading the dependency Sphinx.

The current version is 1.8.5. The latest version is 2.2.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.or

[jira] [Updated] (BEAM-7369) Beam Dependency Update Request: mock

[
https://issues.apache.org/jira/browse/BEAM-7369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7369:

Description:

- 2019-05-20 16:37:42.331317 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:32:01.319061 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:02:34.500201 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-01 12:03:46.078139 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-08 12:02:33.351570 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-15 12:02:47.637435 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-22 12:03:10.835914 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:04:50.359555 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:02:38.989136 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:03:38.282286 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:02:30.405376 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:24.272597 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:07.834005 -

Please consider upgrading the dependency mock.

The current version is 2.0.0. The latest version is 3.0.5

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/

[jira] [Updated] (BEAM-6089) Beam Dependency Update Request: oauth2client

[
https://issues.apache.org/jira/browse/BEAM-6089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6089:

Description:

- 2018-11-19 12:11:53.801885 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:10:31.359164 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:11:18.194090 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:13:40.021791 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:12:09.88 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:20:14.935936 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:23:14.664558 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-14 12:12:14.081917 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-21 12:18:38.928775 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-28 12:10:22.371989 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-04 12:11:00.277439 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-11 12:11:25.422782 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-18 12:21:52.210498 -

Please consider upgrading the dependency oauth2client.

The current version is 3.0.0. The latest version is 4.1.3

[jira] [Updated] (BEAM-5752) Beam Dependency Update Request: org.eclipse.jetty:jetty-server

[
https://issues.apache.org/jira/browse/BEAM-5752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5752:

Description:

- 2018-10-15 12:13:17.575546 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:26.908573 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:10.104039 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:14:56.891103 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:14:57.086704 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:38.454968 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:38.329092 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:03.778648 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:33.026807 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:54.249911 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:13.071360 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:25.376755 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-server.

[jira] [Updated] (BEAM-5748) Beam Dependency Update Request: org.conscrypt:conscrypt-openjdk

[
https://issues.apache.org/jira/browse/BEAM-5748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5748:

Description:

- 2018-10-15 12:11:29.496210 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:11:12.117107 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:13:29.155766 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:12:12.225263 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:12:03.483941 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:12:49.618981 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:11:54.540783 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:12:15.486522 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:14:35.731081 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:15:07.608265 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:21:22.979404 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:24:30.102231 -

Please consider upgrading the dependency
org.conscrypt:conscrypt-openjdk.

The current version is 1.1.3. The latest version is 1.4.2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify

[jira] [Updated] (BEAM-5753) Beam Dependency Update Request: org.eclipse.jetty:jetty-servlet

[
https://issues.apache.org/jira/browse/BEAM-5753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5753:

Description:

- 2018-10-15 12:13:19.256622 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:32.240729 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:15.388249 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:15:02.810121 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:15:02.741564 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.12.RC2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:44.462009 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:43.766710 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:15:09.151091 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:38.451204 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:59.925429 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:18.611955 -

Please consider upgrading the dependency
org.eclipse.jetty:jetty-servlet.

The current version is 9.2.10.v20150310. The latest version is
9.4.14.v20181114

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:30.751791 -

Please consider upgrading the dependency
org.eclipse.jetty:je

[jira] [Updated] (BEAM-6645) Beam Dependency Update Request: com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin

[
https://issues.apache.org/jira/browse/BEAM-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6645:

Description:

- 2019-02-11 12:12:17.710870 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-18 12:22:38.110512 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-02-25 12:12:13.220144 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.20.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-04 12:13:34.975335 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-11 12:13:20.550078 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-03-25 04:18:13.860760 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-01 12:09:02.686241 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-08 12:11:15.303458 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-15 12:30:47.279374 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-22 12:08:36.647085 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:38:23.480410 -

Please consider upgrading the dependency
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin.

The current version is 0.17.0. The latest version is 0.21.0

cc:
Please refer to [Beam Dependency Guide
|ht

[jira] [Updated] (BEAM-5750) Beam Dependency Update Request: javax.servlet:javax.servlet-api

[
https://issues.apache.org/jira/browse/BEAM-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-5750:

Description:

- 2018-10-15 12:13:14.253682 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-22 12:13:20.677351 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-10-29 12:17:03.954722 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-05 12:14:50.209888 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-12 12:14:50.401239 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-19 12:15:31.652290 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-11-26 12:14:29.396363 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-03 12:14:52.265627 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-10 12:17:21.331309 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-17 12:17:42.365110 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2018-12-31 15:24:01.296226 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-01-07 12:27:13.634593 -

Please consider upgrading the dependency
javax.servlet:javax.servlet-api.

The current version is 3.1.0. The latest version is 4.0.1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify

[jira] [Updated] (BEAM-7792) Beam Dependency Update Request: com.github.spotbugs:spotbugs

[
https://issues.apache.org/jira/browse/BEAM-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7792:

Description:

- 2019-07-22 12:15:41.216793 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:06:22.055418 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-05 12:03:56.940731 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-12 12:05:04.611973 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-19 12:03:13.035336 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-08-26 12:02:56.981076 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-02 12:03:52.780365 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-09 12:02:54.192199 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-16 12:02:39.812475 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-09-23 12:03:12.396932 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.12. The latest version is 4.0.0-beta4

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

was:

- 2019-07-22 12:15:41.216793 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-07-29 12:06:22.055418 -

Please consider upgrading the dependency com.github.spotbugs:spotbugs.

The current version is 3.1.10. The latest version is 4.0.0-beta3

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/depen

[jira] [Updated] (BEAM-6951) Beam Dependency Update Request: com.github.spotbugs:spotbugs-annotations

[
https://issues.apache.org/jira/browse/BEAM-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-6951:

Description:

- 2019-04-01 12:15:05.460427 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-08 12:15:37.305259 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-15 12:35:52.817108 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-04-22 12:13:25.261372 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:39:18.034675 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 16:54:09.180503 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-20 17:37:40.326607 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta1

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-05-27 12:06:48.236836 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-03 12:04:13.166255 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-10 12:04:08.241217 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-17 12:33:32.545061 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current version is 3.1.11. The latest version is 4.0.0-beta2

cc:
Please refer to [Beam Dependency Guide
|https://beam.apache.org/contribute/dependencies/]for more information.
Do Not Modify The Description Above.

- 2019-06-24 12:04:02.498856 -

Please consider upgrading the dependency
com.github.spotbugs:spotbugs-annotations.

The current ver

[jira] [Updated] (BEAM-8002) Beam Dependency Update Request: Gradle:



 [ 
https://issues.apache.org/jira/browse/BEAM-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Beam JIRA Bot updated BEAM-8002:

Description: 


 - 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.1 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.2 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-16 12:03:47.037326 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.2 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-23 12:04:12.360299 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.2 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


  was:


 - 2019-08-19 12:04:19.192688 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-08-26 12:04:17.802062 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-02 12:04:53.057461 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.1 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-09 12:03:53.298241 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.2 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 


 - 2019-09-16 12:03:47.037326 -

Please consider upgrading the dependency Gradle:. 

The current version is 5.2.1. The latest version is 5.6.2 

cc: 
 Please refer to [Beam Dependency Guide 
|https://beam.apache.org/contribute/dependencies/]for more information. 
Do Not Modify The Description Above. 



> Beam Dependency Update Request: Gradle:
> ---
>
> Key: BEAM-8002
> URL: https://issues.apache.org/jira/browse/BEAM-8002
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>
>  - 2019-08-19 12:04:19.192688 
> -
> Please consider upgrading the dependency Gradle:. 
> The current version is 5.2.1. The latest version is 5.6 
> cc: 
>  Please refer to [Beam Dependency Guide 
> |https://beam.apache

[jira] [Work logged] (BEAM-5820) Vendor Calcite



 [ 
https://issues.apache.org/jira/browse/BEAM-5820?focusedWorklogId=316615&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316615
 ]

ASF GitHub Bot logged work on BEAM-5820:


Author: ASF GitHub Bot
Created on: 23/Sep/19 12:10
Start Date: 23/Sep/19 12:10
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9189: [BEAM-5820] vendor 
calcite
URL: https://github.com/apache/beam/pull/9189#issuecomment-534071557
 
 
   @vectorijk this aparently still needs a rebase
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316615)
Time Spent: 11h 50m  (was: 11h 40m)

> Vendor Calcite
> --
>
> Key: BEAM-5820
> URL: https://issues.apache.org/jira/browse/BEAM-5820
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kai Jiang
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7660) Create ParDo Python Load Test Jenkins Job [Flink]



 [ 
https://issues.apache.org/jira/browse/BEAM-7660?focusedWorklogId=316618&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316618
 ]

ASF GitHub Bot logged work on BEAM-7660:


Author: ASF GitHub Bot
Created on: 23/Sep/19 12:12
Start Date: 23/Sep/19 12:12
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9449: [BEAM-7660] Create 
Python ParDo load test job on Flink
URL: https://github.com/apache/beam/pull/9449#issuecomment-534072196
 
 
   @kamilwu My excuses I somehow totally missed this one, since it seems 
@lgajowy is quite advanced with the review I won't have much to add. Sorry 
again.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316618)
Time Spent: 4h 20m  (was: 4h 10m)

> Create ParDo Python Load Test Jenkins Job [Flink]
> -
>
> Key: BEAM-7660
> URL: https://issues.apache.org/jira/browse/BEAM-7660
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Kamil Wasilewski
>Assignee: Kamil Wasilewski
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-1296) Providing a small dataset for "Apache Beam Mobile Gaming Pipeline Examples"

2019-09-23 Thread John Patoch (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Patoch reassigned BEAM-1296:
-

Assignee: John Patoch

> Providing a small dataset for "Apache Beam Mobile Gaming Pipeline Examples"
> ---
>
> Key: BEAM-1296
> URL: https://issues.apache.org/jira/browse/BEAM-1296
> Project: Beam
>  Issue Type: Wish
>  Components: examples-java
>Reporter: Keiji Yoshida
>Assignee: John Patoch
>Priority: Trivial
>  Labels: ccoss2019, newbie, starter
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A dataset "gs://apache-beam-samples/game/gaming_data*.csv" for "Apache Beam 
> Mobile Gaming Pipeline Examples" is so huge (about 12 GB) and it takes long 
> time to download the dataset. It might pose difficulties to Apache Beam 
> beginners who want to try "Apache Beam Mobile Gaming Pipeline Examples" 
> quickly.
> How about providing a small dataset (say less than 1 GB) for this examples?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-3845) Avoid calling Class#newInstance



 [ 
https://issues.apache.org/jira/browse/BEAM-3845?focusedWorklogId=316620&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316620
 ]

ASF GitHub Bot logged work on BEAM-3845:


Author: ASF GitHub Bot
Created on: 23/Sep/19 12:16
Start Date: 23/Sep/19 12:16
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #9613: [BEAM-3845] Remove 
deprecated Class.newInstance() method usage
URL: https://github.com/apache/beam/pull/9613#issuecomment-534073344
 
 
   Run Python PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316620)
Time Spent: 20m  (was: 10m)

> Avoid calling Class#newInstance
> ---
>
> Key: BEAM-3845
> URL: https://issues.apache.org/jira/browse/BEAM-3845
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Ted Yu
>Assignee: Lukasz Gajowy
>Priority: Minor
>  Labels: triaged
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Class#newInstance is deprecated starting in Java 9 - 
> https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw 
> undeclared checked exceptions.
> The suggested replacement is getDeclaredConstructor().newInstance(), which 
> wraps the checked exceptions in InvocationException.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316629&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316629
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 12:34
Start Date: 23/Sep/19 12:34
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9614: [BEAM-8025] 
Avoid race condition in cluster cleanup and introduce cluster creation retrial 
under load.
URL: https://github.com/apache/beam/pull/9614#issuecomment-534079101
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316629)
Time Spent: 2h 10m  (was: 2h)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8176) Java BigQuery performance tests don't run



 [ 
https://issues.apache.org/jira/browse/BEAM-8176?focusedWorklogId=316638&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316638
 ]

ASF GitHub Bot logged work on BEAM-8176:


Author: ASF GitHub Bot
Created on: 23/Sep/19 12:50
Start Date: 23/Sep/19 12:50
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9520: [BEAM-8176] Fix 
BigQuery Java test jobs
URL: https://github.com/apache/beam/pull/9520#issuecomment-53408
 
 
   @lgajowy can you take a look at the fix? WDYT now?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316638)
Time Spent: 6h 50m  (was: 6h 40m)

> Java BigQuery performance tests don't run
> -
>
> Key: BEAM-8176
> URL: https://issues.apache.org/jira/browse/BEAM-8176
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 2.15.0
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316670&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316670
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 13:39
Start Date: 23/Sep/19 13:39
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534103254
 
 
   @jklukas I addressed your review comments and added support of lambda and 
default handlers for `AsJsons` and `ParseJasons` transforms.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316670)
Time Spent: 1h 50m  (was: 1h 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316671&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316671
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 13:40
Start Date: 23/Sep/19 13:40
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534103254
 
 
   @jklukas I addressed your review comments and added support of lambda and 
default handlers for `AsJsons` and `ParseJasons` transforms. Could you take a 
look again?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316671)
Time Spent: 2h  (was: 1h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (BEAM-5428) Implement cross-bundle state caching.



 [ 
https://issues.apache.org/jira/browse/BEAM-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned BEAM-5428:


Assignee: Maximilian Michels  (was: Rakesh Kumar)

> Implement cross-bundle state caching.
> -
>
> Key: BEAM-5428
> URL: https://issues.apache.org/jira/browse/BEAM-5428
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 18h 20m
>  Remaining Estimate: 0h
>
> Tech spec: 
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document: 
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link: 
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8297) Evict state cache based on an upper memory limit

Maximilian Michels created BEAM-8297:


 Summary: Evict state cache based on an upper memory limit
 Key: BEAM-8297
 URL: https://issues.apache.org/jira/browse/BEAM-8297
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Maximilian Michels
Assignee: Maximilian Michels


The memory limit is currently specified in terms of a maximum number of cache 
items. We should switch to a memory-based limit which would make it easier to 
reason about the state cache size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5428) Implement cross-bundle state caching.



 [ 
https://issues.apache.org/jira/browse/BEAM-5428?focusedWorklogId=316711&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316711
 ]

ASF GitHub Bot logged work on BEAM-5428:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:24
Start Date: 23/Sep/19 14:24
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #9418: [BEAM-5428] 
Implement cross-bundle user state caching in the Python SDK
URL: https://github.com/apache/beam/pull/9418#discussion_r327143887
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
 ##
 @@ -620,6 +628,106 @@ def _next_id(self):
 return str(self._last_id)
 
 
+class CachingMaterializingStateHandler(object):
+  """ A State handler which retrieves and caches state. """
+
+  def __init__(self, global_state_cache, underlying_state):
+self._underlying = underlying_state
+self._state_cache = global_state_cache
+self._context = threading.local()
+
+  @contextlib.contextmanager
+  def process_instruction_id(self, bundle_id, cache_tokens):
+if getattr(self._context, 'cache_token', None) is not None:
+  raise RuntimeError(
+  'Cache tokens already set to %s' % self._context.cache_token)
+# TODO Also handle cache tokens for side input, if present
 
 Review comment:
   https://issues.apache.org/jira/browse/BEAM-8297
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316711)
Time Spent: 18.5h  (was: 18h 20m)

> Implement cross-bundle state caching.
> -
>
> Key: BEAM-5428
> URL: https://issues.apache.org/jira/browse/BEAM-5428
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 18.5h
>  Remaining Estimate: 0h
>
> Tech spec: 
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document: 
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link: 
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5428) Implement cross-bundle state caching.



 [ 
https://issues.apache.org/jira/browse/BEAM-5428?focusedWorklogId=316712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316712
 ]

ASF GitHub Bot logged work on BEAM-5428:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:27
Start Date: 23/Sep/19 14:27
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #9418: [BEAM-5428] 
Implement cross-bundle user state caching in the Python SDK
URL: https://github.com/apache/beam/pull/9418#discussion_r327143887
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
 ##
 @@ -620,6 +628,106 @@ def _next_id(self):
 return str(self._last_id)
 
 
+class CachingMaterializingStateHandler(object):
+  """ A State handler which retrieves and caches state. """
+
+  def __init__(self, global_state_cache, underlying_state):
+self._underlying = underlying_state
+self._state_cache = global_state_cache
+self._context = threading.local()
+
+  @contextlib.contextmanager
+  def process_instruction_id(self, bundle_id, cache_tokens):
+if getattr(self._context, 'cache_token', None) is not None:
+  raise RuntimeError(
+  'Cache tokens already set to %s' % self._context.cache_token)
+# TODO Also handle cache tokens for side input, if present
 
 Review comment:
   https://issues.apache.org/jira/browse/BEAM-8297
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316712)
Time Spent: 18h 40m  (was: 18.5h)

> Implement cross-bundle state caching.
> -
>
> Key: BEAM-5428
> URL: https://issues.apache.org/jira/browse/BEAM-5428
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 18h 40m
>  Remaining Estimate: 0h
>
> Tech spec: 
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document: 
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link: 
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8297) Evict state cache based on an upper memory limit



 [ 
https://issues.apache.org/jira/browse/BEAM-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-8297:
-
Status: Open  (was: Triage Needed)

> Evict state cache based on an upper memory limit
> 
>
> Key: BEAM-8297
> URL: https://issues.apache.org/jira/browse/BEAM-8297
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Major
>
> The memory limit is currently specified in terms of a maximum number of cache 
> items. We should switch to a memory-based limit which would make it easier to 
> reason about the state cache size.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5428) Implement cross-bundle state caching.



 [ 
https://issues.apache.org/jira/browse/BEAM-5428?focusedWorklogId=316715&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316715
 ]

ASF GitHub Bot logged work on BEAM-5428:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:29
Start Date: 23/Sep/19 14:29
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #9418: [BEAM-5428] 
Implement cross-bundle user state caching in the Python SDK
URL: https://github.com/apache/beam/pull/9418#discussion_r327147182
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/sdk_worker.py
 ##
 @@ -620,6 +628,106 @@ def _next_id(self):
 return str(self._last_id)
 
 
+class CachingMaterializingStateHandler(object):
+  """ A State handler which retrieves and caches state. """
+
+  def __init__(self, global_state_cache, underlying_state):
+self._underlying = underlying_state
+self._state_cache = global_state_cache
+self._context = threading.local()
+
+  @contextlib.contextmanager
+  def process_instruction_id(self, bundle_id, cache_tokens):
+if getattr(self._context, 'cache_token', None) is not None:
+  raise RuntimeError(
+  'Cache tokens already set to %s' % self._context.cache_token)
+# TODO Also handle cache tokens for side input, if present
 
 Review comment:
   https://issues.apache.org/jira/browse/BEAM-8298
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316715)
Time Spent: 19h  (was: 18h 50m)

> Implement cross-bundle state caching.
> -
>
> Key: BEAM-5428
> URL: https://issues.apache.org/jira/browse/BEAM-5428
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 19h
>  Remaining Estimate: 0h
>
> Tech spec: 
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document: 
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link: 
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-5428) Implement cross-bundle state caching.



 [ 
https://issues.apache.org/jira/browse/BEAM-5428?focusedWorklogId=316714&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316714
 ]

ASF GitHub Bot logged work on BEAM-5428:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:29
Start Date: 23/Sep/19 14:29
Worklog Time Spent: 10m 
  Work Description: mxm commented on pull request #9418: [BEAM-5428] 
Implement cross-bundle user state caching in the Python SDK
URL: https://github.com/apache/beam/pull/9418#discussion_r327147001
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/fn_api_runner.py
 ##
 @@ -308,7 +311,8 @@ def __init__(
   default_environment=None,
   bundle_repeat=0,
   use_state_iterables=False,
-  provision_info=None):
+  provision_info=None,
+  state_cache_size=100):
 
 Review comment:
   Jira: https://issues.apache.org/jira/browse/BEAM-8297
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316714)
Time Spent: 18h 50m  (was: 18h 40m)

> Implement cross-bundle state caching.
> -
>
> Key: BEAM-5428
> URL: https://issues.apache.org/jira/browse/BEAM-5428
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Maximilian Michels
>Priority: Major
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>
> Tech spec: 
> [https://docs.google.com/document/d/1BOozW0bzBuz4oHJEuZNDOHdzaV5Y56ix58Ozrqm2jFg/edit#heading=h.7ghoih5aig5m]
> Relevant document: 
> [https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit#|https://docs.google.com/document/d/1ltVqIW0XxUXI6grp17TgeyIybk3-nDF8a0-Nqw-s9mY/edit]
> Mailing list link: 
> [https://lists.apache.org/thread.html/caa8d9bc6ca871d13de2c5e6ba07fdc76f85d26497d95d90893aa1f6@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8298) Implement state caching for side inputs

Maximilian Michels created BEAM-8298:


 Summary: Implement state caching for side inputs
 Key: BEAM-8298
 URL: https://issues.apache.org/jira/browse/BEAM-8298
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Maximilian Michels


Caching is currently only implemented for user state.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316717&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316717
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:35
Start Date: 23/Sep/19 14:35
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #9614: [BEAM-8025] Avoid 
race condition in cluster cleanup and introduce cluster creation retrial under 
load.
URL: https://github.com/apache/beam/pull/9614#issuecomment-534127229
 
 
   Run Java PreCommit
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316717)
Time Spent: 2.5h  (was: 2h 20m)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316716&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316716
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 14:35
Start Date: 23/Sep/19 14:35
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #9614: [BEAM-8025] Avoid 
race condition in cluster cleanup and introduce cluster creation retrial under 
load.
URL: https://github.com/apache/beam/pull/9614#issuecomment-534127136
 
 
   Still failing sometimes on the cleaning. And seems not failing with 
NoHostAvailable exception anymore. I'll launch tests several times, to see
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316716)
Time Spent: 2h 20m  (was: 2h 10m)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8299) Upgrade Jackson to version 2.9.10

2019-09-23 Thread Jira

Ismaël Mejía created BEAM-8299:
--

 Summary: Upgrade Jackson to version 2.9.10
 Key: BEAM-8299
 URL: https://issues.apache.org/jira/browse/BEAM-8299
 Project: Beam
  Issue Type: Improvement
  Components: build-system, sdk-java-core
Affects Versions: 2.15.0
Reporter: Ismaël Mejía
Assignee: Ismaël Mejía


[Jackson 2.9.10 addresses multiple CVE 
issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10] from 
previous Jackson versions, so we need to upgrade it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8299) Upgrade Jackson to version 2.9.10

2019-09-23 Thread Jira



 [ 
https://issues.apache.org/jira/browse/BEAM-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-8299:
---
Status: Open  (was: Triage Needed)

> Upgrade Jackson to version 2.9.10
> -
>
> Key: BEAM-8299
> URL: https://issues.apache.org/jira/browse/BEAM-8299
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Blocker
>
> [Jackson 2.9.10 addresses multiple CVE 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10] from 
> previous Jackson versions, so we need to upgrade it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8299) Upgrade Jackson to version 2.9.10

2019-09-23 Thread Jira



 [ 
https://issues.apache.org/jira/browse/BEAM-8299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-8299:
---
Fix Version/s: 2.16.0

> Upgrade Jackson to version 2.9.10
> -
>
> Key: BEAM-8299
> URL: https://issues.apache.org/jira/browse/BEAM-8299
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [Jackson 2.9.10 addresses multiple CVE 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10] from 
> previous Jackson versions, so we need to upgrade it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8299) Upgrade Jackson to version 2.9.10



 [ 
https://issues.apache.org/jira/browse/BEAM-8299?focusedWorklogId=316742&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316742
 ]

ASF GitHub Bot logged work on BEAM-8299:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:02
Start Date: 23/Sep/19 15:02
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9636: [BEAM-8299] 
Upgrade Jackson to version 2.9.10
URL: https://github.com/apache/beam/pull/9636
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316742)
Remaining Estimate: 0h
Time Spent: 10m

> Upgrade Jackson to version 2.9.10
> -
>
> Key: BEAM-8299
> URL: https://issues.apache.org/jira/browse/BEAM-8299
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [Jackson 2.9.10 addresses multiple CVE 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10] from 
> previous Jackson versions, so we need to upgrade it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8299) Upgrade Jackson to version 2.9.10



 [ 
https://issues.apache.org/jira/browse/BEAM-8299?focusedWorklogId=316750&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316750
 ]

ASF GitHub Bot logged work on BEAM-8299:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:22
Start Date: 23/Sep/19 15:22
Worklog Time Spent: 10m 
  Work Description: iemejia commented on pull request #9637: 
[release-2.16.0][BEAM-8299] Upgrade Jackson to version 2.9.10
URL: https://github.com/apache/beam/pull/9637
 
 
   Cherry pick of #9636 (done in parallel)
   R: @pabloem 
   CC: @markflyhigh 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316750)
Time Spent: 20m  (was: 10m)

> Upgrade Jackson to version 2.9.10
> -
>
> Key: BEAM-8299
> URL: https://issues.apache.org/jira/browse/BEAM-8299
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Blocker
> Fix For: 2.16.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> [Jackson 2.9.10 addresses multiple CVE 
> issues|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.9.10] from 
> previous Jackson versions, so we need to upgrade it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8281) Resize file-based IOITs



 [ 
https://issues.apache.org/jira/browse/BEAM-8281?focusedWorklogId=316751&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316751
 ]

ASF GitHub Bot logged work on BEAM-8281:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:24
Start Date: 23/Sep/19 15:24
Worklog Time Spent: 10m 
  Work Description: mwalenia commented on issue #9638: [BEAM-8281] Resize 
IOITs datasets
URL: https://github.com/apache/beam/pull/9638#issuecomment-534149648
 
 
   R: @lgajowy 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316751)
Remaining Estimate: 0h
Time Spent: 10m

> Resize file-based IOITs 
> 
>
> Key: BEAM-8281
> URL: https://issues.apache.org/jira/browse/BEAM-8281
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Michal Walenia
>Assignee: Michal Walenia
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Resize the IOITs to use known amounts of data saved on GCS so that it's 
> possible to report dataset size and measure throughput as part of the test 
> metrics



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316755&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316755
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:42
Start Date: 23/Sep/19 15:42
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9499: [BEAM-8100] Exception 
handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534158013
 
 
   I'm having a realization that we could probably avoid most of the complexity 
here if `ParseJsons` simply returned the `MapElements` transform it builds 
rather than wrapping it. I don't think that's particularly an option since 
users already have code that expects an object of type `ParseJsons`, but what 
if we exposed a method to return the underlying `MapElements` so that users 
could call the existing error handling methods? It might look like:
   
   ```
   PCollection p = ...
   p.apply(ParseJsons
  .of(MyClass.class)
  .unwrapped()
  .exceptionsVia(...))
   ```
   
   This is slightly less convenient for the user, as they have to know about 
and invoke the `unwrapped` method (we can probably come up with a better name), 
but it avoids us having to re-implement all the logic from `MapElements` and it 
would establish a pattern by which other simple transforms that wrap a 
`MapElements` invocation could provide error handling.
   
   What do you think of that approach?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316755)
Time Spent: 2h 10m  (was: 2h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316760&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316760
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 15:56
Start Date: 23/Sep/19 15:56
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #9614: [BEAM-8025] Avoid 
race condition in cluster cleanup and introduce cluster creation retrial under 
load.
URL: https://github.com/apache/beam/pull/9614#issuecomment-534164189
 
 
   I found it, it was due to cassandra running some compactions some times (to 
avoid fragmentation of files). And in that case if the directory was suppressed 
by the TemporaryFolder JUnit Rule (between the execution of the tests), then 
the compaction which needs to write in the temp folder, fails. => So I simply 
removed the TemporaryFolder rule and used a basic tmp folder that is purged at 
the end of the tests. PTAL
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316760)
Time Spent: 2h 40m  (was: 2.5h)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316764
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:06
Start Date: 23/Sep/19 16:06
Worklog Time Spent: 10m 
  Work Description: jklukas commented on issue #9499: [BEAM-8100] Exception 
handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534158013
 
 
   I'm having a realization that we could probably avoid most of the complexity 
here if `ParseJsons` simply returned the `MapElements` transform it builds 
rather than wrapping it. I don't think that's particularly an option since 
users already have code that expects an object of type `ParseJsons`, but what 
if we exposed a method to return the underlying `MapElements` so that users 
could call the existing error handling methods? It might look like:
   
   ```
   PCollection p = ...
   p.apply(ParseJsons
  .of(MyClass.class)
  .unwrapped()
  .exceptionsVia(...))
   ```
   
   This is slightly less convenient for the user, as they have to know about 
and invoke the `unwrapped` method (we can probably come up with a better name, 
(`toMap`, `asMap`, `asMapElements`)?), but it avoids us having to re-implement 
all the logic from `MapElements` and it would establish a pattern by which 
other simple transforms that wrap a `MapElements` invocation could provide 
error handling.
   
   What do you think of that approach?
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316764)
Time Spent: 2h 20m  (was: 2h 10m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316788&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316788
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:23
Start Date: 23/Sep/19 16:23
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534175290
 
 
   It is easier to maintain but I do think it would be easy for us to have a
   copy of all the methods call the underlying MapElements without needing for
   users to know to invoke unwrapped. Some people discover the API via code
   completion and unwrapped is an unlikely place users will look.
   
   On Mon, Sep 23, 2019 at 8:42 AM Jeff Klukas 
   wrote:
   
   > I'm having a realization that we could probably avoid most of the
   > complexity here if ParseJsons simply returned the MapElements transform
   > it builds rather than wrapping it. I don't think that's particularly an
   > option since users already have code that expects an object of type
   > ParseJsons, but what if we exposed a method to return the underlying
   > MapElements so that users could call the existing error handling methods?
   > It might look like:
   >
   > PCollection p = ...
   > p.apply(ParseJsons
   >.of(MyClass.class)
   >.unwrapped()
   >.exceptionsVia(...))
   >
   > This is slightly less convenient for the user, as they have to know about
   > and invoke the unwrapped method (we can probably come up with a better
   > name), but it avoids us having to re-implement all the logic from
   > MapElements and it would establish a pattern by which other simple
   > transforms that wrap a MapElements invocation could provide error
   > handling.
   >
   > What do you think of that approach?
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316788)
Time Spent: 2.5h  (was: 2h 20m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8025) Cassandra IO classMethod test is flaky



 [ 
https://issues.apache.org/jira/browse/BEAM-8025?focusedWorklogId=316795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316795
 ]

ASF GitHub Bot logged work on BEAM-8025:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:33
Start Date: 23/Sep/19 16:33
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on pull request #9614: 
[BEAM-8025] Avoid race condition in cluster cleanup and introduce cluster 
creation retrial under load.
URL: https://github.com/apache/beam/pull/9614#discussion_r327198827
 
 

 ##
 File path: 
sdks/java/io/cassandra/src/test/java/org/apache/beam/sdk/io/cassandra/CassandraIOTest.java
 ##
 @@ -106,21 +110,25 @@
   private static Cluster cluster;
   private static Session session;
 
-  @ClassRule public static final TemporaryFolder TEMPORARY_FOLDER = new 
TemporaryFolder();
+  private static final String TEMPORARY_FOLDER = 
System.getProperty("java.io.tmpdir");
   @Rule public transient TestPipeline pipeline = TestPipeline.create();
   private static CassandraShutDownHook shutdownHook;
 
   @BeforeClass
   public static void beforeClass() throws Exception {
 jmxPort = NetworkTestHelper.getAvailableLocalPort();
 shutdownHook = new CassandraShutDownHook();
-// randomized port at startup
-String data = TEMPORARY_FOLDER.newFolder("embedded-cassandra", 
"data").getPath();
-String commitLog = TEMPORARY_FOLDER.newFolder("embedded-cassandra", 
"commit-log").getPath();
-String cdcRaw = TEMPORARY_FOLDER.newFolder("embedded-cassandra", 
"cdc-raw").getPath();
-String hints = TEMPORARY_FOLDER.newFolder("embedded-cassandra", 
"hints").getPath();
-String savedCache = TEMPORARY_FOLDER.newFolder("embedded-cassandra", 
"saved-cache").getPath();
-cluster =
+String data = TEMPORARY_FOLDER + "/embedded-cassandra/data";
 
 Review comment:
   Please, extract `TEMPORARY_FOLDER + "/embedded-cassandra/"` into another 
const string since it appears in many other places in this test.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316795)
Time Spent: 2h 50m  (was: 2h 40m)

> Cassandra IO classMethod test is flaky
> --
>
> Key: BEAM-8025
> URL: https://issues.apache.org/jira/browse/BEAM-8025
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-cassandra, test-failures
>Affects Versions: 2.16.0
>Reporter: Kyle Weaver
>Assignee: Etienne Chauchot
>Priority: Blocker
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The most recent runs of this test are failing:
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7398/]
> [https://builds.apache.org/job/beam_PreCommit_Java_Commit/7399/console]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316799
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:40
Start Date: 23/Sep/19 16:40
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534181988
 
 
   Thanks for working on this. I would have liked to review this PR,  but did 
not even know it existed..
   
   Couple issues:
   
   * We should probably show how to run the wordcount example on the runner 
page (it was removed from the portability page)
   * There is still a gradle command left. Doesn't the Flink runner wrapper 
eliminate the need to have that gradle build step?
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316799)
Time Spent: 1h 40m  (was: 1.5h)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316798&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316798
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:40
Start Date: 23/Sep/19 16:40
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534181988
 
 
   Thanks for working on this/ I would have liked to review this PR,  but did 
not even know it existed..
   
   Couple issues:
   
   * We should probably show how to run the wordcount example on the runner 
page (it was removed from the portability page)
   * There is still a gradle command left. Doesn't the Flink runner wrapper 
eliminate the need to have that gradle build step?
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316798)
Time Spent: 1.5h  (was: 1h 20m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316804
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:44
Start Date: 23/Sep/19 16:44
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #9605: [BEAM-8233] [BEAM-8214] 
[BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534183703
 
 
   @tweise These are great follows ups. Want to open a PR? The Gradle command 
is still required to build from master. For releases it is not necessary. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316804)
Time Spent: 1h 50m  (was: 1h 40m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316809
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:49
Start Date: 23/Sep/19 16:49
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534185517
 
 
   I like the idea of simplifying the needed boiler code but I tend to agree 
with @lukecwik that, from user perspective, it will be lees evident. So, 
"wrapping" does this job. 
   
   Not sure that this feature (exception handling) is requested by other 
transforms as some asked for Json ones, but it would be great to have simple 
and unified way how to add it into transform API and how to use it. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316809)
Time Spent: 2h 40m  (was: 2.5h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316811
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:51
Start Date: 23/Sep/19 16:51
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534186111
 
 
   @mxm these are instructions for users. We can mention what people can do 
with master as well, but first we need to cover the case of a user working with 
a release. That is also why I think changes to these docs will benefit from 
more eyes.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316811)
Time Spent: 2h  (was: 1h 50m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316818&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316818
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:56
Start Date: 23/Sep/19 16:56
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#issuecomment-534185517
 
 
   I like the idea of simplifying the needed boiler-plate code but I tend to 
agree with @lukecwik that, from user perspective, it will be lees evident. So, 
"wrapping" does this job. 
   
   Not sure that this feature (exception handling) is requested by other 
transforms as some asked for Json ones, but it would be great to have simple 
and unified way how to add it into transform API and how to use it. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316818)
Time Spent: 2h 50m  (was: 2h 40m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316817&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316817
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:56
Start Date: 23/Sep/19 16:56
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534188126
 
 
   @ibzib should the JIRAs be resolved? Regardless of this PR, Beam has a 
problem with JIRA status. Users get confused to find tickets open for which 
changes have been merged long ago. It is time to bring back the topic of the 
committer merging a PR also being responsible for resolving the ticket.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316817)
Time Spent: 2h 10m  (was: 2h)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316819
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 16:56
Start Date: 23/Sep/19 16:56
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #9605: [BEAM-8233] [BEAM-8214] 
[BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534188367
 
 
   Yes I agree but props for Kyle's initiative to start with that. There is a 
balance to strike in terms of the total number of reviewers when requesting 
reviews from people. The current status is already much better and we can 
iterate further on it using your feedback. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316819)
Time Spent: 2h 20m  (was: 2h 10m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-6923) OOM errors in jobServer when using GCS artifactDir

2019-09-23 Thread Lukasz Gajowy (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy updated BEAM-6923:

Attachment: beam6923-flink156.m4v

> OOM errors in jobServer when using GCS artifactDir
> --
>
> Key: BEAM-6923
> URL: https://issues.apache.org/jira/browse/BEAM-6923
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Lukasz Gajowy
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Instance counts.png, Paths to GC root.png, 
> Telemetries.png, beam6923-flink156.m4v, heapdump size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket: 
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related 
> pipeline options: 
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099 
> --defaultEnvironmentType=DOCKER 
> --defaultEnvironmentConfig=gcr.io//java:latest'{code}
>  
> I'm facing a series of OOM errors, like this: 
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: 
> Java heap space
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at 
> com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>  
> This does not happen when I'm using a local filesystem for the artifact 
> staging location. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316831&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316831
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:03
Start Date: 23/Sep/19 17:03
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534191228
 
 
   @mxm there is actually no harm tagging more people on a PR. Broader review 
is good and it may help with faster feedback as well. Not everyone needs to 
respond. It is sufficient to have a single approval.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316831)
Time Spent: 2.5h  (was: 2h 20m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-6923) OOM errors in jobServer when using GCS artifactDir

2019-09-23 Thread Lukasz Gajowy (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy updated BEAM-6923:

Attachment: beam6923flink182.m4v

> OOM errors in jobServer when using GCS artifactDir
> --
>
> Key: BEAM-6923
> URL: https://issues.apache.org/jira/browse/BEAM-6923
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Lukasz Gajowy
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Instance counts.png, Paths to GC root.png, 
> Telemetries.png, beam6923-flink156.m4v, beam6923flink182.m4v, heapdump 
> size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket: 
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related 
> pipeline options: 
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099 
> --defaultEnvironmentType=DOCKER 
> --defaultEnvironmentConfig=gcr.io//java:latest'{code}
>  
> I'm facing a series of OOM errors, like this: 
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: 
> Java heap space
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at 
> com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>  
> This does not happen when I'm using a local filesystem for the artifact 
> staging location. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316836&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316836
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:14
Start Date: 23/Sep/19 17:14
Worklog Time Spent: 10m 
  Work Description: y1chi commented on pull request #9639: [BEAM-7919] Add 
MongoDB IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_P

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316837&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316837
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:15
Start Date: 23/Sep/19 17:15
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534196043
 
 
   Run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316837)
Time Spent: 20m  (was: 10m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-6923) OOM errors in jobServer when using GCS artifactDir

2019-09-23 Thread Lukasz Gajowy (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936050#comment-16936050
 ] 

Lukasz Gajowy commented on BEAM-6923:
-

[~angoenka] This is odd - maybe it is happening only on my machine? Let me know 
if this is the case.
 I've attached 2 videos that show step by step instructions on how to reproduce 
the issue on 1.5 and 1.8 Flink versions.
 - [^beam6923-flink156.m4v]
 - [^beam6923flink182.m4v]

I think it's best to download them - I don't know why JIRA doesn't let you play 
them with sound...

> OOM errors in jobServer when using GCS artifactDir
> --
>
> Key: BEAM-6923
> URL: https://issues.apache.org/jira/browse/BEAM-6923
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Lukasz Gajowy
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Instance counts.png, Paths to GC root.png, 
> Telemetries.png, beam6923-flink156.m4v, beam6923flink182.m4v, heapdump 
> size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket: 
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related 
> pipeline options: 
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099 
> --defaultEnvironmentType=DOCKER 
> --defaultEnvironmentConfig=gcr.io//java:latest'{code}
>  
> I'm facing a series of OOM errors, like this: 
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: 
> Java heap space
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at 
> com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>  
> This does not happen when I'm using a local filesystem for the artifact 
> staging location. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316842&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316842
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232550
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
+ProcessFunction, FailureT> 
exceptionHandler) {
+  return new AsJsonsWithFailures<>(
+  new InferableFunction, 
FailureT>(
+  exceptionHandler) {},
+  failureType);
+}
+
+@Override
+public WithFailures.Result, FailureT> 
expand(PCollection input) {
+  return input.apply(
+  MapElements.into(TypeDescriptors.strings())
+  .via(
+  Contextful.fn(
+  (Contextful.Fn) (input1, c) -> 
writeValue(input1),
+  Requirements.empty()))
+  .exceptionsInto(failureType)
+  .exceptionsVia(exceptionHandler));
+}
+  }
+
+  /**
+   * A default handler that extracts information from an exception to a {@code 
Map}
 
 Review comment:
   The info from this class doc would be perfect for filling out Javadoc for 
`exceptionsVia` that instantiates this.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316842)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316844
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232852
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -73,4 +139,66 @@ public OutputT apply(String input) {
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link ParseJsons}. 
*/
+  public class ParseJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+ParseJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public ParseJsonsWithFailures exceptionsVia(
 
 Review comment:
   Javadoc needed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316844)
Time Spent: 3h 10m  (was: 3h)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316843
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327232693
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/ParseJsons.java
 ##
 @@ -55,6 +67,61 @@ private ParseJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, with the given type descriptor used for the failure 
collection but the
+   * exception handler yet to be specified using {@link
+   * ParseJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new ParseJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link ParseJsonsWithFailures} transform that catches 
exceptions raised while
+   * parsing elements, passing the raised exception instance and the input 
element being processed
+   * through the given {@code exceptionHandler} and emitting the result to a 
failure collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * json.apply(
+   * ParseJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid POJOs
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  ParseJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new ParseJsonsWithFailures<>(
+exceptionHandler, exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public ParseJsonsWithFailures>> 
exceptionsVia() {
 
 Review comment:
   Javadoc needed.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316843)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316840
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327231280
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -56,6 +70,60 @@ private AsJsons(Class outputClass) {
 return newTransform;
   }
 
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, with the given type descriptor used for the 
failure collection but the
+   * exception handler yet to be specified using {@link
+   * AsJsonsWithFailures#exceptionsVia(ProcessFunction)}.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsInto(
+  TypeDescriptor failureTypeDescriptor) {
+return new AsJsonsWithFailures<>(null, failureTypeDescriptor);
+  }
+
+  /**
+   * Returns a new {@link AsJsonsWithFailures} transform that catches 
exceptions raised while
+   * writing JSON elements, passing the raised exception instance and the 
input element being
+   * processed through the given {@code exceptionHandler} and emitting the 
result to a failure
+   * collection.
+   *
+   * See {@link WithFailures} documentation for usage patterns of the 
returned {@link
+   * WithFailures.Result}.
+   *
+   * Example usage:
+   *
+   * {@code
+   * WithFailures.Result, KV>> 
result =
+   * pojos.apply(
+   * AsJsons.of(MyPojo.class)
+   * .exceptionsVia(new 
WithFailures.ExceptionAsMapHandler() {}));
+   *
+   * PCollection output = result.output(); // valid json elements
+   * PCollection>> failures = result.failures();
+   * }
+   */
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
+  public  AsJsonsWithFailures exceptionsVia(
+  InferableFunction, FailureT> 
exceptionHandler) {
+return new AsJsonsWithFailures<>(exceptionHandler, 
exceptionHandler.getOutputTypeDescriptor());
+  }
+
+  @Experimental(Experimental.Kind.WITH_EXCEPTIONS)
 
 Review comment:
   Need Javadoc with example usage here.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316840)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8100) Add exception handling to Json transforms in Java SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-8100?focusedWorklogId=316841&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316841
 ]

ASF GitHub Bot logged work on BEAM-8100:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:19
Start Date: 23/Sep/19 17:19
Worklog Time Spent: 10m 
  Work Description: jklukas commented on pull request #9499: [BEAM-8100] 
Exception handling for AsJsons and ParseJsons
URL: https://github.com/apache/beam/pull/9499#discussion_r327231959
 
 

 ##
 File path: 
sdks/java/extensions/jackson/src/main/java/org/apache/beam/sdk/extensions/jackson/AsJsons.java
 ##
 @@ -64,13 +132,75 @@ private AsJsons(Class outputClass) {
   @Override
   public String apply(InputT input) {
 try {
-  ObjectMapper mapper = 
Optional.fromNullable(customMapper).or(DEFAULT_MAPPER);
-  return mapper.writeValueAsString(input);
+  return writeValue(input);
 } catch (IOException e) {
   throw new RuntimeException(
   "Failed to serialize " + inputClass.getName() + " value: 
" + input, e);
 }
   }
 }));
   }
+
+  /** A {@code PTransform} that adds exception handling to {@link AsJsons}. */
+  public class AsJsonsWithFailures
+  extends PTransform, 
WithFailures.Result, FailureT>> {
+
+@Nullable
+private InferableFunction, FailureT> 
exceptionHandler;
+
+@Nullable private final transient TypeDescriptor failureType;
+
+AsJsonsWithFailures(
+InferableFunction, FailureT> 
exceptionHandler,
+TypeDescriptor failureType) {
+  this.exceptionHandler = exceptionHandler;
+  this.failureType = failureType;
+}
+
+public AsJsonsWithFailures exceptionsVia(
 
 Review comment:
   Needs Javadoc.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316841)
Time Spent: 3h  (was: 2h 50m)

> Add exception handling to Json transforms in Java SDK
> -
>
> Key: BEAM-8100
> URL: https://issues.apache.org/jira/browse/BEAM-8100
> Project: Beam
>  Issue Type: Improvement
>  Components: extensions-java-json, sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Follow-up to https://issues.apache.org/jira/browse/BEAM-5638. We have 
> exception handling for MapElements and FlatMapElements. It should be 
> straightforward to add parallel signatures to AsJsons and ParseJsons for 
> catching parsing errors.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8240) Fix pipeline proto to contain worker_harness_container_image override



 [ 
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=316845&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316845
 ]

ASF GitHub Bot logged work on BEAM-8240:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:22
Start Date: 23/Sep/19 17:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #9629: [BEAM-8240] 
Sets workerHarnessContaienrImage in the default Environment of DataflowRunner
URL: https://github.com/apache/beam/pull/9629#discussion_r327234524
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 ##
 @@ -1319,6 +1322,60 @@ public void testTemplateRunnerFullCompletion() throws 
Exception {
 expectedLogs.verifyInfo("Template successfully created");
   }
 
+  /**
+   * Tests that when {@link 
DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+   * option is set, {@link DataflowRunner} sets that value as the {@link
+   * DockerPayload#getContainerImage()} of the default {@link Environment} 
used when generating the
+   * model pipeline proto.
+   */
+  @Test
+  public void testSetWorkerHarnessContainerImageInPipelineProto() throws 
Exception {
+File existingFile = tmpFolder.newFile();
+DataflowPipelineOptions options = 
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+
+String containerImage = "gcr.io/IMAGE/foo";
+
options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+options.setJobName("TestJobName");
+options.setGcpCredential(new TestCredential());
+options.setPathValidatorClass(NoopPathValidator.class);
+options.setProject("test-project");
+options.setRunner(DataflowRunner.class);
+
+// Setting a template location so that pipeline does not actually try to 
execute on Dataflow
 
 Review comment:
   use buildPipelineOptions() which sets up the mocks and reduces your test 
boilerplate.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316845)
Time Spent: 4h  (was: 3h 50m)

> Fix pipeline proto to contain worker_harness_container_image override
> -
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container 
> within environment field when building pipeline proto.
>  
> Passing in the experiment *worker_harness_container_image=YYY* doesn't 
> override the pipeline proto environment field and it is still being populated 
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8240) Fix pipeline proto to contain worker_harness_container_image override



 [ 
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=316846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316846
 ]

ASF GitHub Bot logged work on BEAM-8240:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:22
Start Date: 23/Sep/19 17:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #9629: [BEAM-8240] 
Sets workerHarnessContaienrImage in the default Environment of DataflowRunner
URL: https://github.com/apache/beam/pull/9629#discussion_r327213316
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 ##
 @@ -1319,6 +1322,60 @@ public void testTemplateRunnerFullCompletion() throws 
Exception {
 expectedLogs.verifyInfo("Template successfully created");
   }
 
+  /**
+   * Tests that when {@link 
DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+   * option is set, {@link DataflowRunner} sets that value as the {@link
+   * DockerPayload#getContainerImage()} of the default {@link Environment} 
used when generating the
+   * model pipeline proto.
+   */
+  @Test
+  public void testSetWorkerHarnessContainerImageInPipelineProto() throws 
Exception {
+File existingFile = tmpFolder.newFile();
+DataflowPipelineOptions options = 
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+
+String containerImage = "gcr.io/IMAGE/foo";
+
options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+options.setJobName("TestJobName");
+options.setGcpCredential(new TestCredential());
+options.setPathValidatorClass(NoopPathValidator.class);
+options.setProject("test-project");
+options.setRunner(DataflowRunner.class);
+
+// Setting a template location so that pipeline does not actually try to 
execute on Dataflow
+// service.
+options.setTemplateLocation(existingFile.getPath());
+
+options.setTempLocation(tmpFolder.getRoot().getPath());
+Pipeline p = Pipeline.create(options);
+
+p.run();
+
+File stagingDirectory =
+new 
File(options.as(DataflowPipelineOptions.class).getStagingLocation());
+assertTrue(stagingDirectory.isDirectory());
+File[] files = stagingDirectory.listFiles();
+
+File protoFile = null;
+for (File file : files) {
+  String filePath = file.getAbsolutePath();
+  if (filePath.contains("pipeline") && filePath.endsWith("pb")) {
+assertTrue("Cannot uniquely identify the proto pipeline.", (protoFile 
== null));
+protoFile = file;
+  }
+}
+
+byte[] protoBytes = Files.readAllBytes(protoFile.toPath());
+RunnerApi.Pipeline pipeline = RunnerApi.Pipeline.parseFrom(protoBytes);
+
+assertEquals(1, pipeline.getComponents().getEnvironmentsCount());
+assertEquals(1, pipeline.getComponents().getEnvironmentsCount());
+Environment defaultEnvironment =
+
pipeline.getComponents().getEnvironmentsMap().values().iterator().next();
 
 Review comment:
   ```suggestion
   
Iterables.getOnlyElement(pipeline.getComponents().getEnvironmentsMap().values());
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316846)
Time Spent: 4h 10m  (was: 4h)

> Fix pipeline proto to contain worker_harness_container_image override
> -
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container 
> within environment field when building pipeline proto.
>  
> Passing in the experiment *worker_harness_container_image=YYY* doesn't 
> override the pipeline proto environment field and it is still being populated 
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8240) Fix pipeline proto to contain worker_harness_container_image override



 [ 
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=316847&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316847
 ]

ASF GitHub Bot logged work on BEAM-8240:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:22
Start Date: 23/Sep/19 17:22
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on pull request #9629: [BEAM-8240] 
Sets workerHarnessContaienrImage in the default Environment of DataflowRunner
URL: https://github.com/apache/beam/pull/9629#discussion_r327213039
 
 

 ##
 File path: 
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java
 ##
 @@ -1319,6 +1322,60 @@ public void testTemplateRunnerFullCompletion() throws 
Exception {
 expectedLogs.verifyInfo("Template successfully created");
   }
 
+  /**
+   * Tests that when {@link 
DataflowPipelineOptions#setWorkerHarnessContainerImage(String)} pipeline
+   * option is set, {@link DataflowRunner} sets that value as the {@link
+   * DockerPayload#getContainerImage()} of the default {@link Environment} 
used when generating the
+   * model pipeline proto.
+   */
+  @Test
+  public void testSetWorkerHarnessContainerImageInPipelineProto() throws 
Exception {
+File existingFile = tmpFolder.newFile();
+DataflowPipelineOptions options = 
PipelineOptionsFactory.as(DataflowPipelineOptions.class);
+
+String containerImage = "gcr.io/IMAGE/foo";
+
options.as(DataflowPipelineOptions.class).setWorkerHarnessContainerImage(containerImage);
+options.setJobName("TestJobName");
+options.setGcpCredential(new TestCredential());
+options.setPathValidatorClass(NoopPathValidator.class);
+options.setProject("test-project");
+options.setRunner(DataflowRunner.class);
+
+// Setting a template location so that pipeline does not actually try to 
execute on Dataflow
+// service.
+options.setTemplateLocation(existingFile.getPath());
+
+options.setTempLocation(tmpFolder.getRoot().getPath());
+Pipeline p = Pipeline.create(options);
+
+p.run();
+
+File stagingDirectory =
+new 
File(options.as(DataflowPipelineOptions.class).getStagingLocation());
+assertTrue(stagingDirectory.isDirectory());
+File[] files = stagingDirectory.listFiles();
+
+File protoFile = null;
+for (File file : files) {
+  String filePath = file.getAbsolutePath();
+  if (filePath.contains("pipeline") && filePath.endsWith("pb")) {
+assertTrue("Cannot uniquely identify the proto pipeline.", (protoFile 
== null));
+protoFile = file;
+  }
+}
+
+byte[] protoBytes = Files.readAllBytes(protoFile.toPath());
+RunnerApi.Pipeline pipeline = RunnerApi.Pipeline.parseFrom(protoBytes);
+
+assertEquals(1, pipeline.getComponents().getEnvironmentsCount());
 
 Review comment:
   ```suggestion
   ```
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316847)
Time Spent: 4h 10m  (was: 4h)

> Fix pipeline proto to contain worker_harness_container_image override
> -
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container 
> within environment field when building pipeline proto.
>  
> Passing in the experiment *worker_harness_container_image=YYY* doesn't 
> override the pipeline proto environment field and it is still being populated 
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316852&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316852
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:26
Start Date: 23/Sep/19 17:26
Worklog Time Spent: 10m 
  Work Description: mxm commented on issue #9605: [BEAM-8233] [BEAM-8214] 
[BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534200626
 
 
   Slightly diverging from the topic here. I don't think it is as easy as 
tagging the entire set of Beam committers on every PR. Choosing the right 
audience for a PR is vital. You want to avoid swamping people with PRs, but you 
also want a good review. Tagging someone means taking time from that person and 
it also implies leaving some time for the person to comment, as well as 
replying to the comments. 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316852)
Time Spent: 2h 40m  (was: 2.5h)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316851&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316851
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:26
Start Date: 23/Sep/19 17:26
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534200604
 
 
   Run Python MongoDBIO_IT
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316851)
Time Spent: 0.5h  (was: 20m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316856
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:36
Start Date: 23/Sep/19 17:36
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534204470
 
 
   Run Python MongoDBIO_IT
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316856)
Time Spent: 40m  (was: 0.5h)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316857&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316857
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:36
Start Date: 23/Sep/19 17:36
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534200604
 
 
   Run Python MongoDBIO_IT
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316857)
Time Spent: 50m  (was: 40m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316859
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:38
Start Date: 23/Sep/19 17:38
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534196043
 
 
   Run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316859)
Time Spent: 1h  (was: 50m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316861
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:38
Start Date: 23/Sep/19 17:38
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534205168
 
 
   Run seed job
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316861)
Time Spent: 1h 20m  (was: 1h 10m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316860&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316860
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:38
Start Date: 23/Sep/19 17:38
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534204470
 
 
   Run Python MongoDBIO_IT
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316860)
Time Spent: 1h 10m  (was: 1h)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8300) KinesisIO.write causes NPE as the producer is null

2019-09-23 Thread Ankit Jhalaria (Jira)

Ankit Jhalaria created BEAM-8300:


 Summary: KinesisIO.write causes NPE as the producer is null
 Key: BEAM-8300
 URL: https://issues.apache.org/jira/browse/BEAM-8300
 Project: Beam
  Issue Type: Bug
  Components: io-java-kinesis
Affects Versions: 2.15.0
Reporter: Ankit Jhalaria
Assignee: Ankit Jhalaria
 Fix For: Not applicable


While using KinesisIO.write(), we encountered a NPE with the following stack 
trace 

 

```java
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
 org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
java.lang.Thread.run(Thread.java:748)\nCaused by: 
java.lang.NullPointerException: null\n\tat 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669)
```



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-8300) KinesisIO.write causes NPE as the producer is null

2019-09-23 Thread Ankit Jhalaria (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Jhalaria updated BEAM-8300:
-
Description: 
While using KinesisIO.write(), we encountered a NPE with the following stack 
trace 
{code:java}
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
 org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
java.lang.Thread.run(Thread.java:748)\nCaused by: 
java.lang.NullPointerException: null\n\tat 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669){code}

  was:
While using KinesisIO.write(), we encountered a NPE with the following stack 
trace 

 

```java
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
 org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
java.lang.Thread.run(Thread.java:748)\nCaused by: 
java.lang.NullPointerException: null\n\tat 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
 
org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669)
```


> KinesisIO.write causes NPE as the producer is null
> --
>
> Key: BEAM-8300
> URL: https://issues.apache.org/jira/browse/BEAM-8300
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.15.0
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Fix For: Not applicable
>
>
> While using KinesisIO.write(), we encountered a NPE with the following stack 
> trace 
> {code:java}
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
>  org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
> java.lang.Thread.run(Thread.java:748)\nCaused by: 
> java.lang.NullPointerException: null\n\tat 
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
>  
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8300) KinesisIO.write causes NPE as the producer is null

2019-09-23 Thread Ankit Jhalaria (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-8300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936070#comment-16936070
 ] 

Ankit Jhalaria commented on BEAM-8300:
--

Errors were throws when we hit
 * 
[https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java#L685]
 * 
[https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java#L663]

> KinesisIO.write causes NPE as the producer is null
> --
>
> Key: BEAM-8300
> URL: https://issues.apache.org/jira/browse/BEAM-8300
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.15.0
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Fix For: Not applicable
>
>
> While using KinesisIO.write(), we encountered a NPE with the following stack 
> trace 
> {code:java}
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
>  org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
> java.lang.Thread.run(Thread.java:748)\nCaused by: 
> java.lang.NullPointerException: null\n\tat 
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
>  
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8240) Fix pipeline proto to contain worker_harness_container_image override



 [ 
https://issues.apache.org/jira/browse/BEAM-8240?focusedWorklogId=316870&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316870
 ]

ASF GitHub Bot logged work on BEAM-8240:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:46
Start Date: 23/Sep/19 17:46
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #9629: [BEAM-8240] Sets 
workerHarnessContaienrImage in the default Environment of DataflowRunner
URL: https://github.com/apache/beam/pull/9629#issuecomment-534208564
 
 
   Thanks Luke. PTAL.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316870)
Time Spent: 4h 20m  (was: 4h 10m)

> Fix pipeline proto to contain worker_harness_container_image override
> -
>
> Key: BEAM-8240
> URL: https://issues.apache.org/jira/browse/BEAM-8240
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 2.17.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> SDK harness incorrectly identifies itself when using custom SDK container 
> within environment field when building pipeline proto.
>  
> Passing in the experiment *worker_harness_container_image=YYY* doesn't 
> override the pipeline proto environment field and it is still being populated 
> with *gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190802*
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8301) Argument inference breaks on incomparable types as defaults.

2019-09-23 Thread Robert Bradshaw (Jira)

Robert Bradshaw created BEAM-8301:
-

 Summary: Argument inference breaks on incomparable types as 
defaults.
 Key: BEAM-8301
 URL: https://issues.apache.org/jira/browse/BEAM-8301
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Affects Versions: 2.16.0
Reporter: Robert Bradshaw
 Fix For: 2.16.0


A common culprit is numpy arrays, e.g.

{code:python}
class MyDoFn(beam.DoFn):
  def process(element, arg=np.ndarray(...)):
... 

{code}

This bug was introduced as part of [BEAM-7060].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8300) KinesisIO.write causes NPE as the producer is null



 [ 
https://issues.apache.org/jira/browse/BEAM-8300?focusedWorklogId=316871&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316871
 ]

ASF GitHub Bot logged work on BEAM-8300:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:49
Start Date: 23/Sep/19 17:49
Worklog Time Spent: 10m 
  Work Description: jhalaria commented on pull request #9640: [BEAM-8300]: 
KinesisIO.write throws NPE because producer is null
URL: https://github.com/apache/beam/pull/9640
 
 
   Added a readObject method to initialize the transient producer
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apach

[jira] [Work logged] (BEAM-8233) Separate loopback and docker modes on Flink runner guide



 [ 
https://issues.apache.org/jira/browse/BEAM-8233?focusedWorklogId=316872&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316872
 ]

ASF GitHub Bot logged work on BEAM-8233:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:50
Start Date: 23/Sep/19 17:50
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #9605: [BEAM-8233] 
[BEAM-8214] [BEAM-8232] Document environment_type flag
URL: https://github.com/apache/beam/pull/9605#issuecomment-534210168
 
 
   And leaving some time is a good idea regardless who is tagged on the PR :)
   
   When I look for reviewers for my PRs I try to pick folks that I know are 
most knowledgeable with the area or have expressed interest in the topic. I 
have never come across a case that required to tag all committers so far :)
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316872)
Time Spent: 2h 50m  (was: 2h 40m)

> Separate loopback and docker modes on Flink runner guide
> 
>
> Key: BEAM-8233
> URL: https://issues.apache.org/jira/browse/BEAM-8233
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink, website
>Reporter: Kyle Weaver
>Assignee: Kyle Weaver
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Running loopback should be the "getting started" option, and docker mode 
> should be an "advanced" option with its own section of the Flink runner guide 
> with instructions and explanations (you need to build the docker container 
> images, you can't see your output in a local filesystem without 
> workarounds..) [https://beam.apache.org/documentation/runners/flink/]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8300) KinesisIO.write causes NPE as the producer is null



 [ 
https://issues.apache.org/jira/browse/BEAM-8300?focusedWorklogId=316875&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316875
 ]

ASF GitHub Bot logged work on BEAM-8300:


Author: ASF GitHub Bot
Created on: 23/Sep/19 17:56
Start Date: 23/Sep/19 17:56
Worklog Time Spent: 10m 
  Work Description: jhalaria commented on issue #9640: [BEAM-8300]: 
KinesisIO.write throws NPE because producer is null
URL: https://github.com/apache/beam/pull/9640#issuecomment-534212637
 
 
   @iemejia - Please review.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316875)
Time Spent: 20m  (was: 10m)

> KinesisIO.write causes NPE as the producer is null
> --
>
> Key: BEAM-8300
> URL: https://issues.apache.org/jira/browse/BEAM-8300
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.15.0
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> While using KinesisIO.write(), we encountered a NPE with the following stack 
> trace 
> {code:java}
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:297)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93)\n\tat
>  
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:45)\n\tat
>  
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)\n\tat
>  org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)\n\tat 
> java.lang.Thread.run(Thread.java:748)\nCaused by: 
> java.lang.NullPointerException: null\n\tat 
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.flushBundle(KinesisIO.java:685)\n\tat
>  
> org.apache.beam.sdk.io.kinesis.KinesisIO$Write$KinesisWriterFn.finishBundle(KinesisIO.java:669){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316892&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316892
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 18:17
Start Date: 23/Sep/19 18:17
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534221012
 
 
   Run Python MongoDBIO_IT
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316892)
Time Spent: 1.5h  (was: 1h 20m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-8242) Go: unregistered Go functions fail when using -buildmode=pie



 [ 
https://issues.apache.org/jira/browse/BEAM-8242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-8242.

Fix Version/s: Not applicable
   Resolution: Fixed

> Go: unregistered Go functions fail when using -buildmode=pie
> 
>
> Key: BEAM-8242
> URL: https://issues.apache.org/jira/browse/BEAM-8242
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: 2.15.0
> Environment: GNU/Linux
>Reporter: Ian Lance Taylor
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>   Original Estimate: 0h
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> If a Go program is built with -buildmode=pie, the code that transfers an 
> unregistered function fails.  It looks up the symbol in the symbol table, but 
> that is not the location of the function at execution time.  This causes a 
> program crash when calling the function.
> I have a patch for this problem that I will send shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-8096) Allow runner to configure "subnetwork"



 [ 
https://issues.apache.org/jira/browse/BEAM-8096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Burke resolved BEAM-8096.

Fix Version/s: Not applicable
   Resolution: Fixed

> Allow runner to configure "subnetwork"
> --
>
> Key: BEAM-8096
> URL: https://issues.apache.org/jira/browse/BEAM-8096
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Affects Versions: 2.15.0
>Reporter: Jack Whelpton
>Assignee: Robert Burke
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When running a Dataflow job, the network can be specified using the --network 
> flag; however, there is no support for doing the same for the subnetwork. 
> This would be the go equivalent of the following Java code:
> [https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/options/DataflowPipelineWorkerPoolOptions.html#getSubnetwork--|https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java#L151]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-8272) GroupIntoBatches transform for Go SDK



[ 
https://issues.apache.org/jira/browse/BEAM-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936099#comment-16936099
 ] 

Robert Burke commented on BEAM-8272:


Note that the implementation will necessarily be different in the Go SDK. The 
SDK doesn't yet support the State and Timers API, which both the  Java and 
Python implementations use.

> GroupIntoBatches transform for Go SDK
> -
>
> Key: BEAM-8272
> URL: https://issues.apache.org/jira/browse/BEAM-8272
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: John Patoch
>Priority: Major
>
> Add a PTransform that batches inputs to a desired batch size. Batches will 
> contain only elements of a single key.
> It should offer the same API as its Java counterpart:
> [https://github.com/apache/beam/blob/11a977b8b26eff2274d706541127c19dc93131a2/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java]
>  
> And Python counterpart:
> https://github.com/apache/beam/blob/c445fdfdfab4a191aa780210564199f2873f85d8/sdks/python/apache_beam/transforms/util.py#L684



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (BEAM-8272) GroupIntoBatches transform for Go SDK



[ 
https://issues.apache.org/jira/browse/BEAM-8272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936099#comment-16936099
 ] 

Robert Burke edited comment on BEAM-8272 at 9/23/19 6:29 PM:
-

Note that the implementation will necessarily be different in the Go SDK. The 
SDK doesn't yet support the State and Timers API, which both the  Java and 
Python implementations use. Adding state and timers to the Go SDK is a larger 
task.

Though, this looks like a largely streaming construct, which makes alternative 
implementations without State and Timers tricky, if not impossible. 

It also looks like it requires being able to emit  "Iterables" which might be 
handle-able with slices instead, but otherwise the SDK doesn't yet support user 
side streams.


was (Author: lostluck):
Note that the implementation will necessarily be different in the Go SDK. The 
SDK doesn't yet support the State and Timers API, which both the  Java and 
Python implementations use.

> GroupIntoBatches transform for Go SDK
> -
>
> Key: BEAM-8272
> URL: https://issues.apache.org/jira/browse/BEAM-8272
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-go
>Reporter: John Patoch
>Priority: Major
>
> Add a PTransform that batches inputs to a desired batch size. Batches will 
> contain only elements of a single key.
> It should offer the same API as its Java counterpart:
> [https://github.com/apache/beam/blob/11a977b8b26eff2274d706541127c19dc93131a2/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/GroupIntoBatches.java]
>  
> And Python counterpart:
> https://github.com/apache/beam/blob/c445fdfdfab4a191aa780210564199f2873f85d8/sdks/python/apache_beam/transforms/util.py#L684



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8146) SchemaCoder/RowCoder have no equals() function



 [ 
https://issues.apache.org/jira/browse/BEAM-8146?focusedWorklogId=316918&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316918
 ]

ASF GitHub Bot logged work on BEAM-8146:


Author: ASF GitHub Bot
Created on: 23/Sep/19 18:46
Start Date: 23/Sep/19 18:46
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9493: [BEAM-8146] Add 
equals and hashCode to SchemaCoder and RowCoder
URL: https://github.com/apache/beam/pull/9493#issuecomment-534232168
 
 
   Run Apex ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316918)
Time Spent: 50m  (was: 40m)

> SchemaCoder/RowCoder have no equals() function
> --
>
> Key: BEAM-8146
> URL: https://issues.apache.org/jira/browse/BEAM-8146
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> SchemaCoder has no equals function, so it can't be compared in tests, like 
> CloudComponentsTests$DefaultCoders, which is being re-enabled in 
> https://github.com/apache/beam/pull/9446



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8146) SchemaCoder/RowCoder have no equals() function



 [ 
https://issues.apache.org/jira/browse/BEAM-8146?focusedWorklogId=316917&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316917
 ]

ASF GitHub Bot logged work on BEAM-8146:


Author: ASF GitHub Bot
Created on: 23/Sep/19 18:46
Start Date: 23/Sep/19 18:46
Worklog Time Spent: 10m 
  Work Description: TheNeuralBit commented on issue #9493: [BEAM-8146] Add 
equals and hashCode to SchemaCoder and RowCoder
URL: https://github.com/apache/beam/pull/9493#issuecomment-534232127
 
 
   Run Flink ValidatesRunner
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316917)
Time Spent: 40m  (was: 0.5h)

> SchemaCoder/RowCoder have no equals() function
> --
>
> Key: BEAM-8146
> URL: https://issues.apache.org/jira/browse/BEAM-8146
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> SchemaCoder has no equals function, so it can't be compared in tests, like 
> CloudComponentsTests$DefaultCoders, which is being re-enabled in 
> https://github.com/apache/beam/pull/9446



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8213) Run and report python tox tasks separately within Jenkins



 [ 
https://issues.apache.org/jira/browse/BEAM-8213?focusedWorklogId=316922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316922
 ]

ASF GitHub Bot logged work on BEAM-8213:


Author: ASF GitHub Bot
Created on: 23/Sep/19 18:57
Start Date: 23/Sep/19 18:57
Worklog Time Spent: 10m 
  Work Description: chadrik commented on pull request #9642: [BEAM-8213] 
Split up monolithic python preCommit tests on jenkins
URL: https://github.com/apache/beam/pull/9642
 
 
   See the Jira issue for details: 
https://issues.apache.org/jira/browse/BEAM-8213#
   
   
   
   Thank you for your contribution! Follow this checklist to help us 
incorporate your contribution quickly and easily:
   
- [ ] [**Choose 
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and 
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/)
 | --- | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[![Build
 
Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](h

[jira] [Created] (BEAM-8302) beam_PostCommit_XVR_Flink failing

2019-09-23 Thread Robert Bradshaw (Jira)

Robert Bradshaw created BEAM-8302:
-

 Summary: beam_PostCommit_XVR_Flink failing
 Key: BEAM-8302
 URL: https://issues.apache.org/jira/browse/BEAM-8302
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Robert Bradshaw
Assignee: Robert Bradshaw


E.g. see https://builds.apache.org/job/beam_PostCommit_XVR_Flink/432/console



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (BEAM-8303) Filesystems not properly registered using FileIO.write()

2019-09-23 Thread Preston Koprivica (Jira)

Preston Koprivica created BEAM-8303:
---

 Summary: Filesystems not properly registered using FileIO.write()
 Key: BEAM-8303
 URL: https://issues.apache.org/jira/browse/BEAM-8303
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Affects Versions: 2.15.0
Reporter: Preston Koprivica


I’m getting the following error when attempting to use the FileIO apis 
(beam-2.15.0) and integrating with AWS S3.  I have setup the PipelineOptions 
with all the relevant AWS options, so the filesystem registry **should** be 
properly seeded by the time the graph is compiled and executed:
{code:java}
 java.lang.IllegalArgumentException: No filesystem found for scheme s3
    at 
org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456)
    at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526)
    at 
org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1149)
    at 
org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1105)
    at org.apache.beam.sdk.coders.Coder.decode(Coder.java:159)
    at org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:83)
    at org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:32)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:543)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:534)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:480)
    at 
org.apache.beam.runners.flink.translation.types.CoderTypeSerializer.deserialize(CoderTypeSerializer.java:93)
    at 
org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
    at 
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:106)
    at 
org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:72)
    at 
org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:47)
    at 
org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:73)
    at 
org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:107)
    at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503)
    at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
    at java.lang.Thread.run(Thread.java:748)
 {code}
For reference, the write code resembles this:
{code:java}
 FileIO.Write write = FileIO.write()
    .via(ParquetIO.sink(schema))
    .to(options.getOutputDir()). // will be something like: 
s3:///
    .withSuffix(".parquet");

records.apply(String.format("Write(%s)", options.getOutputDir()), write);{code}
The issue does not appear to be related to ParquetIO.sink().  I am able to 
reliably reproduce the issue using JSON formatted records and TextIO.sink(), as 
well.  Moreover, AvroIO is affected if withWindowedWrites() option is added.

Just trying some different knobs, I went ahead and set the following option:
{code:java}
write = write.withNoSpilling();{code}
This actually seemed to fix the issue, only to have it reemerge as I scaled up 
the data set size.  The stack trace, while very similar, reads:
{code:java}
 java.lang.IllegalArgumentException: No filesystem found for scheme s3
    at 
org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456)
    at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526)
    at 
org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1149)
    at 
org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1105)
    at org.apache.beam.sdk.coders.Coder.decode(Coder.java:159)
    at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:82)
    at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:36)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:543)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:534)
    at 
org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:480)
    at 
org.apache.beam.runners.flink.translation.types.CoderTypeSerializer.deserialize(CoderTypeSerializer.java:93)
    at 
org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
    at 
org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:106)
    at 
org.apache.flink.runtime.

[jira] [Commented] (BEAM-8303) Filesystems not properly registered using FileIO.write()

2019-09-23 Thread Preston Koprivica (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936130#comment-16936130
 ] 

Preston Koprivica commented on BEAM-8303:
-

I'll defer to the experts on the priority of this issue.  Currently, I am able 
to workaround it by setting FileIO.write().withIgnoreWindowing(), which is also 
the default for AvroIO 
([https://github.com/apache/beam/blob/release-2.15.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L516]),
 and I suspect other FileBasedSink apis as well.

> Filesystems not properly registered using FileIO.write()
> 
>
> Key: BEAM-8303
> URL: https://issues.apache.org/jira/browse/BEAM-8303
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.15.0
>Reporter: Preston Koprivica
>Priority: Major
>
> I’m getting the following error when attempting to use the FileIO apis 
> (beam-2.15.0) and integrating with AWS S3.  I have setup the PipelineOptions 
> with all the relevant AWS options, so the filesystem registry **should** be 
> properly seeded by the time the graph is compiled and executed:
> {code:java}
>  java.lang.IllegalArgumentException: No filesystem found for scheme s3
>     at 
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456)
>     at 
> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526)
>     at 
> org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1149)
>     at 
> org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1105)
>     at org.apache.beam.sdk.coders.Coder.decode(Coder.java:159)
>     at 
> org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:83)
>     at 
> org.apache.beam.sdk.transforms.join.UnionCoder.decode(UnionCoder.java:32)
>     at 
> org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:543)
>     at 
> org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:534)
>     at 
> org.apache.beam.sdk.util.WindowedValue$FullWindowedValueCoder.decode(WindowedValue.java:480)
>     at 
> org.apache.beam.runners.flink.translation.types.CoderTypeSerializer.deserialize(CoderTypeSerializer.java:93)
>     at 
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
>     at 
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:106)
>     at 
> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNextRecord(AbstractRecordReader.java:72)
>     at 
> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(MutableRecordReader.java:47)
>     at 
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterator.java:73)
>     at 
> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:107)
>     at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503)
>     at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
>     at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>     at java.lang.Thread.run(Thread.java:748)
>  {code}
> For reference, the write code resembles this:
> {code:java}
>  FileIO.Write write = FileIO.write()
>     .via(ParquetIO.sink(schema))
>     .to(options.getOutputDir()). // will be something like: 
> s3:///
>     .withSuffix(".parquet");
> records.apply(String.format("Write(%s)", options.getOutputDir()), 
> write);{code}
> The issue does not appear to be related to ParquetIO.sink().  I am able to 
> reliably reproduce the issue using JSON formatted records and TextIO.sink(), 
> as well.  Moreover, AvroIO is affected if withWindowedWrites() option is 
> added.
> Just trying some different knobs, I went ahead and set the following option:
> {code:java}
> write = write.withNoSpilling();{code}
> This actually seemed to fix the issue, only to have it reemerge as I scaled 
> up the data set size.  The stack trace, while very similar, reads:
> {code:java}
>  java.lang.IllegalArgumentException: No filesystem found for scheme s3
>     at 
> org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:456)
>     at 
> org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:526)
>     at 
> org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1149)
>     at 
> org.apache.beam.sdk.io.FileBasedSink$FileResultCoder.decode(FileBasedSink.java:1105)
>     at org.apache.beam.sdk.coders.Coder.decode(Coder.java:159)
>     at org.apache.beam.sdk.coders.KvCoder.decode(KvCoder.java:82)
>     at org.apache.beam.sdk.coders.KvCoder.

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316930
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 19:13
Start Date: 23/Sep/19 19:13
Worklog Time Spent: 10m 
  Work Description: y1chi commented on issue #9639: [BEAM-7919] Add MongoDB 
IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#issuecomment-534242730
 
 
   R: @tvalentyn 
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316930)
Time Spent: 1h 40m  (was: 1.5h)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-7919) Add a Python 3 test scenario for MongoDB IO



 [ 
https://issues.apache.org/jira/browse/BEAM-7919?focusedWorklogId=316932&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-316932
 ]

ASF GitHub Bot logged work on BEAM-7919:


Author: ASF GitHub Bot
Created on: 23/Sep/19 19:14
Start Date: 23/Sep/19 19:14
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on pull request #9639: [BEAM-7919] 
Add MongoDB IO integration test for py3.7
URL: https://github.com/apache/beam/pull/9639#discussion_r327283975
 
 

 ##
 File path: .test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
 ##
 @@ -32,6 +32,7 @@ 
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_MongoDBIO_IT',
 gradle {
   rootBuildScriptDir(commonJobProperties.checkoutDir)
   tasks(':sdks:python:test-suites:direct:py2:mongodbioIT')
+  tasks(':sdks:python:test-suites:direct:py37:mongodbioIT')
 
 Review comment:
   If we test only one minor version, I suggest python 3.5 as this is the 
lowest version beam supports.
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 316932)
Time Spent: 1h 50m  (was: 1h 40m)

> Add a Python 3 test scenario for MongoDB IO
> ---
>
> Key: BEAM-7919
> URL: https://issues.apache.org/jira/browse/BEAM-7919
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-ideas
>Reporter: Valentyn Tymofieiev
>Assignee: Yichi Zhang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Python 2 MongoDB IO suite was added in:
> https://github.com/apache/beam/commit/17bf89d6070565b715f44ecb5f6394219b94cfe6
> We should also exercise this IO in Python 3. 
> cc: [~chamikara] [~altay]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (BEAM-7045) Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow

2019-09-23 Thread Yichi Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yichi Zhang resolved BEAM-7045.
---
Resolution: Fixed

> Element counters in the Web UI graph representations for transforms for 
> Python streaming jobs in Google Cloud Dataflow
> --
>
> Key: BEAM-7045
> URL: https://issues.apache.org/jira/browse/BEAM-7045
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow, sdk-py-core
> Environment: GCP Dataflow
>Reporter: Fim
>Priority: Major
>  Labels: features, usability
> Fix For: 2.16.0
>
>
> Users don't see the element counters in transforms in the Web UI graph 
> representation when running a Python streaming job, which is expected 
> behavior according to [this Beam 
> page|https://beam.apache.org/documentation/sdks/python-streaming/#dataflowrunner-specific-features].
> The feature request is to enable the element counters in the Web UI graph 
> representations for transforms for Python streaming jobs in Google Cloud 
> Dataflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (BEAM-7045) Element counters in the Web UI graph representations for transforms for Python streaming jobs in Google Cloud Dataflow

2019-09-23 Thread Yichi Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/BEAM-7045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yichi Zhang updated BEAM-7045:
--
Fix Version/s: 2.16.0

> Element counters in the Web UI graph representations for transforms for 
> Python streaming jobs in Google Cloud Dataflow
> --
>
> Key: BEAM-7045
> URL: https://issues.apache.org/jira/browse/BEAM-7045
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-dataflow, sdk-py-core
> Environment: GCP Dataflow
>Reporter: Fim
>Priority: Major
>  Labels: features, usability
> Fix For: 2.16.0
>
>
> Users don't see the element counters in transforms in the Web UI graph 
> representation when running a Python streaming job, which is expected 
> behavior according to [this Beam 
> page|https://beam.apache.org/documentation/sdks/python-streaming/#dataflowrunner-specific-features].
> The feature request is to enable the element counters in the Web UI graph 
> representations for transforms for Python streaming jobs in Google Cloud 
> Dataflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (BEAM-6923) OOM errors in jobServer when using GCS artifactDir

2019-09-23 Thread Ankur Goenka (Jira)



[ 
https://issues.apache.org/jira/browse/BEAM-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936132#comment-16936132
 ] 

Ankur Goenka commented on BEAM-6923:


Thats strange.

I am using linux for testing this.

Will try it on mac as well.

> OOM errors in jobServer when using GCS artifactDir
> --
>
> Key: BEAM-6923
> URL: https://issues.apache.org/jira/browse/BEAM-6923
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Lukasz Gajowy
>Assignee: Ankur Goenka
>Priority: Major
> Attachments: Instance counts.png, Paths to GC root.png, 
> Telemetries.png, beam6923-flink156.m4v, beam6923flink182.m4v, heapdump 
> size-sorted.png
>
>
> When starting jobServer with artifactDir pointing to a GCS bucket: 
> {code:java}
> ./gradlew :beam-runners-flink_2.11-job-server:runShadow 
> -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket{code}
> and running a Java portable pipeline with the following, portability related 
> pipeline options: 
> {code:java}
> --runner=PortableRunner --jobEndpoint=localhost:8099 
> --defaultEnvironmentType=DOCKER 
> --defaultEnvironmentConfig=gcr.io//java:latest'{code}
>  
> I'm facing a series of OOM errors, like this: 
> {code:java}
> Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: 
> Java heap space
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
> at 
> com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
> at 
> com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>  
> This does not happen when I'm using a local filesystem for the artifact 
> staging location. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (BEAM-8299) Upgrade Jackson to version 2.9.10