[jira] [Assigned] (BEAM-4337) Enforce ErrorProne analysis in the Cassandra IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4337:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the Cassandra IO project
> ---
>
> Key: BEAM-4337
> URL: https://issues.apache.org/jira/browse/BEAM-4337
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-cassandra
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-cassandra}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-cassandra:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/cassandra}} project.
> # In {{sdks/java/io/cassandra/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4338) Enforce ErrorProne analysis in the common IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4338:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the common IO project
> 
>
> Key: BEAM-4338
> URL: https://issues.apache.org/jira/browse/BEAM-4338
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-common}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-common:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/common}} project.
> # In {{sdks/java/io/common/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4355) Enforce ErrorProne analysis in the xml IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4355:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the xml IO project
> -
>
> Key: BEAM-4355
> URL: https://issues.apache.org/jira/browse/BEAM-4355
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-xml}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-xml:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/xml}} project.
> # In {{sdks/java/io/xml/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4354) Enforce ErrorProne analysis in the tika IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4354:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the tika IO project
> --
>
> Key: BEAM-4354
> URL: https://issues.apache.org/jira/browse/BEAM-4354
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-tika
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-tika}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-tika:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/tika}} project.
> # In {{sdks/java/io/tika/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4352) Enforce ErrorProne analysis in the redis IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4352:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the redis IO project
> ---
>
> Key: BEAM-4352
> URL: https://issues.apache.org/jira/browse/BEAM-4352
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-redis
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-redis}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-redis:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/redis}} project.
> # In {{sdks/java/io/redis/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4346) Enforce ErrorProne analysis in the jms IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4346:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the jms IO project
> -
>
> Key: BEAM-4346
> URL: https://issues.apache.org/jira/browse/BEAM-4346
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jms
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-jms}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-jms:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/jms}} project.
> # In {{sdks/java/io/jms/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #289

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-4351) Enforce ErrorProne analysis in the mqtt IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4351:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the mqtt IO project
> --
>
> Key: BEAM-4351
> URL: https://issues.apache.org/jira/browse/BEAM-4351
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-mqtt
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-mqtt}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-mqtt:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/mqtt}} project.
> # In {{sdks/java/io/mqtt/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4350) Enforce ErrorProne analysis in the mqtt IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4350:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the mqtt IO project
> --
>
> Key: BEAM-4350
> URL: https://issues.apache.org/jira/browse/BEAM-4350
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-mongodb
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-mongodb}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-mongodb:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/mongodb}} project.
> # In {{sdks/java/io/mongodb/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4336) Enforce ErrorProne analysis in the amqp IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4336:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the amqp IO project
> --
>
> Key: BEAM-4336
> URL: https://issues.apache.org/jira/browse/BEAM-4336
> Project: Beam
>  Issue Type: Improvement
>  Components: io-ideas
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-amqp}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-amqp:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/amqp}} project.
> # In {{sdks/java/io/amqp/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4345) Enforce ErrorProne analysis in the jdbc IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4345:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the jdbc IO project
> --
>
> Key: BEAM-4345
> URL: https://issues.apache.org/jira/browse/BEAM-4345
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-jdbc
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-jdbc}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-jdbc:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/jdbc}} project.
> # In {{sdks/java/io/jdbc/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4353) Enforce ErrorProne analysis in the solr IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4353:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the solr IO project
> --
>
> Key: BEAM-4353
> URL: https://issues.apache.org/jira/browse/BEAM-4353
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-solr
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-solr}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-solr:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/solr}} project.
> # In {{sdks/java/io/solr/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4349) Enforce ErrorProne analysis in the mongodb IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4349:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the mongodb IO project
> -
>
> Key: BEAM-4349
> URL: https://issues.apache.org/jira/browse/BEAM-4349
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-mongodb
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-mongodb}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-mongodb:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/mongodb}} project.
> # In {{sdks/java/io/mongodb/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4348) Enforce ErrorProne analysis in the kinesis IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4348:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the kinesis IO project
> -
>
> Key: BEAM-4348
> URL: https://issues.apache.org/jira/browse/BEAM-4348
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kinesis
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kinesis}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kinesis:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kinesis}} project.
> # In {{sdks/java/io/kinesis/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4347) Enforce ErrorProne analysis in the kafka IO project

[Tim Robertson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson reassigned BEAM-4347:
---

Assignee: Tim Robertson

> Enforce ErrorProne analysis in the kafka IO project
> ---
>
> Key: BEAM-4347
> URL: https://issues.apache.org/jira/browse/BEAM-4347
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Scott Wegner
>Assignee: Tim Robertson
>Priority: Minor
>  Labels: errorprone, starter
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-io-kafka}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-io-kafka:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/io/kafka}} project.
> # In {{sdks/java/io/kafka/build.gradle}}, add {{failOnWarning: true}} to the 
> call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (7b0cd4e -> b03c26e)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7b0cd4e  Merge pull request #5405: [BEAM-4276] Update generated Go SDK 
protobuf files.
 add 3577e6e  Fix error-prone and some static analysis warnings in 
io/hadoop-file-system
 new b03c26e  Merge pull request #5381: Fix error-prone warnings for 
io/hadoop-file-system

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/io/hdfs/HadoopFileSystem.java  |  4 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemModule.java   |  9 +--
 .../beam/sdk/io/hdfs/HadoopFileSystemOptions.java  |  1 +
 .../apache/beam/sdk/io/hdfs/HadoopResourceId.java  |  1 +
 .../sdk/io/hdfs/HadoopFileSystemModuleTest.java|  3 +-
 .../sdk/io/hdfs/HadoopFileSystemOptionsTest.java   |  1 +
 .../sdk/io/hdfs/HadoopFileSystemRegistrarTest.java |  2 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 87 +++---
 .../beam/sdk/io/hdfs/HadoopResourceIdTest.java |  4 +-
 9 files changed, 58 insertions(+), 54 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
t...@apache.org.

[beam] 01/01: Merge pull request #5381: Fix error-prone warnings for io/hadoop-file-system

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b03c26e61315c7184337af9fc6bdba8877fe047b
Merge: 7b0cd4e 3577e6e
Author: Thomas Weise 
AuthorDate: Thu May 17 22:31:30 2018 -0700

Merge pull request #5381: Fix error-prone warnings for io/hadoop-file-system

 .../apache/beam/sdk/io/hdfs/HadoopFileSystem.java  |  4 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemModule.java   |  9 +--
 .../beam/sdk/io/hdfs/HadoopFileSystemOptions.java  |  1 +
 .../apache/beam/sdk/io/hdfs/HadoopResourceId.java  |  1 +
 .../sdk/io/hdfs/HadoopFileSystemModuleTest.java|  3 +-
 .../sdk/io/hdfs/HadoopFileSystemOptionsTest.java   |  1 +
 .../sdk/io/hdfs/HadoopFileSystemRegistrarTest.java |  2 +-
 .../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 87 +++---
 .../beam/sdk/io/hdfs/HadoopResourceIdTest.java |  4 +-
 9 files changed, 58 insertions(+), 54 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
t...@apache.org.

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103223
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189163987
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
+
+It includes:
+  - PrintFn (DoFn) to inspect element, window, and timestamp.
+  - AddTimestampFn (DoFn) to modify timestamps.
+  - assert_that via check_gbk_format and equal_to_per_window (matchers).
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+
+import six
+
+import apache_beam as beam
+import apache_beam.transforms.window as window
+from apache_beam.examples.wordcount import WordExtractingDoFn
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to_per_window
+from apache_beam.transforms.core import ParDo
+
+
+class PrintFn(beam.DoFn):
+  """A DoFn that prints label, element, its window, and its timstamp. """
+  def __init__(self, label):
+self.label = label
+
+  def process(self, element, timestamp=beam.DoFn.TimestampParam,
+  window=beam.DoFn.WindowParam):
+# Log at INFO level each element processed. When executing this pipeline
+# using the Dataflow service, these log lines will appear in the Cloud
+# Logging UI.
+logging.info('[%s]: %s %s %s', self.label, element, window, timestamp)
+yield element
+
+
+class AddTimestampFn(beam.DoFn):
+  """A DoFn that attaches timestamps to its elements.
+
+  It takes a string of integers and it attaches to each of them
+  a timestamp of its same value."""
+  def process(self, element):
+for elem in element.split(' '):
+  logging.info('Adding timestamp to: %s', element)
+  yield beam.window.TimestampedValue(elem, int(elem))
+
+
+def run(argv=None):
+  """Build and run the pipeline."""
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+  '--output_topic', required=True,
+  help=('Output PubSub topic of the form '
+'"projects//topic/".'))
+  parser.add_argument(
+  '--use_assert_that', action='store_true',
+  help=('See that outputs are of the form '
+'"word:count".'))
+  group = parser.add_mutually_exclusive_group(required=True)
+  group.add_argument(
+  '--input_topic',
+  help=('Input PubSub topic of the form '
+'"projects//topics/".'))
+  group.add_argument(
+  '--input_subscription',
+  help=('Input PubSub subscription of the form '
+'"projects//subscriptions/."'))
+  known_args, pipeline_args = parser.parse_known_args(argv)
+
+  # We use the save_main_session option because one or more DoFn's in this
+  # workflow rely on global context (e.g., a module imported at module level).
+  pipeline_options = PipelineOptions(pipeline_args)
+  pipeline_options.view_as(SetupOptions).save_main_session = True
+  pipeline_options.view_as(StandardOptions).streaming = True
+  p = beam.Pipeline(options=pipeline_options)
+
+  # Read from PubSub into a PCollection.
+  if known_args.input_subscription:
+lines = p | beam.io.ReadStringsFromPubSub(
+subscription=known_args.input_subscription)
+  else:
+lines = p | beam.io.ReadStringsFromPubSub(topic=known_args.input_topic)
+
+  # Count the occurrences of each word.
+  def count_ones(word_ones):
+(word, ones) = word_ones
+return (word, sum(ones))
+
+  counts = (lines
+| 'AddTimestampFn' >> 

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103219
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189163830
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
+
+It includes:
+  - PrintFn (DoFn) to inspect element, window, and timestamp.
+  - AddTimestampFn (DoFn) to modify timestamps.
+  - assert_that via check_gbk_format and equal_to_per_window (matchers).
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+
+import six
+
+import apache_beam as beam
+import apache_beam.transforms.window as window
+from apache_beam.examples.wordcount import WordExtractingDoFn
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to_per_window
+from apache_beam.transforms.core import ParDo
+
+
+class PrintFn(beam.DoFn):
+  """A DoFn that prints label, element, its window, and its timstamp. """
+  def __init__(self, label):
+self.label = label
+
+  def process(self, element, timestamp=beam.DoFn.TimestampParam,
+  window=beam.DoFn.WindowParam):
+# Log at INFO level each element processed. When executing this pipeline
+# using the Dataflow service, these log lines will appear in the Cloud
+# Logging UI.
+logging.info('[%s]: %s %s %s', self.label, element, window, timestamp)
+yield element
+
+
+class AddTimestampFn(beam.DoFn):
+  """A DoFn that attaches timestamps to its elements.
+
+  It takes a string of integers and it attaches to each of them
+  a timestamp of its same value."""
 
 Review comment:
   Adding an example here would help I think. For example:
   `"1 3 5" -> (1, timestamp=1), (3, timestamp=3), (5, timestamp=5)`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103219)
Time Spent: 4h 10m  (was: 4h)

> assert_that not working for streaming
> -
>
> Key: BEAM-3377
> URL: https://issues.apache.org/jira/browse/BEAM-3377
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: María GH
>Priority: Major
>  Labels: starter
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> assert_that does not work for AfterWatermark timers.
> Easy way to reproduce: modify test_gbk_execution [1] in this form:
>  
> {code:java}
>  def test_this(self):
> test_stream = (TestStream()
>.add_elements(['a', 'b', 'c'])
>.advance_watermark_to(20))
> def fnc(x):
>   print 'fired_elem:', x
>   return x
> options = PipelineOptions()
> options.view_as(StandardOptions).streaming = True
> p = TestPipeline(options=options)
> records = (p
>| test_stream
>| beam.WindowInto(
>FixedWindows(15),
>
> trigger=trigger.AfterWatermark(early=trigger.AfterCount(2)),
>accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
>| beam.Map(lambda x: 

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103220
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189164151
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
+
+It includes:
+  - PrintFn (DoFn) to inspect element, window, and timestamp.
+  - AddTimestampFn (DoFn) to modify timestamps.
+  - assert_that via check_gbk_format and equal_to_per_window (matchers).
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+
+import six
+
+import apache_beam as beam
+import apache_beam.transforms.window as window
+from apache_beam.examples.wordcount import WordExtractingDoFn
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to_per_window
+from apache_beam.transforms.core import ParDo
+
+
+class PrintFn(beam.DoFn):
+  """A DoFn that prints label, element, its window, and its timstamp. """
+  def __init__(self, label):
+self.label = label
+
+  def process(self, element, timestamp=beam.DoFn.TimestampParam,
+  window=beam.DoFn.WindowParam):
+# Log at INFO level each element processed. When executing this pipeline
+# using the Dataflow service, these log lines will appear in the Cloud
+# Logging UI.
+logging.info('[%s]: %s %s %s', self.label, element, window, timestamp)
+yield element
+
+
+class AddTimestampFn(beam.DoFn):
+  """A DoFn that attaches timestamps to its elements.
+
+  It takes a string of integers and it attaches to each of them
+  a timestamp of its same value."""
+  def process(self, element):
+for elem in element.split(' '):
+  logging.info('Adding timestamp to: %s', element)
+  yield beam.window.TimestampedValue(elem, int(elem))
+
+
+def run(argv=None):
+  """Build and run the pipeline."""
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+  '--output_topic', required=True,
+  help=('Output PubSub topic of the form '
+'"projects//topic/".'))
+  parser.add_argument(
+  '--use_assert_that', action='store_true',
+  help=('See that outputs are of the form '
+'"word:count".'))
+  group = parser.add_mutually_exclusive_group(required=True)
+  group.add_argument(
+  '--input_topic',
+  help=('Input PubSub topic of the form '
+'"projects//topics/".'))
+  group.add_argument(
+  '--input_subscription',
+  help=('Input PubSub subscription of the form '
+'"projects//subscriptions/."'))
+  known_args, pipeline_args = parser.parse_known_args(argv)
+
+  # We use the save_main_session option because one or more DoFn's in this
+  # workflow rely on global context (e.g., a module imported at module level).
+  pipeline_options = PipelineOptions(pipeline_args)
+  pipeline_options.view_as(SetupOptions).save_main_session = True
+  pipeline_options.view_as(StandardOptions).streaming = True
+  p = beam.Pipeline(options=pipeline_options)
+
+  # Read from PubSub into a PCollection.
+  if known_args.input_subscription:
+lines = p | beam.io.ReadStringsFromPubSub(
+subscription=known_args.input_subscription)
+  else:
+lines = p | beam.io.ReadStringsFromPubSub(topic=known_args.input_topic)
+
+  # Count the occurrences of each word.
+  def count_ones(word_ones):
+(word, ones) = word_ones
+return (word, sum(ones))
+
+  counts = (lines
+| 'AddTimestampFn' >> 

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103221
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189163882
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
+
+It includes:
+  - PrintFn (DoFn) to inspect element, window, and timestamp.
+  - AddTimestampFn (DoFn) to modify timestamps.
+  - assert_that via check_gbk_format and equal_to_per_window (matchers).
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+
+import six
+
+import apache_beam as beam
+import apache_beam.transforms.window as window
+from apache_beam.examples.wordcount import WordExtractingDoFn
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to_per_window
+from apache_beam.transforms.core import ParDo
+
+
+class PrintFn(beam.DoFn):
+  """A DoFn that prints label, element, its window, and its timstamp. """
+  def __init__(self, label):
+self.label = label
+
+  def process(self, element, timestamp=beam.DoFn.TimestampParam,
+  window=beam.DoFn.WindowParam):
+# Log at INFO level each element processed. When executing this pipeline
+# using the Dataflow service, these log lines will appear in the Cloud
+# Logging UI.
+logging.info('[%s]: %s %s %s', self.label, element, window, timestamp)
+yield element
+
+
+class AddTimestampFn(beam.DoFn):
+  """A DoFn that attaches timestamps to its elements.
+
+  It takes a string of integers and it attaches to each of them
+  a timestamp of its same value."""
+  def process(self, element):
+for elem in element.split(' '):
+  logging.info('Adding timestamp to: %s', element)
+  yield beam.window.TimestampedValue(elem, int(elem))
+
+
+def run(argv=None):
+  """Build and run the pipeline."""
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+  '--output_topic', required=True,
+  help=('Output PubSub topic of the form '
+'"projects//topic/".'))
+  parser.add_argument(
+  '--use_assert_that', action='store_true',
 
 Review comment:
   `--use_assert_that` option is not used, please remove it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103221)
Time Spent: 4h 10m  (was: 4h)

> assert_that not working for streaming
> -
>
> Key: BEAM-3377
> URL: https://issues.apache.org/jira/browse/BEAM-3377
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: María GH
>Priority: Major
>  Labels: starter
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> assert_that does not work for AfterWatermark timers.
> Easy way to reproduce: modify test_gbk_execution [1] in this form:
>  
> {code:java}
>  def test_this(self):
> test_stream = (TestStream()
>.add_elements(['a', 'b', 'c'])
>.advance_watermark_to(20))
> def fnc(x):
>   print 'fired_elem:', x
>   return x
> options = 

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103224
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189163677
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
+
+It includes:
+  - PrintFn (DoFn) to inspect element, window, and timestamp.
+  - AddTimestampFn (DoFn) to modify timestamps.
+  - assert_that via check_gbk_format and equal_to_per_window (matchers).
+"""
+
+from __future__ import absolute_import
+
+import argparse
+import logging
+import re
+
+import six
+
+import apache_beam as beam
+import apache_beam.transforms.window as window
+from apache_beam.examples.wordcount import WordExtractingDoFn
+from apache_beam.options.pipeline_options import PipelineOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.options.pipeline_options import StandardOptions
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to_per_window
+from apache_beam.transforms.core import ParDo
+
+
+class PrintFn(beam.DoFn):
+  """A DoFn that prints label, element, its window, and its timstamp. """
+  def __init__(self, label):
+self.label = label
+
+  def process(self, element, timestamp=beam.DoFn.TimestampParam,
+  window=beam.DoFn.WindowParam):
+# Log at INFO level each element processed. When executing this pipeline
 
 Review comment:
   Let's drop this comment for two reasons:
   1. First part of the comment is describing the same thing as the code in 
English, it is not adding additional context.
   2. Second part is dataflow runner specific, and the example does not even 
run on dataflow runner yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103224)
Time Spent: 4h 20m  (was: 4h 10m)

> assert_that not working for streaming
> -
>
> Key: BEAM-3377
> URL: https://issues.apache.org/jira/browse/BEAM-3377
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: María GH
>Priority: Major
>  Labels: starter
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> assert_that does not work for AfterWatermark timers.
> Easy way to reproduce: modify test_gbk_execution [1] in this form:
>  
> {code:java}
>  def test_this(self):
> test_stream = (TestStream()
>.add_elements(['a', 'b', 'c'])
>.advance_watermark_to(20))
> def fnc(x):
>   print 'fired_elem:', x
>   return x
> options = PipelineOptions()
> options.view_as(StandardOptions).streaming = True
> p = TestPipeline(options=options)
> records = (p
>| test_stream
>| beam.WindowInto(
>FixedWindows(15),
>
> trigger=trigger.AfterWatermark(early=trigger.AfterCount(2)),
>accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
>| beam.Map(lambda x: ('k', x))
>| beam.GroupByKey())
> assert_that(records, equal_to([
> ('k', ['a', 'b', 'c'])]))
> p.run()
> {code}
> This test will pass, but if the .advance_watermark_to(20) is removed, the 
> test will fail. 

[jira] [Work logged] (BEAM-3377) assert_that not working for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3377?focusedWorklogId=103222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103222
 ]

ASF GitHub Bot logged work on BEAM-3377:


Author: ASF GitHub Bot
Created on: 18/May/18 05:10
Start Date: 18/May/18 05:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5384: 
[BEAM-3377] Add validation for streaming wordcount with assert_that
URL: https://github.com/apache/beam/pull/5384#discussion_r189163531
 
 

 ##
 File path: sdks/python/apache_beam/examples/streaming_wordcount_debugging.py
 ##
 @@ -0,0 +1,179 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""An example to use assert_that to validate streaming wordcount.
 
 Review comment:
   Let's change this text to something that will resonate more to a user. It is 
a streaming wordcount example, that demonstrates the use of logging and 
assert_that in streaming mode. Rest are details, that are more suitable for the 
commit message, please remove them from the comment.
   
   Also please add a note, mentioning that this example only works in 
DirectRunner with a reference to the issue.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103222)
Time Spent: 4h 20m  (was: 4h 10m)

> assert_that not working for streaming
> -
>
> Key: BEAM-3377
> URL: https://issues.apache.org/jira/browse/BEAM-3377
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0
>Reporter: María GH
>Priority: Major
>  Labels: starter
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> assert_that does not work for AfterWatermark timers.
> Easy way to reproduce: modify test_gbk_execution [1] in this form:
>  
> {code:java}
>  def test_this(self):
> test_stream = (TestStream()
>.add_elements(['a', 'b', 'c'])
>.advance_watermark_to(20))
> def fnc(x):
>   print 'fired_elem:', x
>   return x
> options = PipelineOptions()
> options.view_as(StandardOptions).streaming = True
> p = TestPipeline(options=options)
> records = (p
>| test_stream
>| beam.WindowInto(
>FixedWindows(15),
>
> trigger=trigger.AfterWatermark(early=trigger.AfterCount(2)),
>accumulation_mode=trigger.AccumulationMode.ACCUMULATING)
>| beam.Map(lambda x: ('k', x))
>| beam.GroupByKey())
> assert_that(records, equal_to([
> ('k', ['a', 'b', 'c'])]))
> p.run()
> {code}
> This test will pass, but if the .advance_watermark_to(20) is removed, the 
> test will fail. However, both cases fire the same elements:
>   fired_elem: ('k', ['a', 'b', 'c'])
>   fired_elem: ('k', ['a', 'b', 'c'])
> In the passing case, they correspond to the sorted_actual inside the 
> assert_that. In the failing case:
>   sorted_actual: [('k', ['a', 'b', 'c']), ('k', ['a', 'b', 'c'])]
>   sorted_actual: []
> [1] 
> https://github.com/mariapython/incubator-beam/blob/direct-timers-show/sdks/python/apache_beam/testing/test_stream_test.py#L120



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4359) String encoding for a spanner mutation assumes that string length equals bytes length

[Sivanand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sivanand updated BEAM-4359:
---
Description: 
The bug is here:

[https://github.com/apache/beam/blob/3ba96003d31ce98a54c0c51c1c0a9cf7c06e2fa2/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L231-L235]
{code:java}
case STRING: {
String str = value.getString();
VarInt.encode(str.length(), bos);
bos.write(str.getBytes(StandardCharsets.UTF_8));
break;
}
{code}
 

The code assumes that the number of bytes used to represent a UTF-8 String 
equals the string length. This is not true because a UTF-8 character can be 
encoded using 1 - 4 bytes.

>From wikipedia: [https://en.wikipedia.org/wiki/UTF-8]
{quote}UTF-8 is a variable width character encoding capable of encoding all 
1,112,064 valid code points in Unicode using one to four 8-bit bytes
{quote}
Code to recreate the issue:
{code:java}
/*
Schema in spanner
CREATE TABLE test (
  id INT64,
  testString STRING(MAX),
  number INT64,
) PRIMARY KEY (id)
*/

import com.google.cloud.spanner.Mutation;
import com.google.common.collect.Lists;
import org.apache.beam.runners.direct.DirectRunner;
import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.junit.Rule;
import org.junit.Test;

import java.io.Serializable;
import java.util.List;

public class BeamSpannerTest implements Serializable {

@Rule
public transient TestPipeline pipeline = TestPipeline.create();

@Test
public void testSpanner() {
pipeline.getOptions().setRunner(DirectRunner.class);

List strdata = Lists.newArrayList("၃7");


pipeline.apply(
Create.of(strdata)
).apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext c) {
String value = c.element();
c.output(Mutation.newInsertOrUpdateBuilder("test")
.set("id").to(1)
.set("testString").to(value)
.set("number").to(10)
.build());
}
})
   ).apply("Write to Spanner", SpannerIO.write()
.withProjectId("my-project")
.withInstanceId("spanner-instance")
.withDatabaseId("test")
);

pipeline.run();
}
}
{code}
After running the code, the value in the column {{number}} will be {{7043}} and 
not {{10}} because the bytes from the previous column {{testString}} have 
spilled into the {{number}}

  was:
The bug is here:

https://github.com/apache/beam/blob/3ba96003d31ce98a54c0c51c1c0a9cf7c06e2fa2/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L231-L235
{code}
case STRING: {
String str = value.getString();
VarInt.encode(str.length(), bos);
bos.write(str.getBytes(StandardCharsets.UTF_8));
break;
}
{code} 

The code assumes that the number of bytes used to represent a UTF_8 String 
equals the string length. This is not true because a UTF_8 character can be 
encoded using 1 - 4 bytes.

>From wikipedia: https://en.wikipedia.org/wiki/UTF-8
{quote}UTF-8 is a variable width character encoding capable of encoding all 
1,112,064 valid code points in Unicode using one to four 8-bit bytes{quote}

Code to recreate the issue:
{code}
/*
Schema in spanner
CREATE TABLE test (
  id INT64,
  testString STRING(MAX),
  number INT64,
) PRIMARY KEY (id)
*/

import com.google.cloud.spanner.Mutation;
import com.google.common.collect.Lists;
import org.apache.beam.runners.direct.DirectRunner;
import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.junit.Rule;
import org.junit.Test;

import java.io.Serializable;
import java.util.List;

public class BeamSpannerTest implements Serializable {

@Rule
public transient TestPipeline pipeline = TestPipeline.create();

@Test
public void testSpanner() {
pipeline.getOptions().setRunner(DirectRunner.class);

List strdata = Lists.newArrayList("၃7");


pipeline.apply(
Create.of(strdata)
).apply(ParDo.of(new DoFn() {

[jira] [Created] (BEAM-4359) String encoding for a spanner mutation assumes that string length equals bytes length

[Sivanand (JIRA)]

Sivanand created BEAM-4359:
--

 Summary: String encoding for a spanner mutation assumes that 
string length equals bytes length
 Key: BEAM-4359
 URL: https://issues.apache.org/jira/browse/BEAM-4359
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Affects Versions: 2.4.0
Reporter: Sivanand
Assignee: Chamikara Jayalath


The bug is here:

https://github.com/apache/beam/blob/3ba96003d31ce98a54c0c51c1c0a9cf7c06e2fa2/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L231-L235
{code}
case STRING: {
String str = value.getString();
VarInt.encode(str.length(), bos);
bos.write(str.getBytes(StandardCharsets.UTF_8));
break;
}
{code} 

The code assumes that the number of bytes used to represent a UTF_8 String 
equals the string length. This is not true because a UTF_8 character can be 
encoded using 1 - 4 bytes.

>From wikipedia: https://en.wikipedia.org/wiki/UTF-8
{quote}UTF-8 is a variable width character encoding capable of encoding all 
1,112,064 valid code points in Unicode using one to four 8-bit bytes{quote}

Code to recreate the issue:
{code}
/*
Schema in spanner
CREATE TABLE test (
  id INT64,
  testString STRING(MAX),
  number INT64,
) PRIMARY KEY (id)
*/

import com.google.cloud.spanner.Mutation;
import com.google.common.collect.Lists;
import org.apache.beam.runners.direct.DirectRunner;
import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.transforms.Create;
import org.apache.beam.sdk.transforms.DoFn;
import org.apache.beam.sdk.transforms.ParDo;
import org.junit.Rule;
import org.junit.Test;

import java.io.Serializable;
import java.util.List;

public class BeamSpannerTest implements Serializable {

@Rule
public transient TestPipeline pipeline = TestPipeline.create();

@Test
public void testSpanner() {
pipeline.getOptions().setRunner(DirectRunner.class);

List strdata = Lists.newArrayList("၃7");


pipeline.apply(
Create.of(strdata)
).apply(ParDo.of(new DoFn() {
@ProcessElement
public void processElement(ProcessContext c) {
String value = c.element();
c.output(Mutation.newInsertOrUpdateBuilder("test")
.set("id").to(1)
.set("testString").to(value)
.set("number").to(10)
.build());
}
})
   ).apply("Write to Spanner", SpannerIO.write()
.withProjectId("my-project")
.withInstanceId("spanner-instance")
.withDatabaseId("test")
);

pipeline.run();
}
}
{code}

After running the code, the value in the column {{number}} will be {{7043}} and 
not {{10}} because the bytes from the previous column {{testString}} have 
spilled into the {{number}}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #288

[Apache Jenkins Server]

See 


Changes:

[matthias] Futurize metrics subpackage

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[github] Mention adding a reviewer in PR template. (#5397)

[kedin] Fix Maven build

--
[...truncated 10.33 MB...]
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 18, 2018 3:29:11 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 18, 2018 3:29:11 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0518032907-70e731c3/output/results/staging/
May 18, 2018 3:29:11 AM 

[jira] [Updated] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-4127:
---
Labels: portability  (was: )

> Flink portable runner translates streaming pipelines by proto
> -
>
> Key: BEAM-4127
> URL: https://issues.apache.org/jira/browse/BEAM-4127
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud updated BEAM-4357:
-
 Priority: Blocker  (was: Major)
Fix Version/s: 2.5.0

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Blocker
> Fix For: 2.5.0
>
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4297) Flink portable runner executable stage operator for streaming

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4297?focusedWorklogId=103209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103209
 ]

ASF GitHub Bot logged work on BEAM-4297:


Author: ASF GitHub Bot
Created on: 18/May/18 02:37
Start Date: 18/May/18 02:37
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #5407: [BEAM-4297] 
Streaming executable stage translation and operator for portable Flink runner.
URL: https://github.com/apache/beam/pull/5407
 
 
   Executable stage translation for streaming mode based on the generic Flink 
streaming operator. Stage execution and tests adopted from batch translation.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103209)
Time Spent: 10m
Remaining Estimate: 0h

> Flink portable runner executable stage operator for streaming
> -
>
> Key: BEAM-4297
> URL: https://issues.apache.org/jira/browse/BEAM-4297
> Project: Beam
>  Issue Type: Task
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #454

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go integration tests

--
[...truncated 19.33 MB...]
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_19_24_19-2138791765885925817
May 18, 2018 2:24:20 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-17_19_24_19-2138791765885925817 with 0 
expected assertions.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:19.783Z: Autoscaling is enabled for job 
2018-05-17_19_24_19-2138791765885925817. The number of workers will be between 
1 and 1000.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:19.826Z: Autoscaling was automatically enabled for 
job 2018-05-17_19_24_19-2138791765885925817.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:22.447Z: Checking required Cloud APIs are enabled.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:22.638Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:26.979Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:27.471Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:27.721Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:27.768Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.041Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.090Z: Elided trivial flatten 
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.141Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.180Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.235Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.283Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.326Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 2:24:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:24:28.381Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #455

[Apache Jenkins Server]

See 


Changes:

[rober] Update generated protos in Go SDK

--
[...truncated 18.50 MB...]
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:41.023Z: Autoscaling was automatically enabled for 
job 2018-05-17_19_19_40-5003561345419929527.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:43.697Z: Checking required Cloud APIs are enabled.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:44.023Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:47.530Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:47.989Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.257Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.304Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.631Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.670Z: Elided trivial flatten 
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.722Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.768Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.816Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.859Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.902Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.930Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
May 18, 2018 2:19:52 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:19:48.974Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #453

[Apache Jenkins Server]

See 


Changes:

[kedin] Fix Maven build

--
[...truncated 18.88 MB...]
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_18_59_54-15438375337965929270
May 18, 2018 1:59:55 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-17_18_59_54-15438375337965929270 with 0 
expected assertions.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T01:59:54.281Z: Autoscaling is enabled for job 
2018-05-17_18_59_54-15438375337965929270. The number of workers will be between 
1 and 1000.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T01:59:54.317Z: Autoscaling was automatically enabled for 
job 2018-05-17_18_59_54-15438375337965929270.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T01:59:57.047Z: Checking required Cloud APIs are enabled.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T01:59:57.286Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.011Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.392Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.631Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.679Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.938Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:05.984Z: Elided trivial flatten 
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.025Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.064Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.109Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.154Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.202Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 2:00:16 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T02:00:06.247Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into 

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103201
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 01:39
Start Date: 18/May/18 01:39
Worklog Time Spent: 10m 
  Work Description: jasonkuster closed pull request #5405: [BEAM-4276] 
Update generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go 
b/sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
index 80487530a00..f697df8b123 100644
--- a/sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
+++ b/sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
@@ -152,9 +152,7 @@ func (m *Entry) String() string{ return 
proto.CompactTextString(m) }
 func (*Entry) ProtoMessage()   {}
 func (*Entry) Descriptor() ([]byte, []int) { return fileDescriptor0, []int{3} }
 
-type isEntry_Msg interface {
-   isEntry_Msg()
-}
+type isEntry_Msg interface{ isEntry_Msg() }
 
 type Entry_InstReq struct {
InstReq *org_apache_beam_model_fn_execution_v1.InstructionRequest 
`protobuf:"bytes,1000,opt,name=inst_req,json=instReq,oneof"`
diff --git a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go 
b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
index 8e5b5bab25e..e3147f5e031 100644
--- a/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
+++ b/sdks/go/pkg/beam/model/fnexecution_v1/beam_fn_api.pb.go
@@ -48,8 +48,8 @@ import fmt "fmt"
 import math "math"
 import org_apache_beam_model_pipeline_v1 
"github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
 import org_apache_beam_model_pipeline_v11 
"github.com/apache/beam/sdks/go/pkg/beam/model/pipeline_v1"
-import google_protobuf1 "github.com/golang/protobuf/ptypes/timestamp"
-import google_protobuf2 "github.com/golang/protobuf/ptypes/wrappers"
+import google_protobuf2 "github.com/golang/protobuf/ptypes/timestamp"
+import google_protobuf3 "github.com/golang/protobuf/ptypes/wrappers"
 
 import (
context "golang.org/x/net/context"
@@ -218,9 +218,7 @@ func (m *InstructionRequest) String() string{ 
return proto.CompactTe
 func (*InstructionRequest) ProtoMessage()   {}
 func (*InstructionRequest) Descriptor() ([]byte, []int) { return 
fileDescriptor0, []int{2} }
 
-type isInstructionRequest_Request interface {
-   isInstructionRequest_Request()
-}
+type isInstructionRequest_Request interface{ isInstructionRequest_Request() }
 
 type InstructionRequest_Register struct {
Register *RegisterRequest 
`protobuf:"bytes,1000,opt,name=register,oneof"`
@@ -421,9 +419,7 @@ func (m *InstructionResponse) String() string{ 
return proto.CompactT
 func (*InstructionResponse) ProtoMessage()   {}
 func (*InstructionResponse) Descriptor() ([]byte, []int) { return 
fileDescriptor0, []int{3} }
 
-type isInstructionResponse_Response interface {
-   isInstructionResponse_Response()
-}
+type isInstructionResponse_Response interface{ 
isInstructionResponse_Response() }
 
 type InstructionResponse_Register struct {
Register *RegisterResponse 
`protobuf:"bytes,1000,opt,name=register,oneof"`
@@ -761,7 +757,7 @@ type BundleSplit_Application struct {
// represented by invoking this Application and its downstream 
applications.
// The sum of fraction_of_work between all primary_roots and 
residual_roots
// must add up to approximately 1.0.
-   FractionOfWork *google_protobuf2.DoubleValue 
`protobuf:"bytes,5,opt,name=fraction_of_work,json=fractionOfWork" 
json:"fraction_of_work,omitempty"`
+   FractionOfWork *google_protobuf3.DoubleValue 
`protobuf:"bytes,5,opt,name=fraction_of_work,json=fractionOfWork" 
json:"fraction_of_work,omitempty"`
 }
 
 func (m *BundleSplit_Application) Reset(){ *m = 
BundleSplit_Application{} }
@@ -797,7 +793,7 @@ func (m *BundleSplit_Application) GetOutputWatermarks() 
map[string]int64 {
return nil
 }
 
-func (m *BundleSplit_Application) GetFractionOfWork() 
*google_protobuf2.DoubleValue {
+func (m *BundleSplit_Application) GetFractionOfWork() 
*google_protobuf3.DoubleValue {
if m != nil {
return m.FractionOfWork
}
@@ -1081,9 +1077,7 @@ func (m *Metrics_User) String() string{ 
return proto.CompactTextStri
 func (*Metrics_User) ProtoMessage()   {}
 func (*Metrics_User) Descriptor() ([]byte, []int) { return fileDescriptor0, 
[]int{11, 1} }
 
-type isMetrics_User_Data interface 

[beam] 01/01: Merge pull request #5405: [BEAM-4276] Update generated Go SDK protobuf files.

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7b0cd4ecd989da1402b7dcacf0dd83e49591a558
Merge: c1927cd 0071b5f
Author: jasonkuster 
AuthorDate: Thu May 17 18:39:12 2018 -0700

Merge pull request #5405: [BEAM-4276] Update generated Go SDK protobuf 
files.

[BEAM-4276] Update generated Go SDK protobuf files.

 .../core/runtime/harness/session/session.pb.go |4 +-
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go|   44 +-
 .../model/fnexecution_v1/beam_provision_api.pb.go  |6 +-
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |4 +-
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |   10 +-
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 1041 ++--
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |4 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  174 +++-
 8 files changed, 877 insertions(+), 410 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (c1927cd -> 7b0cd4e)

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c1927cd  Merge pull request #5326: [BEAM-3827] Add Go integration tests
 add 0071b5f  Update generated protos in Go SDK
 new 7b0cd4e  Merge pull request #5405: [BEAM-4276] Update generated Go SDK 
protobuf files.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/runtime/harness/session/session.pb.go |4 +-
 .../beam/model/fnexecution_v1/beam_fn_api.pb.go|   44 +-
 .../model/fnexecution_v1/beam_provision_api.pb.go  |6 +-
 .../model/jobmanagement_v1/beam_artifact_api.pb.go |4 +-
 .../beam/model/jobmanagement_v1/beam_job_api.pb.go |   10 +-
 .../beam/model/pipeline_v1/beam_runner_api.pb.go   | 1041 ++--
 sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go |4 +-
 .../model/pipeline_v1/standard_window_fns.pb.go|  174 +++-
 8 files changed, 877 insertions(+), 410 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103200
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 01:38
Start Date: 18/May/18 01:38
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #5405: [BEAM-4276] 
Update generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405#issuecomment-390067961
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103200)
Time Spent: 1h 50m  (was: 1h 40m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (4b4381a -> c1927cd)

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4b4381a  Merge pull request #5404 from akedin/fix-maven-build-1
 add 091da44  Add Go integration tests
 new c1927cd  Merge pull request #5326: [BEAM-3827] Add Go integration tests

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../windowed_wordcount/windowed_wordcount.go   |  2 +-
 .../filesystem/{local/local.go => memfs/memory.go} | 76 ++---
 .../go/pkg/beam/io/filesystem/memfs/memory_test.go | 74 
 .../{log/standard.go => io/filesystem/util.go} | 34 +---
 sdks/go/pkg/beam/options/jobopts/options.go|  9 +-
 sdks/go/pkg/beam/runners/dataflow/dataflow.go  | 23 ++---
 .../beam/runners/universal/runnerlib/compile.go|  8 +-
 .../beam/runners/universal/runnerlib/execute.go|  9 +-
 sdks/go/pkg/beam/testing/passert/hash.go   | 69 +++
 sdks/go/pkg/beam/util.go   |  6 ++
 sdks/go/pkg/beam/x/hooks/perf/perf.go  |  2 +-
 sdks/go/test/integration/driver.go | 98 ++
 .../integration}/wordcount/wordcount.go| 48 ---
 .../integration/wordcount/wordcount_test.go}   | 71 +---
 14 files changed, 436 insertions(+), 93 deletions(-)
 copy sdks/go/pkg/beam/io/filesystem/{local/local.go => memfs/memory.go} (50%)
 create mode 100644 sdks/go/pkg/beam/io/filesystem/memfs/memory_test.go
 copy sdks/go/pkg/beam/{log/standard.go => io/filesystem/util.go} (59%)
 create mode 100644 sdks/go/pkg/beam/testing/passert/hash.go
 create mode 100644 sdks/go/test/integration/driver.go
 rename sdks/go/{examples/windowed_wordcount => 
test/integration}/wordcount/wordcount.go (73%)
 copy sdks/go/{pkg/beam/transforms/filter/distinct_test.go => 
test/integration/wordcount/wordcount_test.go} (51%)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-3827) Add Go SDK integration tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3827?focusedWorklogId=103199=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103199
 ]

ASF GitHub Bot logged work on BEAM-3827:


Author: ASF GitHub Bot
Created on: 18/May/18 01:37
Start Date: 18/May/18 01:37
Worklog Time Spent: 10m 
  Work Description: jasonkuster closed pull request #5326: [BEAM-3827] Add 
Go integration tests
URL: https://github.com/apache/beam/pull/5326
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/examples/windowed_wordcount/windowed_wordcount.go 
b/sdks/go/examples/windowed_wordcount/windowed_wordcount.go
index 753a0a12f57..f6748ab2fa9 100644
--- a/sdks/go/examples/windowed_wordcount/windowed_wordcount.go
+++ b/sdks/go/examples/windowed_wordcount/windowed_wordcount.go
@@ -42,13 +42,13 @@ import (
"reflect"
"time"
 
-   "github.com/apache/beam/sdks/go/examples/windowed_wordcount/wordcount"
"github.com/apache/beam/sdks/go/pkg/beam"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/mtime"
"github.com/apache/beam/sdks/go/pkg/beam/core/graph/window"
"github.com/apache/beam/sdks/go/pkg/beam/io/textio"
"github.com/apache/beam/sdks/go/pkg/beam/log"
"github.com/apache/beam/sdks/go/pkg/beam/x/beamx"
+   "github.com/apache/beam/sdks/go/test/integration/wordcount"
 )
 
 var (
diff --git a/sdks/go/pkg/beam/io/filesystem/memfs/memory.go 
b/sdks/go/pkg/beam/io/filesystem/memfs/memory.go
new file mode 100644
index 000..b9d19ccbbde
--- /dev/null
+++ b/sdks/go/pkg/beam/io/filesystem/memfs/memory.go
@@ -0,0 +1,108 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Package memfs contains a in-memory Beam filesystem. Useful for testing.
+package memfs
+
+import (
+   "bytes"
+   "context"
+   "io"
+   "io/ioutil"
+   "os"
+   "sort"
+   "strings"
+   "sync"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/io/filesystem"
+)
+
+func init() {
+   filesystem.Register("memfs", New)
+}
+
+var instance = {m: make(map[string][]byte)}
+
+type fs struct {
+   m  map[string][]byte
+   mu sync.Mutex
+}
+
+// New returns the global memory filesystem.
+func New(ctx context.Context) filesystem.Interface {
+   return instance
+}
+
+func (f *fs) Close() error {
+   return nil
+}
+
+func (f *fs) List(ctx context.Context, glob string) ([]string, error) {
+   f.mu.Lock()
+   defer f.mu.Unlock()
+
+   var ret []string
+   for k := range f.m {
+   ret = append(ret, k)
+   }
+   sort.Strings(ret)
+   return ret, nil
+}
+
+func (f *fs) OpenRead(ctx context.Context, filename string) (io.ReadCloser, 
error) {
+   f.mu.Lock()
+   defer f.mu.Unlock()
+
+   if v, ok := f.m[normalize(filename)]; ok {
+   return ioutil.NopCloser(bytes.NewReader(v)), nil
+   }
+   return nil, os.ErrNotExist
+}
+
+func (f *fs) OpenWrite(ctx context.Context, filename string) (io.WriteCloser, 
error) {
+   return {key: filename}, nil
+}
+
+// Write stores the given key and value in the global store.
+func Write(key string, value []byte) {
+   instance.mu.Lock()
+   defer instance.mu.Unlock()
+
+   cp := make([]byte, len(value))
+   copy(cp, value)
+
+   instance.m[normalize(key)] = cp
+}
+
+func normalize(key string) string {
+   if strings.HasPrefix(key, "memfs://") {
+   return key
+   }
+   return "memfs://" + key
+}
+
+type commitWriter struct {
+   key string
+   buf bytes.Buffer
+}
+
+func (w *commitWriter) Write(p []byte) (n int, err error) {
+   return w.buf.Write(p)
+}
+
+func (w *commitWriter) Close() error {
+   Write(w.key, w.buf.Bytes())
+   return nil
+}
diff --git a/sdks/go/pkg/beam/io/filesystem/memfs/memory_test.go 
b/sdks/go/pkg/beam/io/filesystem/memfs/memory_test.go
new file mode 100644
index 

[beam] 01/01: Merge pull request #5326: [BEAM-3827] Add Go integration tests

[jaku]

This is an automated email from the ASF dual-hosted git repository.

jaku pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c1927cd339c57125e29a651e614fb5105abf6d33
Merge: 4b4381a 091da44
Author: jasonkuster 
AuthorDate: Thu May 17 18:37:47 2018 -0700

Merge pull request #5326: [BEAM-3827] Add Go integration tests

[BEAM-3827] Add Go integration tests

 .../windowed_wordcount/windowed_wordcount.go   |   2 +-
 sdks/go/pkg/beam/io/filesystem/memfs/memory.go | 108 +
 .../go/pkg/beam/io/filesystem/memfs/memory_test.go |  74 ++
 sdks/go/pkg/beam/io/filesystem/util.go |  45 +
 sdks/go/pkg/beam/options/jobopts/options.go|   9 +-
 sdks/go/pkg/beam/runners/dataflow/dataflow.go  |  23 ++---
 .../beam/runners/universal/runnerlib/compile.go|   8 +-
 .../beam/runners/universal/runnerlib/execute.go|   9 +-
 sdks/go/pkg/beam/testing/passert/hash.go   |  69 +
 sdks/go/pkg/beam/util.go   |   6 ++
 sdks/go/pkg/beam/x/hooks/perf/perf.go  |   2 +-
 sdks/go/test/integration/driver.go |  98 +++
 .../integration}/wordcount/wordcount.go|  48 ++---
 .../test/integration/wordcount/wordcount_test.go   |  85 
 14 files changed, 546 insertions(+), 40 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #29

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #456

[Apache Jenkins Server]

See 

[beam] branch master updated (0c5ebc4 -> 4b4381a)

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0c5ebc4  Merge pull request #5335 from Fematich/metrics
 add 853190f  Fix Maven build
 new 4b4381a  Merge pull request #5404 from akedin/fix-maven-build-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/google-cloud-platform/pom.xml | 24 ++--
 1 file changed, 18 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
t...@apache.org.

[beam] 01/01: Merge pull request #5404 from akedin/fix-maven-build-1

[thw]

This is an automated email from the ASF dual-hosted git repository.

thw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4b4381aba5075361c052fd459d785a5817a9cd57
Merge: 0c5ebc4 853190f
Author: Thomas Weise 
AuthorDate: Thu May 17 18:16:50 2018 -0700

Merge pull request #5404 from akedin/fix-maven-build-1

Fix Maven build

 sdks/java/io/google-cloud-platform/pom.xml | 24 ++--
 1 file changed, 18 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
t...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #452

[Apache Jenkins Server]

See 


Changes:

[matthias] Futurize metrics subpackage

--
[...truncated 18.92 MB...]
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_17_57_55-6581189880367005340
May 18, 2018 12:57:56 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-17_17_57_55-6581189880367005340 with 0 
expected assertions.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:57:55.131Z: Autoscaling is enabled for job 
2018-05-17_17_57_55-6581189880367005340. The number of workers will be between 
1 and 1000.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:57:55.155Z: Autoscaling was automatically enabled for 
job 2018-05-17_17_57_55-6581189880367005340.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:57:57.704Z: Checking required Cloud APIs are enabled.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:57:57.862Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:01.516Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:01.951Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.104Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.132Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.272Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.295Z: Elided trivial flatten 
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.323Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.345Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.370Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.394Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.423Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 12:58:07 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:58:02.450Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #28

[Apache Jenkins Server]

See 


Changes:

[matthias] Futurize metrics subpackage

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[ankurgoenka] Making a few methods static

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 146.40 KB...]
  File 
"
 line 604, in to_runner_api
components=context.to_runner_api())
  File 
"
 line 134, in to_runner_api
getattr(self, name).populate_map(getattr(context_proto, name))
  File 
"
 line 69, in populate_map
proto_map[id].CopyFrom(proto)
  File 
"
 line 276, in signalhandler
raise TimedOutException()
TimedOutException: 'test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)'

==
ERROR: test_multi_valued_singleton_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"
 line 812, in run
test(orig)
  File 
"
 line 45, in __call__
return self.run(*arg, **kwarg)
  File 
"
 line 133, in run
self.runTest(result)
  File 
"
 line 151, in runTest
test(result)
  File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
return self.run(*args, **kwds)
  File "/usr/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
  File 
"
 line 157, in test_multi_valued_singleton_side_input
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run(test_runner_api)
  File 
"
 line 389, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 402, in run
return self.runner.run_pipeline(self)
  File 
"
 line 48, in run_pipeline
self.result = super(TestDataflowRunner, self).run_pipeline(pipeline)
  File 
"
 line 362, in run_pipeline
self.dataflow_client.create_job(self.job), self)
  File 
"
 line 180, in wrapper
return fun(*args, **kwargs)
  File 
"
 line 485, in create_job
self.create_job_description(job)
  File 
"
 line 511, in create_job_description
StringIO(job.proto_pipeline.SerializeToString()))
  File 
"
 line 467, in stage_file
response = self._storage_client.objects.Insert(request, upload=upload)
  File 

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103196
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 00:57
Start Date: 18/May/18 00:57
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5405: [BEAM-4276] Update 
generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405#issuecomment-390060684
 
 
   R: @jasonkuster Please sir, may I have a merge?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103196)
Time Spent: 1h 40m  (was: 1.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103193=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103193
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 00:52
Start Date: 18/May/18 00:52
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on issue #5405: [BEAM-4276] Update 
generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405#issuecomment-390059746
 
 
   The deltas here are a byproduct of using a new version of the go protoc 
plugin. This change is good.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103193)
Time Spent: 1.5h  (was: 1h 20m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #451

[Apache Jenkins Server]

See 


--
[...truncated 18.64 MB...]
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 18, 2018 12:44:41 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_17_44_40-5674336627402485727
May 18, 2018 12:44:41 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-17_17_44_40-5674336627402485727 with 0 
expected assertions.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:40.437Z: Autoscaling is enabled for job 
2018-05-17_17_44_40-5674336627402485727. The number of workers will be between 
1 and 1000.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:40.453Z: Autoscaling was automatically enabled for 
job 2018-05-17_17_44_40-5674336627402485727.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:42.886Z: Checking required Cloud APIs are enabled.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:43.050Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:45.918Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.256Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.396Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.421Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.580Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.601Z: Elided trivial flatten 
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.627Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.654Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.677Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.699Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.723Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 12:44:51 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:44:46.749Z: Fusing consumer 

[jira] [Work logged] (BEAM-4271) Executable stages allow side input coders to be set and/or queried

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4271?focusedWorklogId=103190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103190
 ]

ASF GitHub Bot logged work on BEAM-4271:


Author: ASF GitHub Bot
Created on: 18/May/18 00:36
Start Date: 18/May/18 00:36
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5374: [BEAM-4271] Support 
side inputs for ExecutableStage and provide runner side utilities for handling 
multimap side inputs.
URL: https://github.com/apache/beam/pull/5374#issuecomment-389329329
 
 
   R: @tgroh @bsidhom @tweise 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103190)
Time Spent: 10m
Remaining Estimate: 0h

> Executable stages allow side input coders to be set and/or queried
> --
>
> Key: BEAM-4271
> URL: https://issues.apache.org/jira/browse/BEAM-4271
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> ProcessBundleDescriptors may contain side input references from inner 
> PTransforms. These side inputs do not have explicit coders; instead, SDK 
> harnesses use the PCollection coders by default.
> Using the default PCollection coder as specified at pipeline construction is 
> in general not the correct thing to do. When PCollection elements are 
> materialized, any coders unknown to a runner a length-prefixed. This means 
> that materialized PCollections do not use their original element coders. Side 
> inputs are delivered to SDKs via MultimapSideInput StateRequests. The 
> responses to these requests are expected to contain all of the values for a 
> given key (and window), coded with the PCollection KV.value coder, 
> concatenated. However, at the time of serving these requests on the runner 
> side, we do not have enough information to reconstruct the original value 
> coders.
> There are different ways to address this issue. For example:
>  * Modify the associated PCollection coder to match the coder that the runner 
> uses to materialize elements. This means that anywhere a given PCollection is 
> used within a given bundle, it will use the runner-safe coder. This may 
> introduce inefficiencies but should be "correct".
>  * Annotate side inputs with explicit coders. This guarantees that the key 
> and value coders used by the runner match the coders used by SDKs. 
> Furthermore, it allows the _runners_ to specify coders. This involves changes 
> to the proto models and all SDKs.
>  * Annotate side input state requests with both key and value coders. This 
> inverts the expected responsibility and has the SDK determine runner coders. 
> Additionally, because runners do not understand all SDK types, additional 
> coder substitution will need to be done at request handling time to make sure 
> that the requested coder can be instantiated and will remain consistent with 
> the SDK coder. This requires only small changes to SDKs because they may opt 
> to use their default PCollection coders.
> All of the these approaches have their own downsides. Explicit side input 
> coders is probably the right thing to do long-term, but the simplest change 
> for now is to modify PCollection coders to match exactly how they're 
> materialized.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103188=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103188
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 00:34
Start Date: 18/May/18 00:34
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5402: [BEAM-4276] Update 
generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5402#issuecomment-390056562
 
 
   Closing since it's conflating updating the protos, and updating the version 
of protoc-go-gen used.
   Moving work to https://github.com/apache/beam/pull/5405


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103188)
Time Spent: 1h 10m  (was: 1h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #450

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103187
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 00:24
Start Date: 18/May/18 00:24
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5405: [BEAM-4276] Update 
generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405#issuecomment-390055090
 
 
   R: @herohde @wcn3 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103187)
Time Spent: 1h  (was: 50m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=103185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103185
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 18/May/18 00:21
Start Date: 18/May/18 00:21
Worklog Time Spent: 10m 
  Work Description: lostluck opened a new pull request #5405: [BEAM-4276] 
Update generated Go SDK protobuf files.
URL: https://github.com/apache/beam/pull/5405
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103185)
Time Spent: 50m  (was: 40m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #455

[Apache Jenkins Server]

See 


Changes:

[matthias] Futurize metrics subpackage

--
[...truncated 517.20 KB...]
Successfully started process 'Gradle Test Executor 253'
Starting process 'Gradle Test Executor 254'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 254'
Successfully started process 'Gradle Test Executor 254'
Starting process 'Gradle Test Executor 255'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 255'
Successfully started process 'Gradle Test Executor 255'
Starting process 'Gradle Test Executor 256'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 256'
Successfully started process 'Gradle Test Executor 256'
Starting process 'Gradle Test Executor 257'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 257'
Successfully started process 'Gradle Test Executor 257'
Starting process 'Gradle Test Executor 258'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 
-Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager
 -Dorg.gradle.native=false -Dspark.ui.enabled=false 
-Dspark.ui.showConsoleProgress=false -Dfile.encoding=UTF-8 -Duser.country=US 
-Duser.language=en -Duser.variant -ea -cp 
/home/jenkins/.gradle/caches/4.7/workerMain/gradle-worker.jar 
worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test 
Executor 258'
Successfully started process 'Gradle Test Executor 258'
Starting process 'Gradle Test Executor 259'. Working directory: 

 Command: /usr/local/asfpackages/java/jdk1.8.0_152/bin/java 
-Dbeam.spark.test.reuseSparkContext=true 
-DbeamTestPipelineOptions=["--runner=TestSparkRunner","--streaming=false","--enableSparkMetricSinks=false"]
 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #182

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 375.19 KB...]
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #182

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 577.37 KB...]
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:924)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.AssignWindowsParDoFnFactory$AssignWindowsParDoFn.processElement(AssignWindowsParDoFnFactory.java:118)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 

[jira] [Work logged] (BEAM-4001) Futurize and fix python 2 compatibility for metrics subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4001?focusedWorklogId=103184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103184
 ]

ASF GitHub Bot logged work on BEAM-4001:


Author: ASF GitHub Bot
Created on: 18/May/18 00:16
Start Date: 18/May/18 00:16
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5335: [BEAM-4001] Futurize 
metrics subpackage
URL: https://github.com/apache/beam/pull/5335
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/metrics/__init__.py 
b/sdks/python/apache_beam/metrics/__init__.py
index 8ce7bbb173f..e74168f09e4 100644
--- a/sdks/python/apache_beam/metrics/__init__.py
+++ b/sdks/python/apache_beam/metrics/__init__.py
@@ -14,5 +14,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+from __future__ import absolute_import
+
 from apache_beam.metrics.metric import Metrics
 from apache_beam.metrics.metric import MetricsFilter
diff --git a/sdks/python/apache_beam/metrics/cells.py 
b/sdks/python/apache_beam/metrics/cells.py
index 302d79ab512..8f93d7fffe5 100644
--- a/sdks/python/apache_beam/metrics/cells.py
+++ b/sdks/python/apache_beam/metrics/cells.py
@@ -23,10 +23,13 @@
 Cells depend on a 'dirty-bit' in the CellCommitState class that tracks whether
 a cell's updates have been committed.
 """
+
+from __future__ import absolute_import
 from __future__ import division
 
 import threading
 import time
+from builtins import object
 
 from google.protobuf import timestamp_pb2
 
@@ -245,6 +248,9 @@ def __eq__(self, other):
 else:
   return False
 
+  def __hash__(self):
+return hash(self.data)
+
   def __ne__(self, other):
 return not self.__eq__(other)
 
@@ -292,6 +298,9 @@ def __eq__(self, other):
 else:
   return False
 
+  def __hash__(self):
+return hash(self.data)
+
   def __ne__(self, other):
 return not self.__eq__(other)
 
@@ -326,6 +335,9 @@ def __init__(self, value, timestamp=None):
   def __eq__(self, other):
 return self.value == other.value and self.timestamp == other.timestamp
 
+  def __hash__(self):
+return hash((self.value, self.timestamp))
+
   def __ne__(self, other):
 return not self.__eq__(other)
 
@@ -386,6 +398,9 @@ def __eq__(self, other):
 self.min == other.min and
 self.max == other.max)
 
+  def __hash__(self):
+return hash((self.sum, self.count, self.min, self.max))
+
   def __ne__(self, other):
 return not self.__eq__(other)
 
diff --git a/sdks/python/apache_beam/metrics/cells_test.py 
b/sdks/python/apache_beam/metrics/cells_test.py
index 14e7e537ea1..64b9df95bbf 100644
--- a/sdks/python/apache_beam/metrics/cells_test.py
+++ b/sdks/python/apache_beam/metrics/cells_test.py
@@ -15,8 +15,11 @@
 # limitations under the License.
 #
 
+from __future__ import absolute_import
+
 import threading
 import unittest
+from builtins import range
 
 from apache_beam.metrics.cells import CellCommitState
 from apache_beam.metrics.cells import CounterCell
diff --git a/sdks/python/apache_beam/metrics/execution.py 
b/sdks/python/apache_beam/metrics/execution.py
index cb0f07141a1..157fa0aaf5e 100644
--- a/sdks/python/apache_beam/metrics/execution.py
+++ b/sdks/python/apache_beam/metrics/execution.py
@@ -29,7 +29,11 @@
 - MetricsContainer - Holds the metrics of a single step and a single
 unit-of-commit (bundle).
 """
+
+from __future__ import absolute_import
+
 import threading
+from builtins import object
 from collections import defaultdict
 
 from apache_beam.metrics.cells import CounterCell
@@ -59,6 +63,9 @@ def __eq__(self, other):
 return (self.step == other.step and
 self.metric == other.metric)
 
+  def __hash__(self):
+return hash((self.step, self.metric))
+
   def __repr__(self):
 return 'MetricKey(step={}, metric={})'.format(
 self.step, self.metric)
@@ -98,6 +105,9 @@ def __eq__(self, other):
 self.committed == other.committed and
 self.attempted == other.attempted)
 
+  def __hash__(self):
+return hash((self.key, self.committed, self.attempted))
+
   def __repr__(self):
 return 'MetricResult(key={}, committed={}, attempted={})'.format(
 self.key, str(self.committed), str(self.attempted))
diff --git a/sdks/python/apache_beam/metrics/execution_test.py 
b/sdks/python/apache_beam/metrics/execution_test.py
index 37d24f3407b..fbf5492f05d 100644
--- a/sdks/python/apache_beam/metrics/execution_test.py
+++ b/sdks/python/apache_beam/metrics/execution_test.py
@@ -15,7 +15,10 @@
 # limitations under the License.
 #
 
+from __future__ import absolute_import
+
 import 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #183

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 206.91 KB...]
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=146.148.62.9:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #275

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 103.17 KB...]
:beam-sdks-java-io-hadoop-input-format:classes (Thread[Task worker for ':' 
Thread 7,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:classes UP-TO-DATE
Skipping task ':beam-sdks-java-io-hadoop-input-format:classes' as it has no 
actions.
:beam-sdks-java-io-hadoop-input-format:classes (Thread[Task worker for ':' 
Thread 7,5,main]) completed. Took 0.0 secs.

> Task :beam-sdks-java-io-google-cloud-platform:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:compileTestJava' is 
a2606a1c31bda2a96202e5a5d7728f8c
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 3,5,main]) completed. Took 0.067 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 2,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 2,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 2,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
abb6f3d76fd1f2c96464a7549d32249a
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 2,5,main]) completed. Took 0.03 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
6b85bfa012acd53eea2d6ef376522d89
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.047 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
7696681a375e2cb05eec13aa169bd053
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.03 secs.
:beam-sdks-java-io-hadoop-input-format:compileTestJava (Thread[Task worker for 
':' Thread 9,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-hadoop-input-format:compileTestJava' is 
aadebcc9dc30c7bc0ea543f5dd23d295
Skipping task ':beam-sdks-java-io-hadoop-input-format:compileTestJava' as it is 
up-to-date.
:beam-sdks-java-io-hadoop-input-format:compileTestJava (Thread[Task worker for 
':' Thread 9,5,main]) completed. Took 0.428 secs.
:beam-sdks-java-io-hadoop-input-format:testClasses (Thread[Task worker for ':' 
Thread 9,5,main]) started.

> Task :beam-sdks-java-io-hadoop-input-format:testClasses 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103179=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103179
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189132874
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103177=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103177
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189132979
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103175
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189133721
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #449

[Apache Jenkins Server]

See 


Changes:

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

--
[...truncated 18.71 MB...]
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_17_03_08-16752505420234330816
May 18, 2018 12:03:09 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-17_17_03_08-16752505420234330816 with 0 
expected assertions.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:08.438Z: Autoscaling is enabled for job 
2018-05-17_17_03_08-16752505420234330816. The number of workers will be between 
1 and 1000.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:08.481Z: Autoscaling was automatically enabled for 
job 2018-05-17_17_03_08-16752505420234330816.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:11.005Z: Checking required Cloud APIs are enabled.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:11.190Z: Checking permissions granted to controller 
Service Account.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:14.954Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.329Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.569Z: Expanding GroupByKey operations into 
optimizable parts.
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.617Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.874Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.917Z: Elided trivial flatten 
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.955Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:15.993Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:16.033Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:16.079Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:16.138Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 18, 2018 12:03:19 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-18T00:03:16.175Z: 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #5

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [BEAM-3733] [SQL] Use saffron.properties for charset

[Pablo] Add future dependency to the Dockerfile

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[axelmagn] Create jobsubmission package for job management.

[axelmagn] Add Job Invocation Interfaces

[axelmagn] Add in-memory implementation of JobService

[Pablo] Remove stale TODO comments for adding archetype to java precommit

[Pablo] Upgrade Gradle shadowJar plugin to latest.

[github] Mention adding a reviewer in PR template. (#5397)

--
[...truncated 92.90 KB...]
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 8,5,main]) completed. Took 0.011 secs.

> Task :beam-sdks-java-core:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:compileTestJava' is 
83259156e0c75a0759c02a43af1b3714
Skipping task ':beam-sdks-java-core:compileTestJava' as it is up-to-date.
:beam-sdks-java-core:compileTestJava (Thread[Task worker for ':' Thread 
15,5,main]) completed. Took 0.245 secs.
:beam-sdks-java-core:processTestResources (Thread[Task worker for ':' Thread 
15,5,main]) started.

> Task :beam-sdks-java-core:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-sdks-java-core:processTestResources' as it has no source 
files and no previous output files.
:beam-sdks-java-core:processTestResources (Thread[Task worker for ':' Thread 
15,5,main]) completed. Took 0.001 secs.
:beam-sdks-java-core:testClasses (Thread[Task worker for ':' Thread 15,5,main]) 
started.

> Task :beam-sdks-java-core:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-core:testClasses' as it has no actions.
:beam-sdks-java-core:testClasses (Thread[Task worker for ':' Thread 15,5,main]) 
completed. Took 0.0 secs.
:beam-sdks-java-core:shadowTestJar (Thread[Task worker for ':' Thread 
15,5,main]) started.

> Task :beam-sdks-java-core:shadowTestJar UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:shadowTestJar' is 
31759c045f2c1a67827711e2376d3dcc
Caching disabled for task ':beam-sdks-java-core:shadowTestJar': Caching has not 
been enabled for the task
Skipping task ':beam-sdks-java-core:shadowTestJar' as it is up-to-date.
:beam-sdks-java-core:shadowTestJar (Thread[Task worker for ':' Thread 
15,5,main]) completed. Took 0.021 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 15,5,main]) started.
:beam-sdks-java-core:jar (Thread[Task worker for ':' Thread 3,5,main]) started.

> Task :beam-sdks-java-core:jar UP-TO-DATE
Build cache key for task ':beam-sdks-java-core:jar' is 
fb6717ddd61274b9b7fd044aabac7564
Caching disabled for task ':beam-sdks-java-core:jar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-core:jar' as it is up-to-date.
:beam-sdks-java-core:jar (Thread[Task worker for ':' Thread 3,5,main]) 
completed. Took 0.01 secs.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' is 
7b3a497d1969a01b80d00dbdc6d9a630
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:compileTestJava 
(Thread[Task worker for ':' Thread 15,5,main]) completed. Took 0.03 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 15,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:testClasses 
> UP-TO-DATE
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:testClasses' as it has 
no actions.
:beam-sdks-java-extensions-google-cloud-platform-core:testClasses (Thread[Task 
worker for ':' Thread 15,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
(Thread[Task worker for ':' Thread 15,5,main]) started.

> Task :beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 
> UP-TO-DATE
Build cache key for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' is 
5f9f72355f25aa1466187736f64c01f7
Caching disabled for task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar': Caching 
has not been enabled for the task
Skipping task 
':beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-extensions-google-cloud-platform-core:shadowTestJar 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103180=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103180
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189133606
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103178
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189133805
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103174
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189047348
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,307 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  private final ServerFactory serverFactory;
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final GrpcFnServer provisioningServer;
+
+  private final LoadingCache 
environmentCache;
+
+  public static DockerJobBundleFactory create(ArtifactSource artifactSource) 
throws Exception {
+DockerCommand 

[jira] [Updated] (BEAM-4358) Create test artifacts

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-4358:
--
Description: Currently things like TestPipeline and TestPubsub implement 
TestRule and thus require the project to depend on Junit. We need to create 
separate artifacts for these and depend on Junit only in test scope.  (was: 
Currently things like TestPipeline implement TestRule and thus require 
dependency on Junit. We need to create separate artifacts for these and depend 
on Junit only in test scope.)

> Create test artifacts
> -
>
> Key: BEAM-4358
> URL: https://issues.apache.org/jira/browse/BEAM-4358
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Anton Kedin
>Priority: Major
>
> Currently things like TestPipeline and TestPubsub implement TestRule and thus 
> require the project to depend on Junit. We need to create separate artifacts 
> for these and depend on Junit only in test scope.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?focusedWorklogId=103176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103176
 ]

ASF GitHub Bot logged work on BEAM-4267:


Author: ASF GitHub Bot
Created on: 18/May/18 00:01
Start Date: 18/May/18 00:01
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5392: 
[BEAM-4267] JobBundleFactory that uses Docker-backed environments
URL: https://github.com/apache/beam/pull/5392#discussion_r189133847
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
 ##
 @@ -0,0 +1,339 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.control;
+
+import com.google.common.cache.CacheBuilder;
+import com.google.common.cache.CacheLoader;
+import com.google.common.cache.LoadingCache;
+import com.google.common.cache.RemovalNotification;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import com.google.common.net.HostAndPort;
+import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
+import java.io.IOException;
+import java.time.Duration;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import javax.annotation.concurrent.ThreadSafe;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.fnexecution.v1.ProvisionApi.ProvisionInfo;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcContextHeaderAccessorProvider;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.ServerFactory;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactSource;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.BundleProcessor;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.DockerCommand;
+import 
org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import org.apache.beam.runners.fnexecution.logging.Slf4jLogWriter;
+import 
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.IdGenerator;
+import org.apache.beam.sdk.fn.IdGenerators;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A {@link JobBundleFactory} that uses a {@link DockerEnvironmentFactory} for 
environment
+ * management. Note that returned {@link StageBundleFactory stage bundle 
factories} are not
+ * thread-safe. Instead, a new stage factory should be created for each client.
+ */
+@ThreadSafe
+public class DockerJobBundleFactory implements JobBundleFactory {
+  private static final Logger LOG = 
LoggerFactory.getLogger(DockerJobBundleFactory.class);
+
+  // TODO: This host name seems to change with every other Docker release. Do 
we attempt to keep up
+  // or attempt to document the supported Docker version(s)?
+  private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+  private final IdGenerator stageIdGenerator;
+  private final GrpcFnServer controlServer;
+  private final GrpcFnServer loggingServer;
+  private final GrpcFnServer retrievalServer;
+  private final 

[jira] [Updated] (BEAM-4358) Create test artifacts

[Anton Kedin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Kedin updated BEAM-4358:
--
Description: Currently things like TestPipeline and TestPubsub implement 
TestRule and thus require the project to depend on Junit. We need to create 
separate artifacts for these test utilities and depend on Junit only in test 
scope.  (was: Currently things like TestPipeline and TestPubsub implement 
TestRule and thus require the project to depend on Junit. We need to create 
separate artifacts for these and depend on Junit only in test scope.)

> Create test artifacts
> -
>
> Key: BEAM-4358
> URL: https://issues.apache.org/jira/browse/BEAM-4358
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Anton Kedin
>Priority: Major
>
> Currently things like TestPipeline and TestPubsub implement TestRule and thus 
> require the project to depend on Junit. We need to create separate artifacts 
> for these test utilities and depend on Junit only in test scope.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #26

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-4358) Create test artifacts

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-4358:
-

 Summary: Create test artifacts
 Key: BEAM-4358
 URL: https://issues.apache.org/jira/browse/BEAM-4358
 Project: Beam
  Issue Type: Improvement
  Components: build-system, testing
Reporter: Anton Kedin


Currently things like TestPipeline implement TestRule and thus require 
dependency on Junit. We need to create separate artifacts for these and depend 
on Junit only in test scope.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #447

[Apache Jenkins Server]

See 


Changes:

[Pablo] Remove stale TODO comments for adding archetype to java precommit

--
[...truncated 19.33 MB...]
INFO: Uploading <115879 bytes, hash 6JebjFuFBDWZ0SKZZun08w> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testreportfailures-jenkins-0517234554-487a7577/output/results/staging/pipeline-6JebjFuFBDWZ0SKZZun08w.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 17, 2018 11:46:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-05-17_16_46_01-1435962551010514314?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-05-17_16_46_01-1435962551010514314

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
May 17, 2018 11:46:02 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_16_46_01-1435962551010514314
May 17, 2018 11:46:02 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-17_16_46_01-1435962551010514314 with 0 
expected assertions.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:01.630Z: Autoscaling is enabled for job 
2018-05-17_16_46_01-1435962551010514314. The number of workers will be between 
1 and 1000.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:01.673Z: Autoscaling was automatically enabled for 
job 2018-05-17_16_46_01-1435962551010514314.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:04.105Z: Checking required Cloud APIs are enabled.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:04.237Z: Checking permissions granted to controller 
Service Account.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:08.177Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:08.620Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:08.861Z: Expanding GroupByKey operations into 
optimizable parts.
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:08.907Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:09.197Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:09.241Z: Elided trivial flatten 
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:09.290Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:09.346Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 17, 2018 11:46:13 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:46:09.391Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 17, 2018 11:46:13 PM 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #448

[Apache Jenkins Server]

See 


Changes:

[Pablo] Upgrade Gradle shadowJar plugin to latest.

--
[...truncated 18.86 MB...]
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-17_16_44_50-14269768543203248994
May 17, 2018 11:44:51 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-05-17_16_44_50-14269768543203248994 with 0 
expected assertions.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:50.931Z: Autoscaling is enabled for job 
2018-05-17_16_44_50-14269768543203248994. The number of workers will be between 
1 and 1000.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:50.956Z: Autoscaling was automatically enabled for 
job 2018-05-17_16_44_50-14269768543203248994.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:53.514Z: Checking required Cloud APIs are enabled.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:53.672Z: Checking permissions granted to controller 
Service Account.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.046Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.433Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.567Z: Expanding GroupByKey operations into 
optimizable parts.
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.592Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.754Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.772Z: Elided trivial flatten 
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.799Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.826Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.850Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.875Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.897Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 17, 2018 11:45:03 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-17T23:44:57.922Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=103171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103171
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 23:35
Start Date: 17/May/18 23:35
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r189130230
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,551 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  @staticmethod
+  def _download_file(from_url, to_path):
+"""Downloads a file over http/https from a url or copy it from a remote
+path to local path."""
+if from_url.startswith('http://') or from_url.startswith('https://'):
+  # TODO(silviuc): We should cache downloads so we do not do it for every
+  # job.
+  try:
+# We check if the file is actually there because wget returns a file
+# even for a 404 response (file will contain the contents of the 404
+# response).
+# TODO(angoenka): Extract and use the filename when downloading file.
+response, content = __import__('httplib2').Http().request(from_url)
+if int(response['status']) >= 400:
+  raise RuntimeError(
+  'Artifact not found at %s (response: %s)' % 

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=103172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103172
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 23:35
Start Date: 17/May/18 23:35
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-390046389
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103172)
Time Spent: 15.5h  (was: 15h 20m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 15.5h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Mention adding a reviewer in PR template. (#5397)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 31d7e95  Mention adding a reviewer in PR template. (#5397)
31d7e95 is described below

commit 31d7e951be728b5b2e6fb489fd27c16e875b58f1
Author: Pablo 
AuthorDate: Thu May 17 16:34:28 2018 -0700

Mention adding a reviewer in PR template. (#5397)

Simplifying PR template. Mention adding a reviewer.
---
 .github/PULL_REQUEST_TEMPLATE.md | 13 +++--
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index 0cf5002..00c270c 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,17 +1,10 @@
-DESCRIPTION HERE
+**Please** add a meaningful description for your change here
 
 
 
 Follow this checklist to help us incorporate your contribution quickly and 
easily:
 
- - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- - [ ] Write a pull request description that is detailed enough to understand:
-   - [ ] What the pull request does
-   - [ ] Why it does it
-   - [ ] How it does it
-   - [ ] Why this approach
- - [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- - [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
+ - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
  - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
 
+It will help us expedite review of your Pull Request if you tag someone (e.g. 
`@username`) to look at it.

-- 
To stop receiving notification emails like this one, please contact
pabl...@apache.org.

[jira] [Updated] (BEAM-4254) Upgrade Bigtable client to 1.3

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath updated BEAM-4254:
-
Fix Version/s: (was: 2.5.0)

> Upgrade Bigtable client to 1.3
> --
>
> Key: BEAM-4254
> URL: https://issues.apache.org/jira/browse/BEAM-4254
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4254) Upgrade Bigtable client to 1.3

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479877#comment-16479877
 ] 

Chamikara Jayalath commented on BEAM-4254:
--

cc: [~sduskis]

 

Looks like this requires upgrading protobuf and gRPC dependencies which is 
planned but cannot be done before 2.5.0.

> Upgrade Bigtable client to 1.3
> --
>
> Key: BEAM-4254
> URL: https://issues.apache.org/jira/browse/BEAM-4254
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4001) Futurize and fix python 2 compatibility for metrics subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4001?focusedWorklogId=103170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103170
 ]

ASF GitHub Bot logged work on BEAM-4001:


Author: ASF GitHub Bot
Created on: 17/May/18 23:29
Start Date: 17/May/18 23:29
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #5335: [BEAM-4001] 
Futurize metrics subpackage
URL: https://github.com/apache/beam/pull/5335#issuecomment-390045147
 
 
   Yes, I think we can merge this, thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103170)
Time Spent: 40m  (was: 0.5h)

> Futurize and fix python 2 compatibility for metrics subpackage
> --
>
> Key: BEAM-4001
> URL: https://issues.apache.org/jira/browse/BEAM-4001
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #25

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Add support for DQL to PubsubJsonTable

[kedin] [SQL] Add integration test for Pubsub Json Table DLQ

[apilloud] [SQL] Add jdbc:beam:

[apilloud] [SQL] Use CalciteConnection as context

[ankurgoenka] Copyting dependency.py and dependency_test.py

[ankurgoenka] reformatting code

[ankurgoenka] Introducing classes

[ankurgoenka] gruping public and non public methods

[ankurgoenka] Renaming method and updating reference for file_copy and 
file_download

[ankurgoenka] Removing Google specific code and unused code in stager.

[ankurgoenka] Removing reference to Google/GCS/Dataflow and restructuring a bit 
of

[ankurgoenka] Fixing Stager Tests

[ankurgoenka] Making dependency.py use stager.py and fixing dependency_test.py 
test

[ankurgoenka] Moving stager to portability

[ankurgoenka] Removing stage_job_resources from dependencies.py and other minor

[ankurgoenka] Merging filehandler in stager

[ankurgoenka] renaming GCSStager to _ParameterizedStager

[ankurgoenka] Making a few methods static

--
[...truncated 156.14 KB...]
  File 
"
 line 180, in wrapper
return fun(*args, **kwargs)
  File 
"
 line 485, in create_job
self.create_job_description(job)
  File 
"
 line 511, in create_job_description
StringIO(job.proto_pipeline.SerializeToString()))
  File 
"
 line 467, in stage_file
response = self._storage_client.objects.Insert(request, upload=upload)
  File 
"
 line 971, in Insert
download=download)
  File 
"
 line 720, in _RunMethod
http, http_request, **opts)
  File 
"
 line 346, in MakeRequest
check_response_func=check_response_func)
  File 
"
 line 396, in _MakeRequestNoRetry
redirections=redirections, connection_type=connection_type)
  File 
"
 line 175, in new_request
redirections, connection_type)
  File 
"
 line 282, in request
connection_type=connection_type)
  File 
"
 line 1609, in request
(response, content) = self._request(conn, authority, uri, request_uri, 
method, body, headers, redirections, cachekey)
  File 
"
 line 1351, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, 
headers)
  File 
"
 line 1307, in _conn_request
response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 756, in recv
return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 643, in read
v = self._sslobj.read(len)
  File 

[jira] [Commented] (BEAM-2588) Portable Flink Runner Job API

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479875#comment-16479875
 ] 

Eugene Kirpichov commented on BEAM-2588:


Axel said Robert is going to work on this.

> Portable Flink Runner Job API
> -
>
> Key: BEAM-2588
> URL: https://issues.apache.org/jira/browse/BEAM-2588
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> The portable Flink runner needs to be wired into a job server so that it can 
> accept jobs the job api (https://s.apache.org/beam-job-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2588) Portable Flink Runner Job API

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479874#comment-16479874
 ] 

Eugene Kirpichov commented on BEAM-2588:


Now it remains to create a Flink-specific JobInvoker (presumably wrapping 
[https://github.com/apache/beam/pull/5226] ), and wrap it with the JobService 
from [https://github.com/apache/beam/pull/5262] .

> Portable Flink Runner Job API
> -
>
> Key: BEAM-2588
> URL: https://issues.apache.org/jira/browse/BEAM-2588
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> The portable Flink runner needs to be wired into a job server so that it can 
> accept jobs the job api (https://s.apache.org/beam-job-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-2588) Portable Flink Runner Job API

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479874#comment-16479874
 ] 

Eugene Kirpichov edited comment on BEAM-2588 at 5/17/18 11:26 PM:
--

Now it remains to create a Flink-specific JobInvoker (presumably wrapping 
[https://github.com/apache/beam/pull/5226] ), and wrap it with the JobService 
from [https://github.com/apache/beam/pull/5262] . PR numbers similar but not 
identical.


was (Author: jkff):
Now it remains to create a Flink-specific JobInvoker (presumably wrapping 
[https://github.com/apache/beam/pull/5226] ), and wrap it with the JobService 
from [https://github.com/apache/beam/pull/5262] .

> Portable Flink Runner Job API
> -
>
> Key: BEAM-2588
> URL: https://issues.apache.org/jira/browse/BEAM-2588
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> The portable Flink runner needs to be wired into a job server so that it can 
> accept jobs the job api (https://s.apache.org/beam-job-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2588) Portable Flink Runner Job API

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-2588:
--

Assignee: Robert Bradshaw  (was: Axel Magnuson)

> Portable Flink Runner Job API
> -
>
> Key: BEAM-2588
> URL: https://issues.apache.org/jira/browse/BEAM-2588
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> The portable Flink runner needs to be wired into a job server so that it can 
> accept jobs the job api (https://s.apache.org/beam-job-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2588) Portable Flink Runner Job API

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2588?focusedWorklogId=103167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103167
 ]

ASF GitHub Bot logged work on BEAM-2588:


Author: ASF GitHub Bot
Created on: 17/May/18 23:23
Start Date: 17/May/18 23:23
Worklog Time Spent: 10m 
  Work Description: jkff closed pull request #5262: [BEAM-2588] Portability 
Runner Job Service
URL: https://github.com/apache/beam/pull/5262
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
new file mode 100644
index 000..fde34826e14
--- /dev/null
+++ 
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import com.google.protobuf.Struct;
+import io.grpc.Status;
+import io.grpc.StatusException;
+import io.grpc.StatusRuntimeException;
+import io.grpc.stub.StreamObserver;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A InMemoryJobService that prepares and runs jobs on behalf of a client 
using a
+ * {@link JobInvoker}.
+ *
+ * Job management is handled in-memory rather than any persistent storage, 
running the risk of
+ * leaking jobs if the InMemoryJobService crashes.
+ *
+ * TODO: replace in-memory job management state with persistent solution.
+ */
+public class InMemoryJobService extends JobServiceGrpc.JobServiceImplBase 
implements FnService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(InMemoryJobService.class);
+
+  public static InMemoryJobService create(
+  Endpoints.ApiServiceDescriptor stagingServiceDescriptor, JobInvoker 
invoker) {
+return new InMemoryJobService(stagingServiceDescriptor, invoker);
+  }
+
+  private final ConcurrentMap preparations;
+  private final ConcurrentMap invocations;
+  private final Endpoints.ApiServiceDescriptor stagingServiceDescriptor;
+  private final JobInvoker invoker;
+
+  private InMemoryJobService(
+  Endpoints.ApiServiceDescriptor stagingServiceDescriptor, JobInvoker 
invoker) {
+this.stagingServiceDescriptor = stagingServiceDescriptor;
+this.invoker = invoker;
+
+this.preparations = new ConcurrentHashMap<>();
+this.invocations = new ConcurrentHashMap<>();
+  }
+
+  

[beam] 01/01: Merge pull request #5262: [BEAM-2588] Portability Runner Job Service

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fd3a3b3d20391f387bce65c3e20aed3c9cc0d821
Merge: 5a942f6 1412587
Author: Eugene Kirpichov 
AuthorDate: Thu May 17 16:23:27 2018 -0700

Merge pull request #5262: [BEAM-2588] Portability Runner Job Service

[BEAM-2588] Portability Runner Job Service

 .../jobsubmission/InMemoryJobService.java  | 264 +
 .../fnexecution/jobsubmission/JobInvocation.java   |  59 +
 .../fnexecution/jobsubmission/JobInvoker.java  |  34 +++
 .../fnexecution/jobsubmission/JobPreparation.java  |  42 
 .../fnexecution/jobsubmission/package-info.java|  22 ++
 .../jobsubmission/InMemoryJobServiceTest.java  | 138 +++
 6 files changed, 559 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (5a942f6 -> fd3a3b3)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5a942f6  Upgrade Gradle shadowJar plugin to latest.
 add 3b72d10  Create jobsubmission package for job management.
 add f1fbf93  Add Job Invocation Interfaces
 add 1412587  Add in-memory implementation of JobService
 new fd3a3b3  Merge pull request #5262: [BEAM-2588] Portability Runner Job 
Service

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../jobsubmission/InMemoryJobService.java  | 264 +
 .../fnexecution/jobsubmission/JobInvocation.java   |  59 +
 .../fnexecution/jobsubmission/JobInvoker.java} |  21 +-
 .../fnexecution/jobsubmission/JobPreparation.java} |  31 ++-
 .../fnexecution/jobsubmission}/package-info.java   |   4 +-
 .../jobsubmission/InMemoryJobServiceTest.java  | 138 +++
 6 files changed, 495 insertions(+), 22 deletions(-)
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
 create mode 100644 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocation.java
 copy 
runners/{core-construction-java/src/main/java/org/apache/beam/runners/core/construction/BeamUrns.java
 => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvoker.java}
 (64%)
 copy 
runners/{reference/java/src/main/java/org/apache/beam/runners/reference/ReferenceRunner.java
 => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobPreparation.java}
 (58%)
 copy runners/{apex/src/main/java/org/apache/beam/runners/apex => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission}/package-info.java
 (88%)
 create mode 100644 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobServiceTest.java

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Work logged] (BEAM-2588) Portable Flink Runner Job API

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2588?focusedWorklogId=103165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103165
 ]

ASF GitHub Bot logged work on BEAM-2588:


Author: ASF GitHub Bot
Created on: 17/May/18 23:21
Start Date: 17/May/18 23:21
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5262: 
[BEAM-2588] Portability Runner Job Service
URL: https://github.com/apache/beam/pull/5262#discussion_r189128166
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/InMemoryJobService.java
 ##
 @@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import com.google.protobuf.Struct;
+import io.grpc.Status;
+import io.grpc.StatusException;
+import io.grpc.StatusRuntimeException;
+import io.grpc.stub.StreamObserver;
+import java.util.UUID;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import org.apache.beam.model.pipeline.v1.Endpoints;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A InMemoryJobService that prepares and runs jobs on behalf of a client 
using a
+ * {@link JobInvoker}.
+ *
+ * Job management is handled in-memory rather than any persistent storage, 
running the risk of
+ * leaking jobs if the InMemoryJobService crashes.
+ *
+ * TODO: replace in-memory job management state with persistent solution.
+ */
+public class InMemoryJobService extends JobServiceGrpc.JobServiceImplBase 
implements FnService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(InMemoryJobService.class);
+
+  public static InMemoryJobService create(
+  Endpoints.ApiServiceDescriptor stagingServiceDescriptor, JobInvoker 
invoker) {
+return new InMemoryJobService(stagingServiceDescriptor, invoker);
+  }
+
+  private final ConcurrentMap preparations;
+  private final ConcurrentMap invocations;
+  private final Endpoints.ApiServiceDescriptor stagingServiceDescriptor;
+  private final JobInvoker invoker;
+
+  private InMemoryJobService(
+  Endpoints.ApiServiceDescriptor stagingServiceDescriptor, JobInvoker 
invoker) {
+this.stagingServiceDescriptor = stagingServiceDescriptor;
+this.invoker = invoker;
+
+this.preparations = new ConcurrentHashMap<>();
+this.invocations = new ConcurrentHashMap<>();
+  }
+
+  @Override
+  public void prepare(
+  PrepareJobRequest request,
+  StreamObserver responseObserver) {
+try {
+  LOG.trace("{} {}", PrepareJobRequest.class.getSimpleName(), request);
+  // insert preparation
+  String preparationId =
+  String.format("%s_%s", request.getJobName(), 
UUID.randomUUID().toString());
+  Struct pipelineOptions = request.getPipelineOptions();
+  if (pipelineOptions == null) {
+throw new NullPointerException("Encountered null pipeline 

[jira] [Work logged] (BEAM-2588) Portable Flink Runner Job API

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2588?focusedWorklogId=103164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103164
 ]

ASF GitHub Bot logged work on BEAM-2588:


Author: ASF GitHub Bot
Created on: 17/May/18 23:14
Start Date: 17/May/18 23:14
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5262: [BEAM-2588] 
Portability Runner Job Service
URL: https://github.com/apache/beam/pull/5262#issuecomment-390042225
 
 
   @jkff: This is ready for review, and no longer a work in progress.  Please 
take a look.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103164)
Time Spent: 5h 40m  (was: 5.5h)

> Portable Flink Runner Job API
> -
>
> Key: BEAM-2588
> URL: https://issues.apache.org/jira/browse/BEAM-2588
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> The portable Flink runner needs to be wired into a job server so that it can 
> accept jobs the job api (https://s.apache.org/beam-job-api).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=103163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103163
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 23:07
Start Date: 17/May/18 23:07
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r189126287
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,551 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  @staticmethod
+  def _download_file(from_url, to_path):
+"""Downloads a file over http/https from a url or copy it from a remote
+path to local path."""
+if from_url.startswith('http://') or from_url.startswith('https://'):
+  # TODO(silviuc): We should cache downloads so we do not do it for every
+  # job.
+  try:
+# We check if the file is actually there because wget returns a file
+# even for a 404 response (file will contain the contents of the 404
+# response).
+# TODO(angoenka): Extract and use the filename when downloading file.
+response, content = __import__('httplib2').Http().request(from_url)
+if int(response['status']) >= 400:
+  raise RuntimeError(
+  'Artifact not found at %s (response: %s)' % 

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=103161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103161
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 17/May/18 23:03
Start Date: 17/May/18 23:03
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5401: [BEAM-4328] Upgrade 
Gradle shadowJar plugin to latest.
URL: https://github.com/apache/beam/pull/5401#issuecomment-390040119
 
 
   Nice. Merging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103161)
Time Spent: 1h  (was: 50m)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4328) gradle release build failed on task :beam-sdks-java-io-google-cloud-platform:test

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4328?focusedWorklogId=103162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103162
 ]

ASF GitHub Bot logged work on BEAM-4328:


Author: ASF GitHub Bot
Created on: 17/May/18 23:03
Start Date: 17/May/18 23:03
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5401: [BEAM-4328] Upgrade 
Gradle shadowJar plugin to latest.
URL: https://github.com/apache/beam/pull/5401
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/build.gradle b/build.gradle
index 74e82b886d7..377f4598067 100644
--- a/build.gradle
+++ b/build.gradle
@@ -60,7 +60,7 @@ buildscript {
 classpath "gradle.plugin.com.github.blindpirate:gogradle:0.9"  
 // Enable Go code compilation
 classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
 // Enable building Docker containers
 classpath "cz.malohlava:visteg:1.0.3"  
 // Enable generating Gradle task dependencies as 
".dot" files
-classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
 // Enable shading Java dependencies
+classpath "com.github.jengelman.gradle.plugins:shadow:2.0.4"   
 // Enable shading Java dependencies
 classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
 // Enable the JavaCC parser generator
 classpath 
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
 // Enable creating an offline repository
 classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13"
 // Enable errorprone Java static analysis


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103162)
Time Spent: 1h 10m  (was: 1h)

> gradle release build failed on task 
> :beam-sdks-java-io-google-cloud-platform:test
> -
>
> Key: BEAM-4328
> URL: https://issues.apache.org/jira/browse/BEAM-4328
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Boyuan Zhang
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Running cmd against master branch: ./gradlew clean & ./gradlew -PisRelease 
> build 
> --no-parallel failed into task :beam-sdks-java-io-google-cloud-platform:test.
> Gradle scan: 
> [https://scans.gradle.com/s/sjwr5mixxwen2/console-log?task=:beam-sdks-java-io-google-cloud-platform:test]
>  
> Running ./gradlew clean && ./gradlew  
> :beam-sdks-java-io-google-cloud-platform:test -PisRelease
> --no-parallel succeed: 
> [https://scans.gradle.com/s/rvf7gajba7who|https://www.google.com/url?q=https://scans.gradle.com/s/rvf7gajba7who=D=hangouts=1526598527775000=AFQjCNElmRkjXFh0W-5qxPHJ1h0YMh_jgw]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3256) Add archetype testing/generation to existing GradleBuild PreCommit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3256?focusedWorklogId=103160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103160
 ]

ASF GitHub Bot logged work on BEAM-3256:


Author: ASF GitHub Bot
Created on: 17/May/18 23:02
Start Date: 17/May/18 23:02
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5403: [BEAM-3256] Remove 
stale TODO comments for adding archetype to java precommit
URL: https://github.com/apache/beam/pull/5403
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/maven-archetypes/examples/build.gradle 
b/sdks/java/maven-archetypes/examples/build.gradle
index 79139c4725d..b0f2d2b5da6 100644
--- a/sdks/java/maven-archetypes/examples/build.gradle
+++ b/sdks/java/maven-archetypes/examples/build.gradle
@@ -61,8 +61,6 @@ task generateSources(type: Exec) {
   commandLine './generate-sources.sh'
 }
 
-// Add archetype testing/generation to existing GradleBuild PreCommit
-// https://issues.apache.org/jira/browse/BEAM-3256
 task generateAndBuildArchetypeTest(type: Exec) {
   if (project.hasProperty("maven_home")) {
 commandLine "${maven_home}/bin/mvn", 'clean', 'install'
diff --git a/sdks/java/maven-archetypes/starter/build.gradle 
b/sdks/java/maven-archetypes/starter/build.gradle
index 01d02861345..9c5e6aa0b20 100644
--- a/sdks/java/maven-archetypes/starter/build.gradle
+++ b/sdks/java/maven-archetypes/starter/build.gradle
@@ -23,8 +23,6 @@ description = "Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter"
 ext.summary = """A Maven archetype to create a simple starter pipeline to
 get started using the Apache Beam Java SDK."""
 
-// Add archetype testing/generation to existing GradleBuild PreCommit
-// https://issues.apache.org/jira/browse/BEAM-3256
 task generateAndBuildArchetypeTest(type: Exec) {
   if (project.hasProperty("maven_home")) {
 commandLine "${maven_home}/bin/mvn", 'clean', 'install'


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103160)
Time Spent: 2h 50m  (was: 2h 40m)

> Add archetype testing/generation to existing GradleBuild PreCommit
> --
>
> Key: BEAM-3256
> URL: https://issues.apache.org/jira/browse/BEAM-3256
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> This build currently is not exercising the archetype build and tests 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Java_GradleBuild.groovy
> found here:
> https://github.com/apache/beam/tree/master/sdks/java/maven-archetypes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Upgrade Gradle shadowJar plugin to latest.

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 5a942f6  Upgrade Gradle shadowJar plugin to latest.
5a942f6 is described below

commit 5a942f6ec8929d9c4bc173ef93e83e4ebdca1c51
Author: Scott Wegner 
AuthorDate: Thu May 17 13:46:37 2018 -0700

Upgrade Gradle shadowJar plugin to latest.

http://imperceptiblethoughts.com/shadow/#change_log
---
 build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/build.gradle b/build.gradle
index 879107d..9cc86f0 100644
--- a/build.gradle
+++ b/build.gradle
@@ -60,7 +60,7 @@ buildscript {
 classpath "gradle.plugin.com.github.blindpirate:gogradle:0.9"  
 // Enable Go code compilation
 classpath "gradle.plugin.com.palantir.gradle.docker:gradle-docker:0.13.0"  
 // Enable building Docker containers
 classpath "cz.malohlava:visteg:1.0.3"  
 // Enable generating Gradle task dependencies as 
".dot" files
-classpath "com.github.jengelman.gradle.plugins:shadow:2.0.1"   
 // Enable shading Java dependencies
+classpath "com.github.jengelman.gradle.plugins:shadow:2.0.4"   
 // Enable shading Java dependencies
 classpath "ca.coglinc:javacc-gradle-plugin:2.4.0"  
 // Enable the JavaCC parser generator
 classpath 
"gradle.plugin.io.pry.gradle.offline_dependencies:gradle-offline-dependencies-plugin:0.3"
 // Enable creating an offline repository
 classpath "net.ltgt.gradle:gradle-errorprone-plugin:0.0.13"
 // Enable errorprone Java static analysis

-- 
To stop receiving notification emails like this one, please contact
pabl...@apache.org.

[jira] [Work logged] (BEAM-3256) Add archetype testing/generation to existing GradleBuild PreCommit

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3256?focusedWorklogId=103159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103159
 ]

ASF GitHub Bot logged work on BEAM-3256:


Author: ASF GitHub Bot
Created on: 17/May/18 23:02
Start Date: 17/May/18 23:02
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5403: [BEAM-3256] Remove 
stale TODO comments for adding archetype to java precommit
URL: https://github.com/apache/beam/pull/5403#issuecomment-390039997
 
 
   LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103159)
Time Spent: 2h 40m  (was: 2.5h)

> Add archetype testing/generation to existing GradleBuild PreCommit
> --
>
> Key: BEAM-3256
> URL: https://issues.apache.org/jira/browse/BEAM-3256
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> This build currently is not exercising the archetype build and tests 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_PreCommit_Java_GradleBuild.groovy
> found here:
> https://github.com/apache/beam/tree/master/sdks/java/maven-archetypes



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4001) Futurize and fix python 2 compatibility for metrics subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4001?focusedWorklogId=103158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103158
 ]

ASF GitHub Bot logged work on BEAM-4001:


Author: ASF GitHub Bot
Created on: 17/May/18 23:01
Start Date: 17/May/18 23:01
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5335: [BEAM-4001] Futurize 
metrics subpackage
URL: https://github.com/apache/beam/pull/5335#issuecomment-390039885
 
 
   Should this be merged?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103158)
Time Spent: 0.5h  (was: 20m)

> Futurize and fix python 2 compatibility for metrics subpackage
> --
>
> Key: BEAM-4001
> URL: https://issues.apache.org/jira/browse/BEAM-4001
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4357) SQL shaded jar returns NoClassDefFoundError

[Andrew Pilloud (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Pilloud updated BEAM-4357:
-
Summary: SQL shaded jar returns NoClassDefFoundError  (was: Make SQL shaded 
jar returns NoClassDefFoundError)

> SQL shaded jar returns NoClassDefFoundError
> ---
>
> Key: BEAM-4357
> URL: https://issues.apache.org/jira/browse/BEAM-4357
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Affects Versions: 2.5.0
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> com/fasterxml/jackson/databind/ObjectMapper
> I ran this:
> {code:java}
> java -cp 
> ./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
>  sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=103157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103157
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 17/May/18 22:57
Start Date: 17/May/18 22:57
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #5251: [BEAM-3883] Refactor 
and clean dependency.py to make it reusable with artifact service
URL: https://github.com/apache/beam/pull/5251#issuecomment-390039131
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 103157)
Time Spent: 15h  (was: 14h 50m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 15h
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4357) Make SQL shaded jar returns NoClassDefFoundError

[Andrew Pilloud (JIRA)]

Andrew Pilloud created BEAM-4357:


 Summary: Make SQL shaded jar returns NoClassDefFoundError
 Key: BEAM-4357
 URL: https://issues.apache.org/jira/browse/BEAM-4357
 Project: Beam
  Issue Type: Bug
  Components: dsl-sql
Affects Versions: 2.5.0
Reporter: Andrew Pilloud
Assignee: Andrew Pilloud


Exception in thread "main" java.lang.NoClassDefFoundError: 
com/fasterxml/jackson/databind/ObjectMapper

I ran this:
{code:java}
java -cp 
./sdks/java/extensions/sql/build/libs/beam-sdks-java-extensions-sql-2.5.0-SNAPSHOT-shaded.jar:$HOME/sqlline/sqlline-1.3.0.jar:$HOME/sqlline/jline-2.14.6.jar
 sqlline.SqlLine -u 'jdbc:beam:' -n admin -p admin{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)