date:20180605



 [ 
https://issues.apache.org/jira/browse/BEAM-4479?focusedWorklogId=109292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109292
 ]

ASF GitHub Bot logged work on BEAM-4479:


Author: ASF GitHub Bot
Created on: 06/Jun/18 05:46
Start Date: 06/Jun/18 05:46
Worklog Time Spent: 10m 
  Work Description: vesense commented on issue #459: [BEAM-4479] Fixed 
document for Coder
URL: https://github.com/apache/beam-site/pull/459#issuecomment-394947464
 
 
   @lukecwik Updated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109292)
Time Spent: 40m  (was: 0.5h)

> Fixed document for Coder
> 
>
> Key: BEAM-4479
> URL: https://issues.apache.org/jira/browse/BEAM-4479
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Xin Wang
>Assignee: Melissa Pashniak
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> {code:java}CoderRegistry.getDefaultCoder{code} had been removed since 
> release-2.0.0, however, the document wasn't updated. This patch fixed that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.5.0 created (now 0a7fc57)

2018-06-05 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch release-2.5.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at 0a7fc57  Prepare 2.5.0 release

This branch includes the following new commits:

 new 0a7fc57  Prepare 2.5.0 release

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] branch master updated: Moving to 2.6.0-SNAPSHOT on master branch

2018-06-05 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 72cbd99  Moving to 2.6.0-SNAPSHOT on master branch
72cbd99 is described below

commit 72cbd99d6b62bc7ed16dbd1288cd61d54e8bda37
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Jun 6 07:34:11 2018 +0200

Moving to 2.6.0-SNAPSHOT on master branch
---
 build_rules.gradle  | 2 +-
 examples/java/pom.xml   | 2 +-
 examples/pom.xml| 2 +-
 gradle.properties   | 2 +-
 model/fn-execution/pom.xml  | 2 +-
 model/job-management/pom.xml| 2 +-
 model/pipeline/pom.xml  | 2 +-
 model/pom.xml   | 2 +-
 pom.xml | 2 +-
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/extensions-java/metrics/pom.xml | 2 +-
 runners/extensions-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/gcp/gcemd/pom.xml   | 2 +-
 runners/gcp/gcsproxy/pom.xml| 2 +-
 runners/gcp/pom.xml | 2 +-
 runners/gearpump/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/java-fn-execution/pom.xml   | 2 +-
 runners/local-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/reference/java/pom.xml  | 2 +-
 runners/reference/pom.xml   | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/go/container/pom.xml   | 2 +-
 sdks/go/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/container/pom.xml | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sketching/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/extensions/sql/pom.xml| 2 +-
 sdks/java/fn-execution/pom.xml  | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amazon-web-services/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml| 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/file-based-io-tests/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-

[beam] 01/01: Prepare 2.5.0 release

2018-06-05 Thread jbonofre

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.5.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 0a7fc5798943bd8c0196409c4136731b281b0313
Author: Jean-Baptiste Onofré 
AuthorDate: Wed Jun 6 07:37:31 2018 +0200

Prepare 2.5.0 release
---
 runners/google-cloud-dataflow-java/build.gradle | 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 sdks/python/apache_beam/version.py  | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/runners/google-cloud-dataflow-java/build.gradle 
b/runners/google-cloud-dataflow-java/build.gradle
index c5deb9c..79d646a 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -36,7 +36,7 @@ processResources {
   filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
 'dataflow.legacy_environment_major_version' : '7',
 'dataflow.fnapi_environment_major_version' : '7',
-'dataflow.container_version' : 'beam-master-20180601'
+'dataflow.container_version' : 'beam-2.5.0'
   ]
 }
 
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 086c6c7..0f497be 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20180601
+beam-2.5.0
 
7
 
7
   
diff --git a/sdks/python/apache_beam/version.py 
b/sdks/python/apache_beam/version.py
index 6559a36b8..f332567 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -18,4 +18,4 @@
 """Apache Beam SDK version information and utilities."""
 
 
-__version__ = '2.5.0.dev'
+__version__ = '2.5.0'

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project



 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109288=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109288
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 06/Jun/18 05:13
Start Date: 06/Jun/18 05:13
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #5540: [BEAM-4313] 
Fix and enforce FindBugs and ErrorProne violations in Dataflow Runner
URL: https://github.com/apache/beam/pull/5540#issuecomment-394942469
 
 
   I don't have time to do a thorough review today/tomorrow @iemejia but I've 
read the commits (not pulled them) and they all look reasonable. The only thing 
I noted was to verify the use of `transient` for the runner - I just didn't 
verify the implication. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109288)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4507) Objects that have Fixed Window after a Global Window disappear in Directrunner

2018-06-05 Thread Kenneth Knowles (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-4507:
-

Assignee: Batkhuyag Batsaikhan  (was: Thomas Groh)

> Objects that have Fixed Window after a Global Window disappear in Directrunner
> --
>
> Key: BEAM-4507
> URL: https://issues.apache.org/jira/browse/BEAM-4507
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Reza ardeshir rokni
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>
> 1- Create a elements that go into a combinar and then drops into a  Keyed 
> Global Window, on entry setup a timer.
> Have the OnTime function:
> 1- c.output the original element with a timestamp == to the original timestamp
> 2- Create a new timestamp with element in the next window
> 2- LOG the output in a DoFn after the keyed state code, you should see all 
> the values.
> 3- Now apply a FixedWindow... and then try and log the output.. the original 
> elements wont appear, but the new OnTime generated ones will.
>  
> Tested in Dataflow runner and things work ok, the direct runner fails. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4507) Objects that have Fixed Window after a Global Window disappear in Directrunner

2018-06-05 Thread Reza ardeshir rokni (JIRA)

Reza ardeshir rokni created BEAM-4507:
-

 Summary: Objects that have Fixed Window after a Global Window 
disappear in Directrunner
 Key: BEAM-4507
 URL: https://issues.apache.org/jira/browse/BEAM-4507
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Affects Versions: 2.4.0
Reporter: Reza ardeshir rokni
Assignee: Thomas Groh


1- Create a elements that go into a combinar and then drops into a  Keyed 
Global Window, on entry setup a timer.
Have the OnTime function:
1- c.output the original element with a timestamp == to the original timestamp
2- Create a new timestamp with element in the next window
2- LOG the output in a DoFn after the keyed state code, you should see all the 
values.
3- Now apply a FixedWindow... and then try and log the output.. the original 
elements wont appear, but the new OnTime generated ones will.
 
Tested in Dataflow runner and things work ok, the direct runner fails. 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5532: [BEAM-4423] Mark pull requests stale after 60 days; close 7 days after warning

2018-06-05 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 19de5232086d7026950ea55e69a2733a8e9e3a4b
Merge: 5b9faa4 254796e
Author: Kenn Knowles 
AuthorDate: Tue Jun 5 21:17:48 2018 -0700

Merge pull request #5532: [BEAM-4423] Mark pull requests stale after 60 
days; close 7 days after warning

 .github/stale.yml | 27 +++
 1 file changed, 27 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] branch master updated (5b9faa4 -> 19de523)

2018-06-05 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5b9faa4  [BEAM-4276] Add combiner lifting support to Go SDK
 add 254796e  [BEAM-4423] Mark pull requests stale after 60 days; close 7 
days after warning
 new 19de523  Merge pull request #5532: [BEAM-4423] Mark pull requests 
stale after 60 days; close 7 days after warning

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/stale.yml | 27 +++
 1 file changed, 27 insertions(+)
 create mode 100644 .github/stale.yml

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-4423) Automatically clean-up stale pull requests



 [ 
https://issues.apache.org/jira/browse/BEAM-4423?focusedWorklogId=109284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109284
 ]

ASF GitHub Bot logged work on BEAM-4423:


Author: ASF GitHub Bot
Created on: 06/Jun/18 04:17
Start Date: 06/Jun/18 04:17
Worklog Time Spent: 10m 
  Work Description: kennknowles closed pull request #5532: [BEAM-4423] Mark 
pull requests stale after 60 days; close 7 days after warning
URL: https://github.com/apache/beam/pull/5532
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.github/stale.yml b/.github/stale.yml
new file mode 100644
index 000..ea86c964141
--- /dev/null
+++ b/.github/stale.yml
@@ -0,0 +1,27 @@
+# Configuration for probot-stale - https://github.com/probot/stale
+
+# Issues or Pull Requests with these labels will never be considered stale. 
Set to `[]` to disable
+exemptLabels:
+  - pinned
+  - security
+
+# Limit to only `issues` or `pulls`
+only: pulls
+
+# Configuration settings that are specific to just 'issues' or 'pulls':
+pulls:
+  daysUntilStale: 60
+  daysUntilClose: 7
+  markComment: >
+This pull request has been marked as stale due to 60 days of inactivity.
+It will be closed in 1 week if no further activity occurs. If you think
+that’s incorrect or this pull request requires a review, please simply
+write any comment. If closed, you can revive the PR at any time and 
@mention
+a reviewer or discuss it on the d...@beam.apache.org list.
+Thank you for your contributions.
+  unmarkComment: >
+This pull request is no longer marked as stale.
+  closeComment: >
+This pull request has been closed due to lack of activity. If you think 
that
+is incorrect, or the pull request requires review, you can revive the PR at
+any time.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109284)
Time Spent: 1h 10m  (was: 1h)

> Automatically clean-up stale pull requests
> --
>
> Key: BEAM-4423
> URL: https://issues.apache.org/jira/browse/BEAM-4423
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Minor
>  Labels: github, infra
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> See conversation on the [dev@ 
> list|https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E]
> We have many pull requests which have gone inactive for one reason or 
> another. These accumulate over time and make it more difficult to track 
> active pull requests. The idea is to have some policy and automation around 
> cleaning up stale PR's.
> For example:
> * Add a comment after X weeks stating the stale PR policy and that this PR 
> will be closed if there is no activity in X days
> * If there is still no activity after X additional days, close the PR.
> We should explore tools that are already built for this, such as 
> https://github.com/probot/stale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4223) game_stats.py not working with the DataflowRunner

2018-06-05 Thread JIRA



[ 
https://issues.apache.org/jira/browse/BEAM-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502805#comment-16502805
 ] 

María GH commented on BEAM-4223:


I ran it locally with Dataflow today and I was able to pull messages sent with 
the Java injector via 

{noformat}
$ gcloud alpha pubsub subscriptions pull 
--project=google.com:deft-testing-integration 
projects/google.com:clouddfe/subscriptions/leader_board-mariagh-sub-1 
--auto-ack --limit 1000
{noformat}

but nothing appeared read in PubSub:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-06-05_00_51_29-14166534973167145023?project=google.com:clouddfe=433637338589



> game_stats.py not working with the DataflowRunner
> -
>
> Key: BEAM-4223
> URL: https://issues.apache.org/jira/browse/BEAM-4223
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
>Reporter: María GH
>Priority: Minor
>
> Command used:
> python apache_beam/examples/complete/game/game_stats.py \
>  --project=google.com:clouddfe \
>  --topic projects/google.com:clouddfe/topics/leader_board\-$USER-topic-1 \
>  --dataset ${USER}_bq_dataset \
>  --runner DataflowRunner \
>  --temp_location gs://<...> \
>  --sdk_location /<...>/dist/apache-beam-2.5.0.dev0.tar.gz
>  
> BigQuery at:
> [https://bigquery.cloud.google.com/dataset/google.com:clouddfe:mariagh_bq_dataset?pli=1]
> Nothing gets output to game_stats_sessions.
>  
> Job:
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-30_12_56_57-10424752922445606094?project=google.com:clouddfe



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4423) Automatically clean-up stale pull requests



 [ 
https://issues.apache.org/jira/browse/BEAM-4423?focusedWorklogId=109283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109283
 ]

ASF GitHub Bot logged work on BEAM-4423:


Author: ASF GitHub Bot
Created on: 06/Jun/18 04:13
Start Date: 06/Jun/18 04:13
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on issue #5532: [BEAM-4423] Mark 
pull requests stale after 60 days; close 7 days after warning
URL: https://github.com/apache/beam/pull/5532#issuecomment-394934511
 
 
   @kennknowles Squashed the commits


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109283)
Time Spent: 1h  (was: 50m)

> Automatically clean-up stale pull requests
> --
>
> Key: BEAM-4423
> URL: https://issues.apache.org/jira/browse/BEAM-4423
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Minor
>  Labels: github, infra
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> See conversation on the [dev@ 
> list|https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E]
> We have many pull requests which have gone inactive for one reason or 
> another. These accumulate over time and make it more difficult to track 
> active pull requests. The idea is to have some policy and automation around 
> cleaning up stale PR's.
> For example:
> * Add a comment after X weeks stating the stale PR policy and that this PR 
> will be closed if there is no activity in X days
> * If there is still no activity after X additional days, close the PR.
> We should explore tools that are already built for this, such as 
> https://github.com/probot/stale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4326) Enforce ErrorProne analysis in the fn-execution project



 [ 
https://issues.apache.org/jira/browse/BEAM-4326?focusedWorklogId=109282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109282
 ]

ASF GitHub Bot logged work on BEAM-4326:


Author: ASF GitHub Bot
Created on: 06/Jun/18 04:01
Start Date: 06/Jun/18 04:01
Worklog Time Spent: 10m 
  Work Description: cademarkegard commented on issue #5542: [BEAM-4326] 
Enforce ErrorProne analysis in the fn-execution project
URL: https://github.com/apache/beam/pull/5542#issuecomment-394933060
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109282)
Time Spent: 1h  (was: 50m)

> Enforce ErrorProne analysis in the fn-execution project
> ---
>
> Key: BEAM-4326
> URL: https://issues.apache.org/jira/browse/BEAM-4326
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
> # In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4326) Enforce ErrorProne analysis in the fn-execution project



 [ 
https://issues.apache.org/jira/browse/BEAM-4326?focusedWorklogId=109281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109281
 ]

ASF GitHub Bot logged work on BEAM-4326:


Author: ASF GitHub Bot
Created on: 06/Jun/18 03:59
Start Date: 06/Jun/18 03:59
Worklog Time Spent: 10m 
  Work Description: cademarkegard commented on issue #5542: [BEAM-4326] 
Enforce ErrorProne analysis in the fn-execution project
URL: https://github.com/apache/beam/pull/5542#issuecomment-394932805
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109281)
Time Spent: 50m  (was: 40m)

> Enforce ErrorProne analysis in the fn-execution project
> ---
>
> Key: BEAM-4326
> URL: https://issues.apache.org/jira/browse/BEAM-4326
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-harness
>Reporter: Scott Wegner
>Assignee: Cade Markegard
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-sdks-java-fn-execution}}. Additional context discussed on the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-sdks-java-fn-execution:assemble}}
> # Fix each ErrorProne warning from the {{sdks/java/fn-execution}} project.
> # In {{sdks/java/fn-execution/build.gradle}}, add {{failOnWarning: true}} to 
> the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109280
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 06/Jun/18 02:43
Start Date: 06/Jun/18 02:43
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394922272
 
 
   I think the rule is legit, so I've added it to the non-JDBC path. Now I have 
plenty of errors to work with.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109280)
Time Spent: 3.5h  (was: 3h 20m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery



 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=109270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109270
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 06/Jun/18 01:21
Start Date: 06/Jun/18 01:21
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r193266362
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeJobService.java
 ##
 @@ -78,7 +78,7 @@
 /**
  * A fake implementation of BigQuery's job service.
  */
-class FakeJobService implements JobService, Serializable {
+public class FakeJobService implements JobService, Serializable {
 
 Review comment:
   @Experimental(Experimental.Kind.SOURCE_SINK)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109270)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery



 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=109269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109269
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 06/Jun/18 01:21
Start Date: 06/Jun/18 01:21
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r193266581
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkConfiguration.java
 ##
 @@ -47,6 +47,10 @@
   @JsonProperty
   public NexmarkUtils.SinkType sinkType = NexmarkUtils.SinkType.DEVNULL;
 
+  /** Shall we export the summary to BigQuery. */
 
 Review comment:
   Please rewrite this comment (sounds like a TODO currently). What you wrote 
above is fine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109269)
Time Spent: 5h 10m  (was: 5h)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery



 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=109268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109268
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 06/Jun/18 01:21
Start Date: 06/Jun/18 01:21
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r193265905
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
 ##
 @@ -1452,7 +1452,10 @@ static String getExtractDestinationUri(String 
extractDestinationDir) {
 }
 
 @VisibleForTesting
-Write withTestServices(BigQueryServices testServices) {
+/**
+ * This method is for test usage only
+ */
+public Write withTestServices(BigQueryServices testServices) {
 
 Review comment:
   Yes. I assume "inject BigQueryServices to savePerfsToBigQuery()" is for 
testing. BigQueryServicesImpl should not be public. I don't think there's a 
point in making 'withTestServices' available for testing without making 
'BigQueryServices' interface available. I think 'withTestServices' in 
combination with FakeBigQueryServices will be useful for other Beam components 
that need to test pipelines that write to BigQuery as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109268)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery



 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=109271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109271
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 06/Jun/18 01:21
Start Date: 06/Jun/18 01:21
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r193266493
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/Main.java
 ##
 @@ -300,4 +390,6 @@ public static void main(String[] args) throws IOException {
 NexmarkLauncher nexmarkLauncher = new 
NexmarkLauncher<>(options);
 new Main<>().runAll(options, nexmarkLauncher);
   }
+
+
 
 Review comment:
   Remove extra newlines.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109271)
Time Spent: 5h 20m  (was: 5h 10m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4283) Export nexmark execution times to bigQuery



 [ 
https://issues.apache.org/jira/browse/BEAM-4283?focusedWorklogId=109267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109267
 ]

ASF GitHub Bot logged work on BEAM-4283:


Author: ASF GitHub Bot
Created on: 06/Jun/18 01:21
Start Date: 06/Jun/18 01:21
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5464: [BEAM-4283] Write Nexmark execution times to bigquery
URL: https://github.com/apache/beam/pull/5464#discussion_r193266011
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/FakeDatasetService.java
 ##
 @@ -46,7 +46,7 @@
 import org.apache.beam.sdk.values.ValueInSingleWindow;
 
 /** A fake dataset service that can be serialized, for use in 
testReadFromTable. */
-class FakeDatasetService implements DatasetService, Serializable {
+public class FakeDatasetService implements DatasetService, Serializable {
 
 Review comment:
   @Experimental(Experimental.Kind.SOURCE_SINK)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109267)
Time Spent: 5h  (was: 4h 50m)

> Export nexmark execution times to bigQuery
> --
>
> Key: BEAM-4283
> URL: https://issues.apache.org/jira/browse/BEAM-4283
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> Nexmark only outputs the results collection to bigQuery and prints in the 
> console the execution times. To supervise Nexmark execution times, we need to 
> store them as well per runner/query/mode



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4223) game_stats.py not working with the DataflowRunner



[ 
https://issues.apache.org/jira/browse/BEAM-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502710#comment-16502710
 ] 

Ahmet Altay commented on BEAM-4223:
---

I do not think anybody looked at it. My best guess based on looking at the job 
was that the job did not have any input. It is also possible that example has 
problems.

> game_stats.py not working with the DataflowRunner
> -
>
> Key: BEAM-4223
> URL: https://issues.apache.org/jira/browse/BEAM-4223
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
>Reporter: María GH
>Priority: Minor
>
> Command used:
> python apache_beam/examples/complete/game/game_stats.py \
>  --project=google.com:clouddfe \
>  --topic projects/google.com:clouddfe/topics/leader_board\-$USER-topic-1 \
>  --dataset ${USER}_bq_dataset \
>  --runner DataflowRunner \
>  --temp_location gs://<...> \
>  --sdk_location /<...>/dist/apache-beam-2.5.0.dev0.tar.gz
>  
> BigQuery at:
> [https://bigquery.cloud.google.com/dataset/google.com:clouddfe:mariagh_bq_dataset?pli=1]
> Nothing gets output to game_stats_sessions.
>  
> Job:
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-30_12_56_57-10424752922445606094?project=google.com:clouddfe



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4223) game_stats.py not working with the DataflowRunner



 [ 
https://issues.apache.org/jira/browse/BEAM-4223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-4223:
-

Assignee: (was: Ahmet Altay)

> game_stats.py not working with the DataflowRunner
> -
>
> Key: BEAM-4223
> URL: https://issues.apache.org/jira/browse/BEAM-4223
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.4.0
>Reporter: María GH
>Priority: Minor
>
> Command used:
> python apache_beam/examples/complete/game/game_stats.py \
>  --project=google.com:clouddfe \
>  --topic projects/google.com:clouddfe/topics/leader_board\-$USER-topic-1 \
>  --dataset ${USER}_bq_dataset \
>  --runner DataflowRunner \
>  --temp_location gs://<...> \
>  --sdk_location /<...>/dist/apache-beam-2.5.0.dev0.tar.gz
>  
> BigQuery at:
> [https://bigquery.cloud.google.com/dataset/google.com:clouddfe:mariagh_bq_dataset?pli=1]
> Nothing gets output to game_stats_sessions.
>  
> Job:
> https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-30_12_56_57-10424752922445606094?project=google.com:clouddfe



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #268

See 


Changes:

[rober] [BEAM-4276] Add combiner lifting support to Go SDK

[rober] fixup! Address comments.

[lcwik] [BEAM-4481] Remove duplicate definitions of dependencies.

[github] Improve default value for experiments set in RuntimeValueProvider

--
[...truncated 192.98 KB...]
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435)
at 
com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:234)
at 
com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67)
at 
com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37)
at 
com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159)
at 
com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286)
at 
com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$2.executeWriteCommandProtocol(MixedBulkWriteOperation.java:455)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run$RunExecutor.execute(MixedBulkWriteOperation.java:646)
at 
com.mongodb.operation.MixedBulkWriteOperation$Run.execute(MixedBulkWriteOperation.java:401)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:179)
at 
com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.188.104.50:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.188.104.50:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #258

See 


Changes:

[rober] [BEAM-4276] Add combiner lifting support to Go SDK

[rober] fixup! Address comments.

[lcwik] [BEAM-4481] Remove duplicate definitions of dependencies.

[github] Improve default value for experiments set in RuntimeValueProvider

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5b9faa416059299d0fe442b67dea5b3c1cb3d83b (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5b9faa416059299d0fe442b67dea5b3c1cb3d83b
Commit message: "[BEAM-4276] Add combiner lifting support to Go SDK"
 > git rev-list --no-walk c1743ccae68a57b46cff3bb13441fb2fbc55e511 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5050179319959696556.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2191640849729469897.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8052042295114095276.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-258
Error from server (AlreadyExists): namespaces "filebasedioithdfs-258" already 
exists
Build step 'Execute shell' marked build as failure

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109259
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 23:56
Start Date: 05/Jun/18 23:56
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5544: [BEAM-4449] Replace 
project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394897067
 
 
   The JDBC path sets up the planner here: 
https://github.com/apache/calcite/blob/master/core/src/main/java/org/apache/calcite/prepare/CalcitePrepareImpl.java#L520
 It doesn't match the non-JDBC path, probably worth fixing. Also keep in mind 
there are a bunch of ways to register rules when various objects are first seen 
by the planner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109259)
Time Spent: 3h 20m  (was: 3h 10m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109257=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109257
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 23:51
Start Date: 05/Jun/18 23:51
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394896280
 
 
   How do we manage the rulesets that go through JDBC? Adding that particular 
rule to our rule sets causes the failure to occur in ~50 more places. So at 
least a quick fix is to not use the rule...


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109257)
Time Spent: 3h 10m  (was: 3h)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109247
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 23:10
Start Date: 05/Jun/18 23:10
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394889156
 
 
   Also noting: `CalcRemoveRule` was not in the rule sets added here. It is a 
sibling class but something in the JDBC codepath caused it to be added. 
Browsing the rules that are nearby in that package, it looks like really there 
are a huge variety and you are intended to use a whole `Program`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109247)
Time Spent: 3h  (was: 2h 50m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4506) PTransform input_types hint with potential empty input

Ahmet Altay created BEAM-4506:
-

 Summary: PTransform input_types hint with potential empty input
 Key: BEAM-4506
 URL: https://issues.apache.org/jira/browse/BEAM-4506
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Ahmet Altay


User reported issue:

... PTransform class that takes a pcollection of Dict[str, numpy.ndarray] as 
input, and I added the annotation of 
@beam.typehints.with_input_types(beam.typehints.Dict[str, numpy.ndarray]). 
However, this causes error to empty dict input, and the error message is 
"""Input type hint violation at _TopKStatsGeneratorImpl: expected Dict[str, 
ndarray], got Dict[Union[], Union[]] """

>From the user report it seems like our type hints could be more flexible in 
>what they accept for empty dictionaries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109237
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:54
Start Date: 05/Jun/18 22:54
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394885850
 
 
   Fixed that. From what I can tell, the trait simplification issue lies 
entirely within bits we didn't author. The `CalcRemoveRule` tries to convert to 
`getTraitSet()` of a `LogicalCalc` node.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109237)
Time Spent: 2h 50m  (was: 2h 40m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109236
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:54
Start Date: 05/Jun/18 22:54
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394885850
 
 
   Fixed that. From what I can tell, the trait simplification issue lies 
entirely within bits we didn't author. The `CalcRemoveRule` tries to convert to 
`getTraits()` of a `LogicalCalc` node.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109236)
Time Spent: 2h 40m  (was: 2.5h)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4505) Archive/Retire apache/beam-site repository

Scott Wegner created BEAM-4505:
--

 Summary: Archive/Retire apache/beam-site repository
 Key: BEAM-4505
 URL: https://issues.apache.org/jira/browse/BEAM-4505
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4504) Disconnect mergebot from apache/beam-site repository

Scott Wegner created BEAM-4504:
--

 Summary:  Disconnect mergebot from apache/beam-site repository
 Key: BEAM-4504
 URL: https://issues.apache.org/jira/browse/BEAM-4504
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4503) Drain/migrate in-progress PR's from apache/beam-site

Scott Wegner created BEAM-4503:
--

 Summary: Drain/migrate in-progress PR's from apache/beam-site
 Key: BEAM-4503
 URL: https://issues.apache.org/jira/browse/BEAM-4503
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4501) Update contribution guide for new website contribution process

Scott Wegner created BEAM-4501:
--

 Summary: Update contribution guide for new website contribution 
process
 Key: BEAM-4501
 URL: https://issues.apache.org/jira/browse/BEAM-4501
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4502) Add redirect notice on apache/beam-site README.md

Scott Wegner created BEAM-4502:
--

 Summary: Add redirect notice on apache/beam-site README.md
 Key: BEAM-4502
 URL: https://issues.apache.org/jira/browse/BEAM-4502
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4500) Merge [website-migration] branch into master

Scott Wegner created BEAM-4500:
--

 Summary: Merge [website-migration] branch into master
 Key: BEAM-4500
 URL: https://issues.apache.org/jira/browse/BEAM-4500
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4498) Migrate release Javadocs / Pydocs to [asf-site] branch and update release guide

Scott Wegner created BEAM-4498:
--

 Summary: Migrate release Javadocs / Pydocs to [asf-site] branch 
and update release guide
 Key: BEAM-4498
 URL: https://issues.apache.org/jira/browse/BEAM-4498
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4499) Migrate Apache website publishing to use apache/beam asf-site branch

Scott Wegner created BEAM-4499:
--

 Summary: Migrate Apache website publishing to use apache/beam 
asf-site branch
 Key: BEAM-4499
 URL: https://issues.apache.org/jira/browse/BEAM-4499
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4496) Create Jenkins job to push generated HTML to asf-site branch

Scott Wegner created BEAM-4496:
--

 Summary: Create Jenkins job to push generated HTML to asf-site 
branch
 Key: BEAM-4496
 URL: https://issues.apache.org/jira/browse/BEAM-4496
 Project: Beam
  Issue Type: Sub-task
  Components: build-system, website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4497) Add pages for master Javadocs / Pydocs and incorporate into post-commit job

Scott Wegner created BEAM-4497:
--

 Summary: Add pages for master Javadocs / Pydocs and incorporate 
into post-commit job
 Key: BEAM-4497
 URL: https://issues.apache.org/jira/browse/BEAM-4497
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4495) Create website pre-commits for apache/beam repository

Scott Wegner created BEAM-4495:
--

 Summary: Create website pre-commits for apache/beam repository
 Key: BEAM-4495
 URL: https://issues.apache.org/jira/browse/BEAM-4495
 Project: Beam
  Issue Type: Sub-task
  Components: testing, website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4494) Migrate website source code to apache/beam [website-migration] branch

Scott Wegner created BEAM-4494:
--

 Summary: Migrate website source code to apache/beam 
[website-migration] branch
 Key: BEAM-4494
 URL: https://issues.apache.org/jira/browse/BEAM-4494
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4493) Beam-Site Automation Reliability



 [ 
https://issues.apache.org/jira/browse/BEAM-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4493:
---
Labels: beam-site-automation-reliability  (was: )

> Beam-Site Automation Reliability
> 
>
> Key: BEAM-4493
> URL: https://issues.apache.org/jira/browse/BEAM-4493
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Major
>  Labels: beam-site-automation-reliability
>
> https://s.apache.org/beam-site-automation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4445) Filter pre-commit triggering based on touched files



 [ 
https://issues.apache.org/jira/browse/BEAM-4445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-4445:
---
Issue Type: Sub-task  (was: New Feature)
Parent: BEAM-4493

> Filter pre-commit triggering based on touched files
> ---
>
> Key: BEAM-4445
> URL: https://issues.apache.org/jira/browse/BEAM-4445
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: beam-site-automation-reliability
>
> This is discussed in the [Beam-Site Automation 
> Reliability|https://s.apache.org/beam-site-automation] design, under 
> "Pre-Commit Job Filtering"
> The proposal is to filter pre-commit job triggered on PR's based on which 
> files are touched. The impact is that most PRs will only run one set of 
> relevant tests, rather than all three. This will decrease test overhead and 
> the impact of flaky tests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4493) Beam-Site Automation Reliability

Scott Wegner created BEAM-4493:
--

 Summary: Beam-Site Automation Reliability
 Key: BEAM-4493
 URL: https://issues.apache.org/jira/browse/BEAM-4493
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Scott Wegner
Assignee: Scott Wegner


https://s.apache.org/beam-site-automation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project



 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109231
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:40
Start Date: 05/Jun/18 22:40
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5540: [BEAM-4313] Fix and 
enforce FindBugs and ErrorProne violations in Dataflow Runner
URL: https://github.com/apache/beam/pull/5540#issuecomment-394883140
 
 
   All tests are green; I believe this is ready to merge.
   
   @iemejia / @timrobertson100, could one of you please take a look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109231)
Time Spent: 40m  (was: 0.5h)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=109230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109230
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:31
Start Date: 05/Jun/18 22:31
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on issue #424: [BEAM-3255] Updating 
release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#issuecomment-394881364
 
 
   Hey @alanmyrvold , please review commit 
https://github.com/apache/beam-site/pull/424/commits/d238c9ec3ba279188c33b098de0a3064fa8b4216,
 which addressed your comments above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109230)
Time Spent: 9.5h  (was: 9h 20m)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=109229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109229
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:30
Start Date: 05/Jun/18 22:30
Worklog Time Spent: 10m 
  Work Description: boyuanzz commented on a change in pull request #424: 
[BEAM-3255] Updating release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#discussion_r193242145
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -178,64 +231,78 @@ Javadoc to the Javadoc for other modules that Beam 
depends on.
   the version number has changed, download a new version of the corresponding
   `-docs/package-list` file.
 
-### Create a release branch
+### Create a release branch in apache/beam repository
 
-Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the code repository, and update version information on the original branch.
+Attention: Only committer has permission to create release branch in 
apache/beam.
+
+Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the Apache code repository, and update version information on the original 
branch.
 
 Check out the version of the codebase from which you start the release. For a 
new minor or major release, this may be `HEAD` of the `master` branch. To build 
a hotfix/incremental release, instead of the `master` branch, use the release 
tag of the release being patched. (Please make sure your cloned repository is 
up-to-date before starting.)
 
 git checkout 
 
+**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please 
check out the previous release tag, rather than the master branch.
 
 Set up a few environment variables to simplify Maven commands that follow. (We 
use `bash` Unix syntax in this guide.)
 
-VERSION="1.2.3"
-NEXT_VERSION="1.2.4"
-BRANCH_NAME="release-${VERSION}"
-DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
+RELEASE=2.5.0
+NEXT_VERSION_IN_BASE_BRANCH=2.6.0
+BRANCH=release-${RELEASE}
 
 Version represents the release currently underway, while next version 
specifies the anticipated next version to be released from that branch. 
Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4.
 
-Use Maven release plugin to create the release branch and update the current 
branch to use the new development version. This command applies for the new 
major or minor version. (Warning: this command automatically pushes changes to 
the code repository.)
+**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), 
please check out the previous release tag, before running the following 
instructions:
+
+BASE_RELEASE=2.5.0
+RELEASE=2.5.1
+NEXT_VERSION_IN_BASE_BRANCH=2.5.0
+git checkout tags/${BASE_RELEASE}
+
+Create a new branch, and update version files in the master branch.
 
-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DdevelopmentVersion=${DEVELOPMENT_VERSION}
+git branch ${BRANCH}
 
-However, if you are doing an incremental/hotfix release, please run the 
following command after checking out the release tag of the release being 
patched.
+# Now change the version in existing gradle files, and Python files
+sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" 
build_rules.gradle
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" 
sdks/python/apache_beam/version.py
 
-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DupdateWorkingCopyVersions=false \
--DupdateBranchVersions=true \
--DreleaseVersion="${VERSION}-SNAPSHOT"
+# Save changes in master branch
+git add gradle.properties build_rules.gradle 
sdks/python/apache_beam/version.py
+git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master 
branch."
 
 Check out the release branch.
 
-git checkout ${BRANCH_NAME}
+git checkout ${BRANCH}
 
 Review comment:
   We do all master update in previous sed scripts. The rest of instructions 
all happen in release branch.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109229)
Time Spent: 9h 20m  (was: 9h 10m)

> Update release process to use Gradle
>

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately



 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=109228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109228
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:19
Start Date: 05/Jun/18 22:19
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5544: [BEAM-4449] 
Replace project and filter with calc
URL: https://github.com/apache/beam/pull/5544#issuecomment-394878758
 
 
   Well, with the new environment I managed to break the world :-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109228)
Time Spent: 2.5h  (was: 2h 20m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4423) Automatically clean-up stale pull requests



 [ 
https://issues.apache.org/jira/browse/BEAM-4423?focusedWorklogId=109227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109227
 ]

ASF GitHub Bot logged work on BEAM-4423:


Author: ASF GitHub Bot
Created on: 05/Jun/18 22:12
Start Date: 05/Jun/18 22:12
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5532: [BEAM-4423] Mark 
pull requests stale after 60 days; close 7 days after warning
URL: https://github.com/apache/beam/pull/5532#issuecomment-394877176
 
 
   Can you squash the commits? Then I'll merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109227)
Time Spent: 50m  (was: 40m)

> Automatically clean-up stale pull requests
> --
>
> Key: BEAM-4423
> URL: https://issues.apache.org/jira/browse/BEAM-4423
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Scott Wegner
>Assignee: Alan Myrvold
>Priority: Minor
>  Labels: github, infra
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> See conversation on the [dev@ 
> list|https://lists.apache.org/thread.html/bda552ea7073ca165aaf47034610afafe22d589e386525023d33609e@%3Cdev.beam.apache.org%3E]
> We have many pull requests which have gone inactive for one reason or 
> another. These accumulate over time and make it more difficult to track 
> active pull requests. The idea is to have some policy and automation around 
> cleaning up stale PR's.
> For example:
> * Add a comment after X weeks stating the stale PR policy and that this PR 
> will be closed if there is no activity in X days
> * If there is still no activity after X additional days, close the PR.
> We should explore tools that are already built for this, such as 
> https://github.com/probot/stale



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4487) Some errors related to GCP in pip install -e .[gcp]



[ 
https://issues.apache.org/jira/browse/BEAM-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502566#comment-16502566
 ] 

Ahmet Altay commented on BEAM-4487:
---

Related: https://issues.apache.org/jira/browse/BEAM-4215

cc: [~tvalentyn]

> Some errors related to GCP in pip install -e .[gcp]
> ---
>
> Key: BEAM-4487
> URL: https://issues.apache.org/jira/browse/BEAM-4487
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Ahmet Altay
>Priority: Major
>
> When installing Beam with GCP options from source, the following errors pop 
> out:
> https://pastebin.com/vrWWvF9y
> The installation goes through, but the errors are confusing, and may confuse 
> users and contributors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4492) Update Python bigquery library to latest version

2018-06-05 Thread Mark Liu (JIRA)

Mark Liu created BEAM-4492:
--

 Summary: Update Python bigquery library to latest version
 Key: BEAM-4492
 URL: https://issues.apache.org/jira/browse/BEAM-4492
 Project: Beam
  Issue Type: Task
  Components: testing
Reporter: Mark Liu
Assignee: Mark Liu


Current google-cloud-bigquery is set to 0.25.0 in 
https://github.com/apache/beam/blob/master/sdks/python/setup.py#L130

However, the latest version is 1.2.0.

According to comment in setup.py, this library is only used for testing, so it 
should be easy to update. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4391) Example of distributed optimization



[ 
https://issues.apache.org/jira/browse/BEAM-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502556#comment-16502556
 ] 

Ahmet Altay commented on BEAM-4391:
---

[~griscz], how about having a web site page for all examples etc. type of 
contributions and linking to them directly in their own user repositories?

[~joachimvdh] I am hesitant to include more complete example before we have a 
policy on what we want. I do not think we want our core to grow with many 
examples. At the same time we should provide a good answer for where to put new 
examples. I think this is different than juliaset. The reason is, julia set is 
our example for using packaging and serves a documentation need in Beam.

> Example of distributed optimization
> ---
>
> Key: BEAM-4391
> URL: https://issues.apache.org/jira/browse/BEAM-4391
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python
>Reporter: Joachim van der Herten
>Assignee: Joachim van der Herten
>Priority: Minor
>
> Currently, we are writing a blogpost on using the Beam Python SDK for solving 
> distributed optimization tasks. It will include an example of a optimization 
> problem with both discrete and continuous parameters, which is then solved 
> using Apache Beam. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4313) Enforce ErrorProne analysis in Dataflow runner project



 [ 
https://issues.apache.org/jira/browse/BEAM-4313?focusedWorklogId=109222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109222
 ]

ASF GitHub Bot logged work on BEAM-4313:


Author: ASF GitHub Bot
Created on: 05/Jun/18 21:35
Start Date: 05/Jun/18 21:35
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5540: [BEAM-4313] Fix and 
enforce FindBugs and ErrorProne violations in Dataflow Runner
URL: https://github.com/apache/beam/pull/5540#issuecomment-394868131
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109222)
Time Spent: 0.5h  (was: 20m)

> Enforce ErrorProne analysis in Dataflow runner project
> --
>
> Key: BEAM-4313
> URL: https://issues.apache.org/jira/browse/BEAM-4313
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>Priority: Minor
>  Labels: errorprone, starter
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Java ErrorProne static analysis was [recently 
> enabled|https://github.com/apache/beam/pull/5161] in the Gradle build 
> process, but only as warnings. ErrorProne errors are generally useful and 
> easy to fix. Some work was done to [make sdks-java-core 
> ErrorProne-clean|https://github.com/apache/beam/pull/5319] and add 
> enforcement. This task is clean ErrorProne warnings and add enforcement in 
> {{beam-runners-google-cloud-dataflow-java}}. Additional context discussed on 
> the [dev 
> list|https://lists.apache.org/thread.html/95aae2785c3cd728c2d3378cbdff2a7ba19caffcd4faa2049d2e2f46@%3Cdev.beam.apache.org%3E].
> Fixing this issue will involve:
> # Follow instructions in the [Contribution 
> Guide|https://beam.apache.org/contribute/] to set up a {{beam}} development 
> environment.
> # Run the following command to compile and run ErrorProne analysis on the 
> project: {{./gradlew :beam-runners-google-cloud-dataflow-java:assemble}}
> # Fix each ErrorProne warning from the {{runners/google-cloud-dataflow-java}} 
> project.
> # In {{runners/google-cloud-dataflow-java/build.gradle}}, add 
> {{failOnWarning: true}} to the call the {{applyJavaNature()}} 
> ([example|https://github.com/apache/beam/pull/5319/files#diff-9390c20635aed5f42f83b97506a87333R20]).
> This starter issue is sponsored by [~swegner]. Feel free to [reach 
> out|https://beam.apache.org/community/contact-us/] with questions or code 
> review:
> * JIRA: [~swegner]
> * GitHub: [@swegner|https://github.com/swegner]
> * Slack: [@Scott Wegner|https://s.apache.org/beam-slack-channel]
> * Email: swegner at google dot com



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3773) [SQL] Investigate JDBC interface for Beam SQL



 [ 
https://issues.apache.org/jira/browse/BEAM-3773?focusedWorklogId=109219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109219
 ]

ASF GitHub Bot logged work on BEAM-3773:


Author: ASF GitHub Bot
Created on: 05/Jun/18 21:32
Start Date: 05/Jun/18 21:32
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5553: [BEAM-3773] More 
JDBC Tests and Cleanup
URL: https://github.com/apache/beam/pull/5553#issuecomment-394867446
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109219)
Time Spent: 12h  (was: 11h 50m)

> [SQL] Investigate JDBC interface for Beam SQL
> -
>
> Key: BEAM-3773
> URL: https://issues.apache.org/jira/browse/BEAM-3773
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 12h
>  Remaining Estimate: 0h
>
> JDBC allows integration with a lot of third-party tools, e.g 
> [Zeppelin|https://zeppelin.apache.org/docs/0.7.0/manual/interpreters.html], 
> [sqlline|https://github.com/julianhyde/sqlline]. We should look into how 
> feasible it is to implement a JDBC interface for Beam SQL



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4391) Example of distributed optimization

2018-06-05 Thread Joachim van der Herten (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502529#comment-16502529
 ] 

Joachim van der Herten commented on BEAM-4391:
--

I started developing the code in a 
[fork|https://github.com/ml2grow/beam/blob/distrib_optim_blogpost/sdks/python/apache_beam/examples/complete/distrib_optimization.py].
 Should the example be added to the repository, I think it makes sense to add 
it to sdks/python/examples/complete/, similar to the juliaset example 
as the code has two dependencies (numpy and scipy) to be added in a setup.py. 
I'm awaiting the opinion of [~melap] and [~griscz] on this.

> Example of distributed optimization
> ---
>
> Key: BEAM-4391
> URL: https://issues.apache.org/jira/browse/BEAM-4391
> Project: Beam
>  Issue Type: New Feature
>  Components: examples-python
>Reporter: Joachim van der Herten
>Assignee: Joachim van der Herten
>Priority: Minor
>
> Currently, we are writing a blogpost on using the Beam Python SDK for solving 
> distributed optimization tasks. It will include an example of a optimization 
> problem with both discrete and continuous parameters, which is then solved 
> using Apache Beam. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109210=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109210
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 05/Jun/18 21:16
Start Date: 05/Jun/18 21:16
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-394863210
 
 
   Run Dependency Check


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109210)
Time Spent: 40h  (was: 39h 50m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 40h
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4302) Fix to dependency hell



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109205
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 05/Jun/18 21:05
Start Date: 05/Jun/18 21:05
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-394859996
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109205)
Time Spent: 39h 50m  (was: 39h 40m)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 39h 50m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (506db5d -> 32f7fef)

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 506db5d  This closes #456
 add c46fc1e  Prepare repository for deployment.
 new 03059c9  Add a pull request template with staged URL
 new 32f7fef  This closes #461

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .github/PULL_REQUEST_TEMPLATE.md   |  10 +
 content/contribute/become-a-committer/index.html   |   1 +
 content/contribute/committer-guide/index.html  |   1 +
 content/contribute/design-documents/index.html | 354 +
 content/contribute/docker-images/index.html|   1 +
 content/contribute/eclipse/index.html  |   1 +
 content/contribute/index.html  |   1 +
 content/contribute/intellij/index.html |   1 +
 content/contribute/portability/index.html  |   1 +
 .../contribute/ptransform-style-guide/index.html   |   1 +
 content/contribute/release-guide/index.html|   1 +
 content/contribute/runner-guide/index.html |   1 +
 content/contribute/testing/index.html  |   1 +
 .../contribute/website-contributions/index.html|   1 +
 14 files changed, 376 insertions(+)
 create mode 100644 .github/PULL_REQUEST_TEMPLATE.md
 create mode 100644 content/contribute/design-documents/index.html

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Add a pull request template with staged URL

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 03059c99d703e09eccd17fe7524c80cbb8d8c846
Author: Kenneth Knowles 
AuthorDate: Tue Jun 5 10:39:49 2018 -0700

Add a pull request template with staged URL
---
 .github/PULL_REQUEST_TEMPLATE.md | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
new file mode 100644
index 000..7862359
--- /dev/null
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,10 @@
+*Please* add a meaningful description for your change here.
+
+Once your pull request has been opened and assigned a number, please edit the
+URL below, replacing `PULL_REQUEST_NUMBER` with the number of your pull 
request.
+
+http://apache-beam-website-pull-requests.storage.googleapis.com/PULL_REQUEST_NUMBER/index.html
+
+Finally, it will help us expedite review of your Pull Request if you tag
+someone (e.g. @username) to look at it.
+

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #461

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 32f7fef83c94b46dd244d886694b019d200b4e4e
Merge: c46fc1e 03059c9
Author: Mergebot 
AuthorDate: Tue Jun 5 13:55:49 2018 -0700

This closes #461

 .github/PULL_REQUEST_TEMPLATE.md | 10 ++
 1 file changed, 10 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-4302) Fix to dependency hell



 [ 
https://issues.apache.org/jira/browse/BEAM-4302?focusedWorklogId=109202=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109202
 ]

ASF GitHub Bot logged work on BEAM-4302:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:54
Start Date: 05/Jun/18 20:54
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #5406: Do Not Merge, 
[BEAM-4302] add beam dependency checks
URL: https://github.com/apache/beam/pull/5406#issuecomment-394856773
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109202)
Time Spent: 39h 40m  (was: 39.5h)

> Fix to dependency hell
> --
>
> Key: BEAM-4302
> URL: https://issues.apache.org/jira/browse/BEAM-4302
> Project: Beam
>  Issue Type: New Feature
>  Components: testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 39h 40m
>  Remaining Estimate: 0h
>
> # For Java, a daily Jenkins test to compare version of all Beam dependencies 
> to the latest version available in Maven Central.
>  # For Python, a daily Jenkins test to compare versions of all Beam 
> dependencies to the latest version available in PyPI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (3c380a4 -> 5b9faa4)

2018-06-05 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3c380a4  Improve default value for experiments set in 
RuntimeValueProvider (#5562)
 add b3ca82b  [BEAM-4276] Add combiner lifting support to Go SDK
 add 1c364fd  fixup! Address comments.
 new 5b9faa4  [BEAM-4276] Add combiner lifting support to Go SDK

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/combine.go|  11 +-
 sdks/go/pkg/beam/core/graph/edge.go|  27 ++-
 sdks/go/pkg/beam/core/runtime/exec/combine.go  | 166 ++
 sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 245 +++--
 .../pkg/beam/core/runtime/exec/fullvalue_test.go   |  26 +++
 sdks/go/pkg/beam/core/runtime/exec/translate.go|  45 ++--
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  |  65 +-
 sdks/go/pkg/beam/transforms/top/top.go |   6 +-
 8 files changed, 541 insertions(+), 50 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109196
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:46
Start Date: 05/Jun/18 20:46
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5507: [BEAM-4276] Add 
combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/combine.go b/sdks/go/pkg/beam/combine.go
index b841d916450..d4508c7379b 100644
--- a/sdks/go/pkg/beam/combine.go
+++ b/sdks/go/pkg/beam/combine.go
@@ -19,6 +19,7 @@ import (
"fmt"
 
"github.com/apache/beam/sdks/go/pkg/beam/core/graph"
+   "github.com/apache/beam/sdks/go/pkg/beam/core/typex"
 )
 
 // Combine inserts a global Combine transform into the pipeline. It
@@ -59,8 +60,16 @@ func TryCombinePerKey(s Scope, combinefn interface{}, col 
PCollection) (PCollect
if err != nil {
return PCollection{}, fmt.Errorf("invalid CombineFn: %v", err)
}
+   // This seems like the best place to infer the accumulator coder type, 
unless
+   // it's a universal type.
+   // We can get the fulltype from the return value of the 
mergeAccumulatorFn
+   // TODO(lostluck): 2018/05/28 Correctly infer universal type coder if 
necessary.
+   accumCoder, err := 
inferCoder(typex.New(fn.MergeAccumulatorsFn().Ret[0].T))
+   if err != nil {
+   return PCollection{}, fmt.Errorf("unable to infer CombineFn 
accumulator coder: %v", err)
+   }
 
-   edge, err := graph.NewCombine(s.real, s.scope, fn, col.n)
+   edge, err := graph.NewCombine(s.real, s.scope, fn, col.n, accumCoder)
if err != nil {
return PCollection{}, err
}
diff --git a/sdks/go/pkg/beam/core/graph/edge.go 
b/sdks/go/pkg/beam/core/graph/edge.go
index 4605470bab0..74eb94c26fa 100644
--- a/sdks/go/pkg/beam/core/graph/edge.go
+++ b/sdks/go/pkg/beam/core/graph/edge.go
@@ -136,21 +136,19 @@ type Payload struct {
Data []byte
 }
 
-// TODO(herohde) 5/24/2017: how should we represent/obtain the coder for 
Combine
-// accumulator types? Coder registry? Assume JSON?
-
 // MultiEdge represents a primitive data processing operation. Each non-user
 // code operation may be implemented by either the harness or the runner.
 type MultiEdge struct {
id int
parent *Scope
 
-   OpOpcode
-   DoFn  *DoFn  // ParDo
-   CombineFn *CombineFn // Combine
-   Value []byte // Impulse
-   Payload   *Payload   // External
-   WindowFn  *window.Fn // WindowInto
+   Op Opcode
+   DoFn   *DoFn// ParDo
+   CombineFn  *CombineFn   // Combine
+   AccumCoder *coder.Coder // Combine
+   Value  []byte   // Impulse
+   Payload*Payload // External
+   WindowFn   *window.Fn   // WindowInto
 
Input  []*Inbound
Output []*Outbound
@@ -317,9 +315,17 @@ func newDoFnNode(op Opcode, g *Graph, s *Scope, u *DoFn, 
in []*Node, typedefs ma
return edge, nil
 }
 
+// CombinePerKeyScope is the Go SDK canonical name for the combine composite
+// scope. With Beam Portability, "primitive" composite transforms like
+// combine have their URNs & payloads attached to a high level scope, with a
+// default representation beneath. The use of this const permits the
+// translation layer to confirm the SDK expects this combine to be liftable
+// by a runner and should set this scope's URN and Payload accordingly.
+const CombinePerKeyScope = "CombinePerKey"
+
 // NewCombine inserts a new Combine edge into the graph. Combines cannot have 
side
 // input.
-func NewCombine(g *Graph, s *Scope, u *CombineFn, in *Node) (*MultiEdge, 
error) {
+func NewCombine(g *Graph, s *Scope, u *CombineFn, in *Node, ac *coder.Coder) 
(*MultiEdge, error) {
inT := in.Type()
if !typex.IsCoGBK(inT) {
return nil, fmt.Errorf("combine requires CoGBK type: %v", inT)
@@ -380,6 +386,7 @@ func NewCombine(g *Graph, s *Scope, u *CombineFn, in *Node) 
(*MultiEdge, error)
edge := g.NewEdge(s)
edge.Op = Combine
edge.CombineFn = u
+   edge.AccumCoder = ac
edge.Input = []*Inbound{{Kind: kinds[0], From: in, Type: inbound[0]}}
for i := 0; i < len(out); i++ {
n := g.NewNode(out[i], in.WindowingStrategy(), in.Bounded())
diff --git a/sdks/go/pkg/beam/core/runtime/exec/combine.go 
b/sdks/go/pkg/beam/core/runtime/exec/combine.go
index

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109197=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109197
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:46
Start Date: 05/Jun/18 20:46
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5507: [BEAM-4276] Add 
combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#issuecomment-394854383
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109197)
Time Spent: 7h  (was: 6h 50m)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: [BEAM-4276] Add combiner lifting support to Go SDK

2018-06-05 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5b9faa416059299d0fe442b67dea5b3c1cb3d83b
Merge: 3c380a4 1c364fd
Author: Lukasz Cwik 
AuthorDate: Tue Jun 5 13:46:32 2018 -0700

[BEAM-4276] Add combiner lifting support to Go SDK

 sdks/go/pkg/beam/combine.go|  11 +-
 sdks/go/pkg/beam/core/graph/edge.go|  27 ++-
 sdks/go/pkg/beam/core/runtime/exec/combine.go  | 166 ++
 sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 245 +++--
 .../pkg/beam/core/runtime/exec/fullvalue_test.go   |  26 +++
 sdks/go/pkg/beam/core/runtime/exec/translate.go|  45 ++--
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  |  65 +-
 sdks/go/pkg/beam/transforms/top/top.go |   6 +-
 8 files changed, 541 insertions(+), 50 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Commented] (BEAM-4484) Shading model-pipeline / model-fn-execution / model-job-management produces corrupted classes

2018-06-05 Thread Luke Cwik (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502473#comment-16502473
 ] 

Luke Cwik commented on BEAM-4484:
-

This has been discussed a long time ago where the plan is to move to vendor 
specific shaded modules and use explicitly repackaged paths:

[https://lists.apache.org/thread.html/12383d2e5d70026427df43294e30d6524334e16f03d86c9a5860792f@%3Cdev.beam.apache.org%3E]

 

The short term solution of not shading/bundling beam-model-pipeline in 
direct-java is the short term option till the discussion on that mailing list 
comes to fruition.

> Shading model-pipeline / model-fn-execution / model-job-management produces 
> corrupted classes
> -
>
> Key: BEAM-4484
> URL: https://issues.apache.org/jira/browse/BEAM-4484
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>
> During investigation for test failures on 
> [https://github.com/apache/beam/pull/5445,] I discovered that the the post 
> shaded beam-model-pipeline proto filedescriptors were corrupted. It turns out 
> that during the shading process, an over eager string replacement inside a 
> class is corrupting an internal field, in this case it modifies the RunnerApi 
> file descriptor storing
> {code:java}
> org.apache.beam.model.pipeline.v1.AccumulationMode.Enum{code}
> and changing it to
> {code:java}
> org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.AccumulationMode.Enum{code}
> This problem exists because the proto package name and the java package name 
> collide and use org.apache.beam.model.pipeline.v1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4491) DockerJobBundleFactory's ProvisionInfo is empty.



 [ 
https://issues.apache.org/jira/browse/BEAM-4491?focusedWorklogId=109191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109191
 ]

ASF GitHub Bot logged work on BEAM-4491:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:33
Start Date: 05/Jun/18 20:33
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #5563: [BEAM-4491] 
Fix ProvisionInfo in DockerJobBundleFactory
URL: https://github.com/apache/beam/pull/5563
 
 
   DockerJobBundleFactory was previously setting up a provisioning
   service that served the default empty ProvisionInfo to workers.
   This change accepts JobInfo to the create call, so that it can
   serve the correct ProvisionInfo for a job.
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109191)
Time Spent: 10m
Remaining Estimate: 0h

> DockerJobBundleFactory's ProvisionInfo is empty.
> 
>
> Key: BEAM-4491
> URL: https://issues.apache.org/jira/browse/BEAM-4491
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> DockerJobBundleFactory creates a ProvisioningService that serves the default 
> empty ProvisionInfo, rather than the ProvisionInfo related to a job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4491) DockerJobBundleFactory's ProvisionInfo is empty.



 [ 
https://issues.apache.org/jira/browse/BEAM-4491?focusedWorklogId=109192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109192
 ]

ASF GitHub Bot logged work on BEAM-4491:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:33
Start Date: 05/Jun/18 20:33
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #5563: [BEAM-4491] Fix 
ProvisionInfo in DockerJobBundleFactory
URL: https://github.com/apache/beam/pull/5563#issuecomment-394850586
 
 
   R: @bsidhom 
   CC: @jkff @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109192)
Time Spent: 20m  (was: 10m)

> DockerJobBundleFactory's ProvisionInfo is empty.
> 
>
> Key: BEAM-4491
> URL: https://issues.apache.org/jira/browse/BEAM-4491
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Axel Magnuson
>Assignee: Axel Magnuson
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> DockerJobBundleFactory creates a ProvisioningService that serves the default 
> empty ProvisionInfo, rather than the ProvisionInfo related to a job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4491) DockerJobBundleFactory's ProvisionInfo is empty.

2018-06-05 Thread Axel Magnuson (JIRA)

Axel Magnuson created BEAM-4491:
---

 Summary: DockerJobBundleFactory's ProvisionInfo is empty.
 Key: BEAM-4491
 URL: https://issues.apache.org/jira/browse/BEAM-4491
 Project: Beam
  Issue Type: Bug
  Components: runner-core
Reporter: Axel Magnuson
Assignee: Axel Magnuson


DockerJobBundleFactory creates a ProvisioningService that serves the default 
empty ProvisionInfo, rather than the ProvisionInfo related to a job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4276) Implement the portable lifted Combiner transforms in Go SDK



 [ 
https://issues.apache.org/jira/browse/BEAM-4276?focusedWorklogId=109190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109190
 ]

ASF GitHub Bot logged work on BEAM-4276:


Author: ASF GitHub Bot
Created on: 05/Jun/18 20:29
Start Date: 05/Jun/18 20:29
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #5507: [BEAM-4276] Add 
combiner lifting support to Go SDK
URL: https://github.com/apache/beam/pull/5507#issuecomment-394849337
 
 
   R: @lukecwik since Jason is out today. Please merge! Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109190)
Time Spent: 6h 40m  (was: 6.5h)

> Implement the portable lifted Combiner transforms in Go SDK
> ---
>
> Key: BEAM-4276
> URL: https://issues.apache.org/jira/browse/BEAM-4276
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Specifically add the necessary code to produce a Combine Composite with the 
> correct URN, and permit the SDK harness to understand the lifted parts when 
> receiving a bundle plan from the worker.
> Not expected as part of this issue is:
> Additional performance tweaks to the in memory cache (See 
> [BEAM-4468|https://issues.apache.org/jira/browse/BEAM-4468])



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4490) Contributor's guide needs to document the flow of a Pull Request for new contributors

Pablo Estrada created BEAM-4490:
---

 Summary: Contributor's guide needs to document the flow of a Pull 
Request for new contributors
 Key: BEAM-4490
 URL: https://issues.apache.org/jira/browse/BEAM-4490
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Pablo Estrada
Assignee: Melissa Pashniak


Some assumptions and expectations can be made clear e.g. Precommit tests, 
tagging a reviewer, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4489) Contributor's guide needs to document expectations for tools to use with Python

Pablo Estrada created BEAM-4489:
---

 Summary: Contributor's guide needs to document expectations for 
tools to use with Python
 Key: BEAM-4489
 URL: https://issues.apache.org/jira/browse/BEAM-4489
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Ahmet Altay


Currently it mentions some python utilities, also gradle. We should improve 
this to be consistent and clear.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4488) Contributor's guide needs to document how to install Python SDK from source

Pablo Estrada created BEAM-4488:
---

 Summary: Contributor's guide needs to document how to install 
Python SDK from source
 Key: BEAM-4488
 URL: https://issues.apache.org/jira/browse/BEAM-4488
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Ahmet Altay


Currently it is not documented, though it does appear in the quickstart



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4487) Some errors related to GCP in pip install -e .[gcp]

Pablo Estrada created BEAM-4487:
---

 Summary: Some errors related to GCP in pip install -e .[gcp]
 Key: BEAM-4487
 URL: https://issues.apache.org/jira/browse/BEAM-4487
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Pablo Estrada
Assignee: Ahmet Altay


When installing Beam with GCP options from source, the following errors pop out:

https://pastebin.com/vrWWvF9y

The installation goes through, but the errors are confusing, and may confuse 
users and contributors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch pabloem-patch-1 updated (f926a15 -> 59e56eb)

2018-06-05 Thread pabloem

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch pabloem-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f926a15  Improve default value for experiments set in 
RuntimeValueProvider
 add 59e56eb  Adding unittest for default value of 
RuntimeValueProvider.experiments

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/options/value_provider_test.py | 2 ++
 1 file changed, 2 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
pabl...@apache.org.

[jira] [Resolved] (BEAM-4485) Incredibly difficult to use KakfaIO + TLS + DataflowRunner

2018-06-05 Thread Raghu Angadi (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi resolved BEAM-4485.

   Resolution: Not A Problem
Fix Version/s: Not applicable

> Incredibly difficult to use KakfaIO + TLS + DataflowRunner
> --
>
> Key: BEAM-4485
> URL: https://issues.apache.org/jira/browse/BEAM-4485
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Andre
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: Not applicable
>
>
> When attempting to use KafkaIO.Read with DataflowRunner, I have hit a lot of 
> walls. The brokers need to be accessible both locally and from the dataflow 
> runner instances. This means, when using TLS authentication, the 
> keystore/truststore files need to be available locally and on the instances. 
> I programmatically add the files to the pipeline options with
> {noformat}
> List filesToStage = 
> PipelineResources.detectClassPathResourcesToStage(IndicatorIngest.class.getClassLoader());
> filesToStage.add("trust.p12");
> filesToStage.add("server.p12");
> {noformat}
> but even when I do this, the remote file names are different. This means that 
> I need to determine the remote file name myself, like this
> {noformat}
> PackageAttributes.forFileToStage(new File(filepath), 
> filepath).getDestination().getName();
> {noformat}
> but that function is package private, so I need to wrap this call with a 
> custom class in org.apache.beam.runners.dataflow.util. When I calculate this 
> filename, I can use it to set the ssl..location, but this is the wrong 
> location locally, and it needs to be correct both locally and remotely. This 
> means in my main I need to calculate the local files remote names, copy them 
> to the local path with the same name, dynamically set the property to this 
> path, and programmatically add these files to be staged so they hopefully 
> have the same name on the worker. KafkaConsumer doesn't seem to provide any 
> other way to specify where to get these keys from.
> My question is, I am supposed to be jumping through all these hoops, or am I 
> doing something (or multiple things) completely wrong?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4485) Incredibly difficult to use KakfaIO + TLS + DataflowRunner

2018-06-05 Thread Raghu Angadi (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502336#comment-16502336
 ] 

Raghu Angadi commented on BEAM-4485:


Fortunately each of the two problems you mentioned have alternate ways:
 # access on local launcher machine : KafkaIO needs this in-order to fetch 
number of partitions for the topic. If you don't have access, then you can also 
provide list of partitons explicity to reader. See `withTopicPartitions()` api. 
Dataflow worker machine do need access.
  
 # Difficulty with providing key files : This was a limitation in KafkaConsumer 
configuration.  Setting ACLs in Kafka got much better in 0.10.2 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients]
 It allows setting these parameters in consumer config itself, which makes it 
simpler to use with Dataflow. KafkaIO allows setting pretty much any of the 
consumer config.

Does this help?

> Incredibly difficult to use KakfaIO + TLS + DataflowRunner
> --
>
> Key: BEAM-4485
> URL: https://issues.apache.org/jira/browse/BEAM-4485
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-kafka
>Reporter: Andre
>Assignee: Raghu Angadi
>Priority: Minor
> Fix For: Not applicable
>
>
> When attempting to use KafkaIO.Read with DataflowRunner, I have hit a lot of 
> walls. The brokers need to be accessible both locally and from the dataflow 
> runner instances. This means, when using TLS authentication, the 
> keystore/truststore files need to be available locally and on the instances. 
> I programmatically add the files to the pipeline options with
> {noformat}
> List filesToStage = 
> PipelineResources.detectClassPathResourcesToStage(IndicatorIngest.class.getClassLoader());
> filesToStage.add("trust.p12");
> filesToStage.add("server.p12");
> {noformat}
> but even when I do this, the remote file names are different. This means that 
> I need to determine the remote file name myself, like this
> {noformat}
> PackageAttributes.forFileToStage(new File(filepath), 
> filepath).getDestination().getName();
> {noformat}
> but that function is package private, so I need to wrap this call with a 
> custom class in org.apache.beam.runners.dataflow.util. When I calculate this 
> filename, I can use it to set the ssl..location, but this is the wrong 
> location locally, and it needs to be correct both locally and remotely. This 
> means in my main I need to calculate the local files remote names, copy them 
> to the local path with the same name, dynamically set the property to this 
> path, and programmatically add these files to be staged so they hopefully 
> have the same name on the worker. KafkaConsumer doesn't seem to provide any 
> other way to specify where to get these keys from.
> My question is, I am supposed to be jumping through all these hoops, or am I 
> doing something (or multiple things) completely wrong?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state



 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109175
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 05/Jun/18 19:07
Start Date: 05/Jun/18 19:07
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5445: [BEAM-2915] Add 
support for handling bag user state to the java-fn-execution library to support 
runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-394825200
 
 
   I thought I already did rebase to fix the `Components` issue.
   
   I can't merge because the tests will fail consistently until I solve/work 
around BEAM-4484


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109175)
Time Spent: 2h 10m  (was: 2h)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4486) BigQuery: FILE_LOADS + CREATE_NEVER + field-based partitioning => missing schema exception

2018-06-05 Thread Glenn Ammons (JIRA)

Glenn Ammons created BEAM-4486:
--

 Summary: BigQuery: FILE_LOADS + CREATE_NEVER + field-based 
partitioning => missing schema exception
 Key: BEAM-4486
 URL: https://issues.apache.org/jira/browse/BEAM-4486
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Affects Versions: 2.4.0
Reporter: Glenn Ammons
Assignee: Chamikara Jayalath


Our pipeline gets this error from BigQuery when using 
BigQueryIO.Write.Method.FILE_LOADS, 
BigQueryIO.Write.CreateDisposition.CREATE_NEVER, and field-based time 
partitioning (full exception at the bottom of this note):

    Table with field based partitioning must have a schema.

We do supply a schema when we create the pipeline by calling 
BigQuery.Write.withSchema, but this schema is ignored because the 
processElement method here:

[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java]

always provides a null schema when using CREATE_NEVER.

I would expect Beam to use the provided schema no matter what setting we are 
using for the CreateDisposition.

 

Full exception:

java.io.IOException: Unable to insert job: 
078646f70a664daaa1ed96832b233036_19e873cd24cf1968559515e49b3d868d_1_0-0,
 aborting after 9 . 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:236)
 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:204)
 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:144)
 org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:259) 
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:77) 
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:155)
 Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 
400 Bad Request \{ "code" : 400, "errors" : [ { "domain" : "global", "message" 
: "Table with field based partitioning must have a schema.", "reason" : 
"invalid" } ], "message" : "Table with field based partitioning must have a 
schema." } 

com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
 
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
 
com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
 
com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
 com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065) 
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
 
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
 
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:218)
 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startJob(BigQueryServicesImpl.java:204)
 
org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$JobServiceImpl.startLoadJob(BigQueryServicesImpl.java:144)
 org.apache.beam.sdk.io.gcp.bigquery.WriteTables.load(WriteTables.java:259) 
org.apache.beam.sdk.io.gcp.bigquery.WriteTables.access$600(WriteTables.java:77) 
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:155)
 
org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn$DoFnInvoker.invokeProcessElement(Unknown
 Source) 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
 
com.google.cloud.dataflow.worker.StreamingSideInputDoFnRunner.startBundle(StreamingSideInputDoFnRunner.java:60)
 
com.google.cloud.dataflow.worker.SimpleParDoFn.reallyStartBundle(SimpleParDoFn.java:300)
 
com.google.cloud.dataflow.worker.SimpleParDoFn.startBundle(SimpleParDoFn.java:226)
 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.start(ParDoOperation.java:35)
 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:67)
 
com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1197)
 
com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:137)
 
com.google.cloud.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:940)

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=109163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109163
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 05/Jun/18 18:32
Start Date: 05/Jun/18 18:32
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request #424: 
[BEAM-3255] Updating release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#discussion_r193176828
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -178,64 +231,78 @@ Javadoc to the Javadoc for other modules that Beam 
depends on.
   the version number has changed, download a new version of the corresponding
   `-docs/package-list` file.
 
-### Create a release branch
+### Create a release branch in apache/beam repository
 
-Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the code repository, and update version information on the original branch.
+Attention: Only committer has permission to create release branch in 
apache/beam.
+
+Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the Apache code repository, and update version information on the original 
branch.
 
 Check out the version of the codebase from which you start the release. For a 
new minor or major release, this may be `HEAD` of the `master` branch. To build 
a hotfix/incremental release, instead of the `master` branch, use the release 
tag of the release being patched. (Please make sure your cloned repository is 
up-to-date before starting.)
 
 git checkout 
 
+**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please 
check out the previous release tag, rather than the master branch.
 
 Set up a few environment variables to simplify Maven commands that follow. (We 
use `bash` Unix syntax in this guide.)
 
-VERSION="1.2.3"
-NEXT_VERSION="1.2.4"
-BRANCH_NAME="release-${VERSION}"
-DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
+RELEASE=2.5.0
+NEXT_VERSION_IN_BASE_BRANCH=2.6.0
+BRANCH=release-${RELEASE}
 
 Version represents the release currently underway, while next version 
specifies the anticipated next version to be released from that branch. 
Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4.
 
-Use Maven release plugin to create the release branch and update the current 
branch to use the new development version. This command applies for the new 
major or minor version. (Warning: this command automatically pushes changes to 
the code repository.)
+**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), 
please check out the previous release tag, before running the following 
instructions:
+
+BASE_RELEASE=2.5.0
+RELEASE=2.5.1
+NEXT_VERSION_IN_BASE_BRANCH=2.5.0
+git checkout tags/${BASE_RELEASE}
+
+Create a new branch, and update version files in the master branch.
 
-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DdevelopmentVersion=${DEVELOPMENT_VERSION}
+git branch ${BRANCH}
 
-However, if you are doing an incremental/hotfix release, please run the 
following command after checking out the release tag of the release being 
patched.
+# Now change the version in existing gradle files, and Python files
+sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g" 
build_rules.gradle
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" 
sdks/python/apache_beam/version.py
 
-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DupdateWorkingCopyVersions=false \
--DupdateBranchVersions=true \
--DreleaseVersion="${VERSION}-SNAPSHOT"
+# Save changes in master branch
+git add gradle.properties build_rules.gradle 
sdks/python/apache_beam/version.py
+git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master 
branch."
 
 Check out the release branch.
 
-git checkout ${BRANCH_NAME}
+git checkout ${BRANCH}
 
 Review comment:
   This step needs to be moved later, after changing versions on master?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109163)
Time Spent: 9h 10m  (was: 9h)

> Update release process to use Gradle
> 
>
>

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle

[
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=109160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109160
]

ASF GitHub Bot logged work on BEAM-3255:

Author: ASF GitHub Bot
Created on: 05/Jun/18 18:31
Start Date: 05/Jun/18 18:31
Worklog Time Spent: 10m
Work Description: alanmyrvold commented on a change in pull request #424:
[BEAM-3255] Updating release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#discussion_r193176561

##
File path: src/contribute/release-guide.md
##
@@ -178,64 +231,78 @@ Javadoc to the Javadoc for other modules that Beam
depends on.
the version number has changed, download a new version of the corresponding
`-docs/package-list` file.

-### Create a release branch
+### Create a release branch in apache/beam repository

-Release candidates are built from a release branch. As a final step in
preparation for the release, you should create the release branch, push it to
the code repository, and update version information on the original branch.
+Attention: Only committer has permission to create release branch in
apache/beam.
+
+Release candidates are built from a release branch. As a final step in
preparation for the release, you should create the release branch, push it to
the Apache code repository, and update version information on the original
branch.

Check out the version of the codebase from which you start the release. For a
new minor or major release, this may be `HEAD` of the `master` branch. To build
a hotfix/incremental release, instead of the `master` branch, use the release
tag of the release being patched. (Please make sure your cloned repository is
up-to-date before starting.)

git checkout

+**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please
check out the previous release tag, rather than the master branch.

Set up a few environment variables to simplify Maven commands that follow. (We
use `bash` Unix syntax in this guide.)

-VERSION="1.2.3"
-NEXT_VERSION="1.2.4"
-BRANCH_NAME="release-${VERSION}"
-DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
+RELEASE=2.5.0
+NEXT_VERSION_IN_BASE_BRANCH=2.6.0
+BRANCH=release-${RELEASE}

Version represents the release currently underway, while next version
specifies the anticipated next version to be released from that branch.
Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4.

-Use Maven release plugin to create the release branch and update the current
branch to use the new development version. This command applies for the new
major or minor version. (Warning: this command automatically pushes changes to
the code repository.)
+**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1),
please check out the previous release tag, before running the following
instructions:
+
+BASE_RELEASE=2.5.0
+RELEASE=2.5.1
+NEXT_VERSION_IN_BASE_BRANCH=2.5.0
+git checkout tags/${BASE_RELEASE}
+
+Create a new branch, and update version files in the master branch.

-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DdevelopmentVersion=${DEVELOPMENT_VERSION}
+git branch ${BRANCH}

-However, if you are doing an incremental/hotfix release, please run the
following command after checking out the release tag of the release being
patched.
+# Now change the version in existing gradle files, and Python files
+sed -i -e "s/'${RELEASE}'/'${NEXT_VERSION_IN_BASE_BRANCH}'/g"
build_rules.gradle
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g" gradle.properties
+sed -i -e "s/${RELEASE}/${NEXT_VERSION_IN_BASE_BRANCH}/g"
sdks/python/apache_beam/version.py

-mvn release:branch \
--DbranchName=${BRANCH_NAME} \
--DupdateWorkingCopyVersions=false \
--DupdateBranchVersions=true \
--DreleaseVersion="${VERSION}-SNAPSHOT"
+# Save changes in master branch
+git add gradle.properties build_rules.gradle
sdks/python/apache_beam/version.py
+git commit -m "Moving to ${NEXT_VERSION_IN_BASE_BRANCH}-SNAPSHOT on master
branch."

Check out the release branch.

-git checkout ${BRANCH_NAME}
+git checkout ${BRANCH}

The rest of this guide assumes that commands are run in the root of a
repository on `${BRANCH_NAME}` with the above environment variables set.

+### Update the beam version on master to the next dev version

Review comment:
This step should be done on master and is the purpose of the sed scripts
above?

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:

[jira] [Work logged] (BEAM-3255) Update release process to use Gradle



 [ 
https://issues.apache.org/jira/browse/BEAM-3255?focusedWorklogId=109159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109159
 ]

ASF GitHub Bot logged work on BEAM-3255:


Author: ASF GitHub Bot
Created on: 05/Jun/18 18:30
Start Date: 05/Jun/18 18:30
Worklog Time Spent: 10m 
  Work Description: alanmyrvold commented on a change in pull request #424: 
[BEAM-3255] Updating release guide to use Gradle commands
URL: https://github.com/apache/beam-site/pull/424#discussion_r193176202
 
 

 ##
 File path: src/contribute/release-guide.md
 ##
 @@ -178,64 +231,78 @@ Javadoc to the Javadoc for other modules that Beam 
depends on.
   the version number has changed, download a new version of the corresponding
   `-docs/package-list` file.
 
-### Create a release branch
+### Create a release branch in apache/beam repository
 
-Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the code repository, and update version information on the original branch.
+Attention: Only committer has permission to create release branch in 
apache/beam.
+
+Release candidates are built from a release branch. As a final step in 
preparation for the release, you should create the release branch, push it to 
the Apache code repository, and update version information on the original 
branch.
 
 Check out the version of the codebase from which you start the release. For a 
new minor or major release, this may be `HEAD` of the `master` branch. To build 
a hotfix/incremental release, instead of the `master` branch, use the release 
tag of the release being patched. (Please make sure your cloned repository is 
up-to-date before starting.)
 
 git checkout 
 
+**NOTE**: If you are doing an incremental/hotfix release (e.g. 2.5.1), please 
check out the previous release tag, rather than the master branch.
 
 Set up a few environment variables to simplify Maven commands that follow. (We 
use `bash` Unix syntax in this guide.)
 
-VERSION="1.2.3"
-NEXT_VERSION="1.2.4"
-BRANCH_NAME="release-${VERSION}"
-DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
+RELEASE=2.5.0
+NEXT_VERSION_IN_BASE_BRANCH=2.6.0
+BRANCH=release-${RELEASE}
 
 Version represents the release currently underway, while next version 
specifies the anticipated next version to be released from that branch. 
Normally, 1.2.0 is followed by 1.3.0, while 1.2.3 is followed by 1.2.4.
 
-Use Maven release plugin to create the release branch and update the current 
branch to use the new development version. This command applies for the new 
major or minor version. (Warning: this command automatically pushes changes to 
the code repository.)
+**NOTE**: Only if you are doing an incremental/hotfix release (e.g. 2.5.1), 
please check out the previous release tag, before running the following 
instructions:
+
+BASE_RELEASE=2.5.0
+RELEASE=2.5.1
+NEXT_VERSION_IN_BASE_BRANCH=2.5.0
 
 Review comment:
   NEXT_VERSION_IN_BASE_BRANCH=2.6.0


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109159)
Time Spent: 8h 50m  (was: 8h 40m)

> Update release process to use Gradle
> 
>
> Key: BEAM-3255
> URL: https://issues.apache.org/jira/browse/BEAM-3255
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Luke Cwik
>Assignee: Alan Myrvold
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>
> This task is about configuring Gradle to generate pom's and artifacts 
> required to perform a release and update the nightly release snapshot Jenkins 
> jobs found here 
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_beam_Release_NightlySnapshot.groovy
> We will also require some integration tests to run against the released 
> nightly snapshot artifacts to ensure that what was built is valid.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_JDBC #684

See

[jira] [Work logged] (BEAM-4481) Remove duplicate dependency declarations from runners/direct-java



 [ 
https://issues.apache.org/jira/browse/BEAM-4481?focusedWorklogId=109157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109157
 ]

ASF GitHub Bot logged work on BEAM-4481:


Author: ASF GitHub Bot
Created on: 05/Jun/18 18:25
Start Date: 05/Jun/18 18:25
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5560: [BEAM-4481] Remove 
duplicate definitions of dependencies in runners/direct-java
URL: https://github.com/apache/beam/pull/5560
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/direct-java/build.gradle b/runners/direct-java/build.gradle
index 1cb6f49709f..d095ae116f6 100644
--- a/runners/direct-java/build.gradle
+++ b/runners/direct-java/build.gradle
@@ -63,13 +63,7 @@ dependencies {
   dependOnProjects.each {
 compile project(path: it, configuration: "shadow")
   }
-  compile project(path: ":beam-model-pipeline", configuration: "shadow")
-  compile project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
-  compile project(path: ":beam-runners-core-java", configuration: "shadow")
-  compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
-  compile project(path: ":beam-runners-local-java-core", configuration: 
"shadow")
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
-  shadow project(path: ":beam-sdks-java-fn-execution", configuration: "shadow")
   shadow library.java.joda_time
   shadow library.java.findbugs_jsr305
   shadow library.java.slf4j_api


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109157)
Time Spent: 40m  (was: 0.5h)

> Remove duplicate dependency declarations from runners/direct-java
> -
>
> Key: BEAM-4481
> URL: https://issues.apache.org/jira/browse/BEAM-4481
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> beam-model-pipeline and others are duplicated in the dependency list



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (c1743cc -> 9d62103)

2018-06-05 Thread lcwik

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c1743cc  [BEAM-3926] Add new metrics protos based on "Defining and 
adding SDK Metrics" htt…
 add ced54ce  [BEAM-4481] Remove duplicate definitions of dependencies.
 new 9d62103  [BEAM-4481] Remove duplicate definitions of dependencies in 
runners/direct-java

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/direct-java/build.gradle | 6 --
 1 file changed, 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #267

See 


Changes:

[ajamato] Add new metrics protos based on s.apache.org/beam-fn-api-metrics

--
[...truncated 365.43 KB...]
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:77)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.226.50.1:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, 
caused by {java.net.SocketTimeoutException: connect timed out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getReadConnectionSource(ClusterBinding.java:63)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:89)
at 
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:84)
at 
com.mongodb.operation.CommandReadOperation.execute(CommandReadOperation.java:55)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:130)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:124)
at com.mongodb.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:114)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.split(MongoDbIO.java:332)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:275)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:197)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:181)
at 
com.google.cloud.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:160)
at 
com.google.cloud.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:77)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches ReadPreferenceServerSelector{readPreference=primary}. 
Client view of cluster state is {type=UNKNOWN, 
servers=[{address=35.226.50.1:27017, type=UNKNOWN, state=CONNECTING, 
exception={com.mongodb.MongoSocketOpenException:

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #354

See

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #265

See

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state



 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109152
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 05/Jun/18 18:11
Start Date: 05/Jun/18 18:11
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #5445: [BEAM-2915] Add 
support for handling bag user state to the java-fn-execution library to support 
runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-394807534
 
 
   Thanks, can you rebase this PR to fix the compile error? Since the shading 
issue is pre-existing in master, I think we can merge this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109152)
Time Spent: 2h  (was: 1h 50m)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4485) Incredibly difficult to use KakfaIO + TLS + DataflowRunner

2018-06-05 Thread Andre (JIRA)

Andre created BEAM-4485:
---

 Summary: Incredibly difficult to use KakfaIO + TLS + DataflowRunner
 Key: BEAM-4485
 URL: https://issues.apache.org/jira/browse/BEAM-4485
 Project: Beam
  Issue Type: Improvement
  Components: io-java-kafka
Reporter: Andre
Assignee: Raghu Angadi


When attempting to use KafkaIO.Read with DataflowRunner, I have hit a lot of 
walls. The brokers need to be accessible both locally and from the dataflow 
runner instances. This means, when using TLS authentication, the 
keystore/truststore files need to be available locally and on the instances. I 
programmatically add the files to the pipeline options with

{noformat}
List filesToStage = 
PipelineResources.detectClassPathResourcesToStage(IndicatorIngest.class.getClassLoader());
filesToStage.add("trust.p12");
filesToStage.add("server.p12");
{noformat}

but even when I do this, the remote file names are different. This means that I 
need to determine the remote file name myself, like this

{noformat}
PackageAttributes.forFileToStage(new File(filepath), 
filepath).getDestination().getName();
{noformat}

but that function is package private, so I need to wrap this call with a custom 
class in org.apache.beam.runners.dataflow.util. When I calculate this filename, 
I can use it to set the ssl..location, but this is the wrong location 
locally, and it needs to be correct both locally and remotely. This means in my 
main I need to calculate the local files remote names, copy them to the local 
path with the same name, dynamically set the property to this path, and 
programmatically add these files to be staged so they hopefully have the same 
name on the worker. KafkaConsumer doesn't seem to provide any other way to 
specify where to get these keys from.

My question is, I am supposed to be jumping through all these hoops, or am I 
doing something (or multiple things) completely wrong?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4484) Shading model-pipeline / model-fn-execution / model-job-management produces corrupted classes

2018-06-05 Thread Luke Cwik (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502211#comment-16502211
 ] 

Luke Cwik commented on BEAM-4484:
-

Options are:

1) Stop shading

2) Change package paths (either java or proto)

3) Try and find an option to disable this in the shade plugin

> Shading model-pipeline / model-fn-execution / model-job-management produces 
> corrupted classes
> -
>
> Key: BEAM-4484
> URL: https://issues.apache.org/jira/browse/BEAM-4484
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>
> During investigation for test failures on 
> [https://github.com/apache/beam/pull/5445,] I discovered that the the post 
> shaded beam-model-pipeline proto filedescriptors were corrupted. It turns out 
> that during the shading process, an over eager string replacement inside a 
> class is corrupting an internal field, in this case it modifies the RunnerApi 
> file descriptor storing
> {code:java}
> org.apache.beam.model.pipeline.v1.AccumulationMode.Enum{code}
> and changing it to
> {code:java}
> org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.AccumulationMode.Enum{code}
> This problem exists because the proto package name and the java package name 
> collide and use org.apache.beam.model.pipeline.v1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c46fc1ea2bf04ba8f22d439bea53c91fc34b2264
Author: Mergebot 
AuthorDate: Tue Jun 5 11:03:53 2018 -0700

Prepare repository for deployment.
---
 content/contribute/become-a-committer/index.html   |   1 +
 content/contribute/committer-guide/index.html  |   1 +
 content/contribute/design-documents/index.html | 354 +
 content/contribute/docker-images/index.html|   1 +
 content/contribute/eclipse/index.html  |   1 +
 content/contribute/index.html  |   1 +
 content/contribute/intellij/index.html |   1 +
 content/contribute/portability/index.html  |   1 +
 .../contribute/ptransform-style-guide/index.html   |   1 +
 content/contribute/release-guide/index.html|   1 +
 content/contribute/runner-guide/index.html |   1 +
 content/contribute/testing/index.html  |   1 +
 .../contribute/website-contributions/index.html|   1 +
 13 files changed, 366 insertions(+)

diff --git a/content/contribute/become-a-committer/index.html 
b/content/contribute/become-a-committer/index.html
index 5f8c08d..3d17119 100644
--- a/content/contribute/become-a-committer/index.html
+++ b/content/contribute/become-a-committer/index.html
@@ -107,6 +107,7 @@
 Runner authoring guide
 Portability Framework
 Docker images
+Design documents
   
 
 
diff --git a/content/contribute/committer-guide/index.html 
b/content/contribute/committer-guide/index.html
index 699d7e6..1206e88 100644
--- a/content/contribute/committer-guide/index.html
+++ b/content/contribute/committer-guide/index.html
@@ -107,6 +107,7 @@
 Runner authoring guide
 Portability Framework
 Docker images
+Design documents
   
 
 
diff --git a/content/contribute/design-documents/index.html 
b/content/contribute/design-documents/index.html
new file mode 100644
index 000..669fa3e
--- /dev/null
+++ b/content/contribute/design-documents/index.html
@@ -0,0 +1,354 @@
+
+
+  
+  
+  
+  
+  Beam Design Documents
+  
+  https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  
+  
+  
+  https://beam.apache.org/contribute/design-documents/; data-proofer-ignore>
+  
+  https://beam.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+  
+
+
+  
+
+
+  
+Toggle navigation
+
+
+
+  
+
+  
+
+  
+
+
+
+
+
+  
+
+  Get Started
+
+
+  Documentation
+
+
+  SDKS
+
+
+  RUNNERS
+
+
+  Contribute
+
+
+  Community
+
+Blog
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:20px;">
+  
+http://www.apache.org/;>ASF Homepage
+http://www.apache.org/licenses/;>License
+http://www.apache.org/security/;>Security
+http://www.apache.org/foundation/thanks.html;>Thanks
+http://www.apache.org/foundation/sponsorship.html;>Sponsorship
+https://www.apache.org/foundation/policies/conduct;>Code of 
Conduct
+  
+
+  
+
+
+
+
+  
+
+
+  
+Contribute
+Get started contributing
+
+  IDE tips
+
+  
+IntelliJ Tips
+Eclipse Tips
+  
+
+
+  Technical Docs
+
+  
+Testing guide
+PTransform style 
guide
+Runner authoring guide
+Portability Framework
+Docker images
+Design documents
+  
+
+
+  Committers
+
+  
+Become a committer
+Committer guide
+Release guide
+  
+
+
+  
+
+  
+
+  
+
+
+
+  Documents by category
+
+  Project Incubation (2016)
+  Beam Model
+  IO / Filesystem
+  Metrics
+  Runners
+  SQL / Schema
+  Portability
+  Build / Testing
+  Python
+  Go
+
+  
+  Other
+
+
+
+  
+
+  
+Design Documents
+This is a collection of documents that may or may not be up to date.
+
+Documents by category
+Project Incubation (2016)
+
+  Technical Vision [https://docs.google.com/document/d/1UyAeugHxZmVlQ5cEWo_eOPgXNQA1oD-rGooWOSwAqh8/edit;>doc],

[beam-site] branch asf-site updated (69d9535 -> c46fc1e)

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 69d9535  Prepare repository for deployment.
 add 0c182de  Added design docs page
 add 63956c9  Added first intro sentence.
 add c5f4e16  Moved one doc to other section and added several new docs
 add 48a5136  Added more docs
 add 506db5d  This closes #456
 new c46fc1e  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/become-a-committer/index.html   |   1 +
 content/contribute/committer-guide/index.html  |   1 +
 content/contribute/design-documents/index.html | 354 +
 content/contribute/docker-images/index.html|   1 +
 content/contribute/eclipse/index.html  |   1 +
 content/contribute/index.html  |   1 +
 content/contribute/intellij/index.html |   1 +
 content/contribute/portability/index.html  |   1 +
 .../contribute/ptransform-style-guide/index.html   |   1 +
 content/contribute/release-guide/index.html|   1 +
 content/contribute/runner-guide/index.html |   1 +
 content/contribute/testing/index.html  |   1 +
 .../contribute/website-contributions/index.html|   1 +
 src/_includes/section-menu/contribute.html |   1 +
 src/contribute/design-documents.md | 126 
 15 files changed, 493 insertions(+)
 create mode 100644 content/contribute/design-documents/index.html
 create mode 100644 src/contribute/design-documents.md

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #257

See 


Changes:

[ajamato] Add new metrics protos based on s.apache.org/beam-fn-api-metrics

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision c1743ccae68a57b46cff3bb13441fb2fbc55e511 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f c1743ccae68a57b46cff3bb13441fb2fbc55e511
Commit message: "[BEAM-3926] Add new metrics protos based on "Defining and 
adding SDK Metrics" htt…"
 > git rev-list --no-walk 73cc33292dee500c0e9f1a072f4d516f753f8e9d # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3204586369745478553.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2761906552548556593.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5547393046214228792.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-257
Error from server (AlreadyExists): namespaces "filebasedioithdfs-257" already 
exists
Build step 'Execute shell' marked build as failure

[jira] [Work logged] (BEAM-2915) Java SDK support for portable user state



 [ 
https://issues.apache.org/jira/browse/BEAM-2915?focusedWorklogId=109149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-109149
 ]

ASF GitHub Bot logged work on BEAM-2915:


Author: ASF GitHub Bot
Created on: 05/Jun/18 18:02
Start Date: 05/Jun/18 18:02
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #5445: [BEAM-2915] Add 
support for handling bag user state to the java-fn-execution library to support 
runner integration.
URL: https://github.com/apache/beam/pull/5445#issuecomment-394804785
 
 
   Filed BEAM-4484 for the shading issue. Will investigate to see if there is a 
way to keep shading beam-model-pipeline in the direct runner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 109149)
Time Spent: 1h 50m  (was: 1h 40m)

> Java SDK support for portable user state
> 
>
> Key: BEAM-2915
> URL: https://issues.apache.org/jira/browse/BEAM-2915
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Henning Rohde
>Assignee: Luke Cwik
>Priority: Minor
>  Labels: portability
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4484) Shading model-pipeline / model-fn-execution / model-job-management produces corrupted classes

2018-06-05 Thread Luke Cwik (JIRA)

Luke Cwik created BEAM-4484:
---

 Summary: Shading model-pipeline / model-fn-execution / 
model-job-management produces corrupted classes
 Key: BEAM-4484
 URL: https://issues.apache.org/jira/browse/BEAM-4484
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Luke Cwik
Assignee: Luke Cwik


During investigation for test failures on 
[https://github.com/apache/beam/pull/5445,] I discovered that the the post 
shaded beam-model-pipeline proto filedescriptors were corrupted. It turns out 
that during the shading process, an over eager string replacement inside a 
class is corrupting an internal field, in this case it modifies the RunnerApi 
file descriptor storing
{code:java}
org.apache.beam.model.pipeline.v1.AccumulationMode.Enum{code}
and changing it to
{code:java}
org.apache.beam.repackaged.beam_runners_direct_java.model.pipeline.v1.AccumulationMode.Enum{code}
This problem exists because the proto package name and the java package name 
collide and use org.apache.beam.model.pipeline.v1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 03/05: Moved one doc to other section and added several new docs

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c5f4e16e8608e5497663d0dd0086d78e0237ebb8
Author: Alexey Romanenko 
AuthorDate: Fri Jun 1 10:33:50 2018 +0200

Moved one doc to other section and added several new docs
---
 src/contribute/design-documents.md | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/src/contribute/design-documents.md 
b/src/contribute/design-documents.md
index d021eef..5ed7a13 100644
--- a/src/contribute/design-documents.md
+++ b/src/contribute/design-documents.md
@@ -23,7 +23,6 @@ This is a collection of documents that may or may not be up 
to date.
   - Reimplementing Beam API classes on top of Splittable DoFn on top of Source 
API [[doc](https://s.apache.org/sdf-via-source)]
   - New TextIO features based on SDF [[doc](http://s.apache.org/textio-sdf)]
   - Watch transform [[doc](http://s.apache.org/beam-watch-transform)]
-  - Technical contract for runtime rebalancing of work using Splittable DoFn 
[[doc](https://s.apache.org/beam-breaking-fusion)]
 - State and Timers for DoFn [[doc](https://s.apache.org/beam-state)]
 - ContextFn [[doc](http://s.apache.org/context-fn)]
 - Static Display Data 
[[doc](https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM)]
@@ -60,6 +59,7 @@ This is a collection of documents that may or may not be up 
to date.
   - Progress [[doc](https://s.apache.org/beam-fn-api-progress-reporting)]
   - Graphical view of progress 
[[doc](https://docs.google.com/document/d/1Dx18qBTvFWNqwLeecemOpKfleKzFyeV3Qwh71SHATvY)]
   - Fn State API and Bundle Processing 
[[doc](https://s.apache.org/beam-fn-state-api-and-bundle-processing)]
+  - Checkpointing and splitting of Beam bundles over the Fn API, with 
application to SDF [[doc](https://s.apache.org/beam-breaking-fusion)]
   - How to send and receive data 
[[doc](https://s.apache.org/beam-fn-api-send-and-receive-data)]
   - Defining and adding SDK Metrics 
[[doc](https://s.apache.org/beam-fn-api-metrics)]
   - SDK harness container contract 
[[doc](https://s.apache.org/beam-fn-api-container-contract)]
@@ -67,6 +67,9 @@ This is a collection of documents that may or may not be up 
to date.
 - Cross-language Beam Pipelines 
[[doc](https://s.apache.org/beam-mixed-language-pipelines)]
 - SDK X with Runner Y using Runner API 
[[doc](https://s.apache.org/beam-job-api)]
 - Flink Portable Runner Overview 
[[doc](https://s.apache.org/portable-flink-runner-overview)]
+- Launching portable pipeline on Flink Runner 
[[doc](https://docs.google.com/document/d/1xOaEEJrMmiSHprd-WiYABegfT129qqF-idUBINjxz8s)]
+- Portability support 
[[table](https://docs.google.com/spreadsheets/d/1KDa_FGn1ShjomGd-UUDOhuh2q73de2tPz6BqHpzqvNI)]
+- Portability Prototype [[doc](https://s.apache.org/beam-portability-team-doc)]
 
 ### Testing
 - More Expressive PAsserts 
[[doc](https://docs.google.com/document/d/1fZUUbG2LxBtqCVabQshldXIhkMcXepsbv2vuuny8Ix4)]
@@ -76,6 +79,7 @@ This is a collection of documents that may or may not be up 
to date.
 
 ### Python
 - Beam Python User State and Timer APIs 
[[doc](https://s.apache.org/beam-python-user-state-and-timers)]
+- Python Kafka connector 
[[doc](https://docs.google.com/document/d/1ogRS-e-HYYTHsXi_l2zDUUOnvfzEbub3BFkPrYIOawU)]
 
 ### Go
 - Apache Beam Go SDK design 
[[doc](https://s.apache.org/beam-go-sdk-design-rfc)]

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 05/05: This closes #456

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 506db5d48640cc9b0875f6d899be3fbb4e0e2703
Merge: 69d9535 48a5136
Author: Mergebot 
AuthorDate: Tue Jun 5 11:00:50 2018 -0700

This closes #456

 src/_includes/section-menu/contribute.html |   1 +
 src/contribute/design-documents.md | 126 +
 2 files changed, 127 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/05: Added first intro sentence.