Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #395

[Apache Jenkins Server]

See 


--
[...truncated 19.22 MB...]
May 15, 2018 5:23:41 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-14_22_23_40-14308107025382129962
May 15, 2018 5:23:41 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-14_22_23_40-14308107025382129962 with 0 
expected assertions.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:40.628Z: Autoscaling is enabled for job 
2018-05-14_22_23_40-14308107025382129962. The number of workers will be between 
1 and 1000.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:40.659Z: Autoscaling was automatically enabled for 
job 2018-05-14_22_23_40-14308107025382129962.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:43.399Z: Checking required Cloud APIs are enabled.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:43.544Z: Checking permissions granted to controller 
Service Account.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:47.127Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:47.551Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:47.814Z: Expanding GroupByKey operations into 
optimizable parts.
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:47.860Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.163Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.200Z: Elided trivial flatten 
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.247Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.291Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.329Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.367Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.410Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 15, 2018 5:23:57 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T05:23:48.456Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 520d1aec5ea4a0ff40a381296c4e247fc52b27bf
Author: Mergebot 
AuthorDate: Mon May 14 21:56:01 2018 -0700

Prepare repository for deployment.
---
 .../05/27/where-is-my-pcollection-dot-map.html |   5 +-
 content/contribute/become-a-committer/index.html   |  34 +-
 content/contribute/committer-guide/index.html  |  34 +-
 content/contribute/contribution-guide/index.html   | 652 +
 content/contribute/design-principles/index.html| 306 +-
 content/contribute/docker-images/index.html|  34 +-
 content/contribute/eclipse/index.html  |  68 +--
 content/contribute/feature-branches/index.html | 309 +-
 content/contribute/index.html  | 261 +++--
 content/contribute/intellij/index.html |  34 +-
 content/contribute/portability/index.html  |  34 +-
 .../contribute/ptransform-style-guide/index.html   |  34 +-
 content/contribute/release-guide/index.html|  34 +-
 content/contribute/runner-guide/index.html |  34 +-
 content/contribute/source-repository/index.html| 246 +---
 content/contribute/testing/index.html  |  34 +-
 .../contribute/website-contributions/index.html|  34 +-
 content/contribute/work-in-progress/index.html | 280 +
 content/contribution-guide/index.html  |   8 +-
 .../{contribute.html => contribute/index.html} |   0
 content/source_repository/index.html   |  10 -
 21 files changed, 368 insertions(+), 2117 deletions(-)

diff --git a/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html 
b/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
index fd8fcd7..d896c58 100644
--- a/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
+++ b/content/blog/2016/05/27/where-is-my-pcollection-dot-map.html
@@ -98,7 +98,10 @@
 
 
 
-Though Beam is relatively new, its design draws heavily on many years of 
experience with real-world pipelines. One of the primary inspirations is http://research.google.com/pubs/pub35650.html;>FlumeJava, which is 
Google’s internal successor to MapReduce first introduced in 2009.
+Though Beam is relatively new, its design draws heavily on many years of
+experience with real-world pipelines. One of the primary inspirations is
+https://ai.google/research/pubs/pub35650;>FlumeJava, which is 
Google’s
+internal successor to MapReduce first introduced in 2009.
 
 The original FlumeJava API has methods like count and parallelDo on the PCollections. Though 
slightly more succinct, this approach has many disadvantages to extensibility. 
Every new user to FlumeJava wanted to add transforms, and adding them as 
methods to PCollection simply doesn’t scale well. In contrast, a PCollection in 
Beam has a single apply method which 
takes any  [...]
 
diff --git a/content/contribute/become-a-committer/index.html 
b/content/contribute/become-a-committer/index.html
index 45408f9..e0cfe9c 100644
--- a/content/contribute/become-a-committer/index.html
+++ b/content/contribute/become-a-committer/index.html
@@ -89,24 +89,7 @@
 
   
 Contribute
-Get started contributing
-
-  How to contribute
-
-  
-Contribution guide
-Website 
contributions
-Feature branches
-  
-
-
-  Committers
-
-  
-Become a committer
-Committer guide
-  
-
+Get started contributing
 
   IDE tips
 
@@ -116,24 +99,23 @@
   
 
 
-  Other Guides
+  Technical Docs
 
   
 Testing guide
-Release guide
 PTransform style 
guide
 Runner authoring guide
+Portability Framework
+Docker images
   
 
 
-  Technical References
+  Committers
 
   
-Design principles
-Portability Framework
-Ongoing projects
-Source repository
-Docker images
+Become a committer
+Committer guide
+Release guide
   
 
 
diff --git a/content/contribute/committer-guide/index.html 
b/content/contribute/committer-guide/index.html
index b5ab0eb..699d7e6 100644
--- a/content/contribute/committer-guide/index.html
+++ b/content/contribute/committer-guide/index.html
@@ -89,24 +89,7 @@
 
   
 Contribute
-Get started contributing
-
-  How to contribute
-
-  
-Contribution guide
-Website 
contributions
-Feature branches
-  
-
-
-  Committers
-
-  
-Become a committer
-Committer guide
-  
-
+Get started contributing
 
   IDE tips
 
@@ -116,24 +99,23 @@
   
 
 
-  Other Guides
+  Technical Docs
 
   
 Testing guide
-Release guide
 PTransform style 
guide
 Runner authoring guide
+Portability Framework
+Docker images
   
 
 
-  Technical References
+  Committers
 
   
-Design principles
-Portability Framework
-Ongoing projects
-Source repository
-Docker images
+Become a 

[beam-site] branch asf-site updated (7a34444 -> 520d1ae)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 7a3  Merge #436: Add Community top-level section
 add 67b9a07  Move Release Guide to committers section
 add 333740a  Move committer section to the bottom (we can find it :-)
 add a4f2e5c  Streamline the contribution section
 add 4284b6a  Google has scrapped research, now only AI
 add 25a50fe  Temporarily suppress validation of ai.google links
 add d0d7c53  This closes #438
 new 520d1ae  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 Rakefile   |   6 +-
 .../05/27/where-is-my-pcollection-dot-map.html |   5 +-
 content/contribute/become-a-committer/index.html   |  34 +-
 content/contribute/committer-guide/index.html  |  34 +-
 content/contribute/contribution-guide/index.html   | 652 +
 content/contribute/design-principles/index.html| 306 +-
 content/contribute/docker-images/index.html|  34 +-
 content/contribute/eclipse/index.html  |  68 +--
 content/contribute/feature-branches/index.html | 309 +-
 content/contribute/index.html  | 261 +++--
 content/contribute/intellij/index.html |  34 +-
 content/contribute/portability/index.html  |  34 +-
 .../contribute/ptransform-style-guide/index.html   |  34 +-
 content/contribute/release-guide/index.html|  34 +-
 content/contribute/runner-guide/index.html |  34 +-
 content/contribute/source-repository/index.html| 246 +---
 content/contribute/testing/index.html  |  34 +-
 .../contribute/website-contributions/index.html|  34 +-
 content/contribute/work-in-progress/index.html | 280 +
 content/contribution-guide/index.html  |   8 +-
 .../{contribute.html => contribute/index.html} |   0
 content/source_repository/index.html   |  10 -
 src/_includes/section-menu/contribute.html |  34 +-
 .../2016-05-20-where-is-my-pcollection-dot-map.md  |   5 +-
 src/contribute/contribution-guide.md   | 387 
 src/contribute/design-principles.md|  53 --
 src/contribute/eclipse.md  |  18 +-
 src/contribute/index.md| 187 +-
 src/contribute/source-repository.md|  29 -
 src/contribute/work-in-progress.md |  33 --
 30 files changed, 563 insertions(+), 2674 deletions(-)
 rename content/docs/{contribute.html => contribute/index.html} (100%)
 delete mode 100644 content/source_repository/index.html
 delete mode 100644 src/contribute/contribution-guide.md
 delete mode 100644 src/contribute/design-principles.md
 delete mode 100644 src/contribute/source-repository.md
 delete mode 100644 src/contribute/work-in-progress.md

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (21a7749 -> d0d7c53)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 21a7749  This closes #438
 new d0d7c53  This closes #438

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (21a7749)
\
 N -- N -- N   refs/heads/mergebot (d0d7c53)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: This closes #438

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d0d7c5345b37b7ccfc9f11fd133dd1d9e576fa67
Merge: 7a3 25a50fe
Author: Mergebot 
AuthorDate: Mon May 14 21:53:35 2018 -0700

This closes #438

 Rakefile   |   6 +-
 src/_includes/section-menu/contribute.html |  34 +-
 .../2016-05-20-where-is-my-pcollection-dot-map.md  |   5 +-
 src/contribute/contribution-guide.md   | 387 -
 src/contribute/design-principles.md|  53 ---
 src/contribute/eclipse.md  |  18 +-
 src/contribute/index.md| 187 +-
 src/contribute/source-repository.md|  29 --
 src/contribute/work-in-progress.md |  33 --
 9 files changed, 195 insertions(+), 557 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=102001=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102001
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 15/May/18 04:50
Start Date: 15/May/18 04:50
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-389041955
 
 
   This one is probably not wired up yet, I only added the targets to SQL build 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102001)
Time Spent: 2h 50m  (was: 2h 40m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 05/06: Temporarily suppress validation of ai.google links

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 25a50fe5537c56e1a58fdfe24ce274266fe2519f
Author: Kenneth Knowles 
AuthorDate: Mon May 14 15:17:50 2018 -0700

Temporarily suppress validation of ai.google links
---
 Rakefile | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/Rakefile b/Rakefile
index 3398fe9..c2e7069 100644
--- a/Rakefile
+++ b/Rakefile
@@ -12,7 +12,11 @@ task :test do
 :allow_hash_href => true,
 :check_html => true,
 :file_ignore => [/javadoc/, /v2/, /pydoc/],
-:url_ignore => [/jstorm.io/, /datatorrent.com/],
+:url_ignore => [
+/jstorm.io/,
+/datatorrent.com/,
+/ai.google/ # https://issues.apache.org/jira/browse/INFRA-16527
+],
 :parallel => { :in_processes => Etc.nprocessors },
 }).run
 end

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 03/06: Streamline the contribution section

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit a4f2e5cad54e5787e2796bf9df091ac0adb56f74
Author: Kenneth Knowles 
AuthorDate: Fri May 11 15:53:15 2018 -0700

Streamline the contribution section
---
 src/_includes/section-menu/contribute.html |  22 +-
 src/contribute/contribution-guide.md   | 387 -
 src/contribute/design-principles.md|  53 
 src/contribute/eclipse.md  |  18 +-
 src/contribute/index.md| 187 --
 src/contribute/source-repository.md|  29 ---
 src/contribute/work-in-progress.md |  33 ---
 7 files changed, 180 insertions(+), 549 deletions(-)

diff --git a/src/_includes/section-menu/contribute.html 
b/src/_includes/section-menu/contribute.html
index a9bea28..b1836c7 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -1,14 +1,5 @@
 Contribute
-Get started contributing
-
-  How to contribute
-
-  
-Contribution guide
-Website 
contributions
-Feature 
branches
-  
-
+Get started contributing
 
   IDE tips
 
@@ -18,22 +9,13 @@
   
 
 
-  Other Guides
+  Technical Docs
 
   
 Testing guide
 PTransform style guide
 Runner authoring 
guide
-  
-
-
-  Technical References
-
-  
-Design 
principles
 Portability 
Framework
-Ongoing 
projects
-Source 
repository
 Docker 
images
   
 
diff --git a/src/contribute/contribution-guide.md 
b/src/contribute/contribution-guide.md
deleted file mode 100644
index fa08ba7..000
--- a/src/contribute/contribution-guide.md
+++ /dev/null
@@ -1,387 +0,0 @@

-layout: section
-title: "Beam Contribution Guide"
-permalink: /contribute/contribution-guide/
-section_menu: section-menu/contribute.html
-redirect_from: /contribution-guide/

-
-# Apache Beam Contribution Guide
-
-The Apache Beam community welcomes contributions from anyone!
-
-There are lots of opportunities:
-
- - write new examples
- - improve the documentation
- - add new user-facing libraries (new statistical libraries, new IO connectors,
-   etc)
- - improve your favorite language SDK (Java, Python, Go, etc)
- - improve specific runners (Apache Apex, Apache Flink, Apache Spark, Google
-   Cloud Dataflow, etc)
- - work on the core programming model (what is a Beam pipeline and how does it
-   run?)
-
-Most importantly, if you have an idea of how to contribute, then do it! 
-
-For a list of open starter tasks, check
-[https://s.apache.org/beam-starter-tasks](https://s.apache.org/beam-starter-tasks).
-
-What follows is a more comprehensive document covering details of things that
-may come up.
-
-## Review-then-commit workflow
-
-We use a review-then-commit workflow in Beam.
-
-![The Beam contribution workflow has 5 steps: engage, design, code, review, 
and commit.](
-  {{ "/images/contribution-guide-1.png" | prepend: site.baseurl }})
-
-**For larger contributions or those that affect multiple components:**
-
-1. **Engage**: We encourage you to work with the Beam community on the [Apache
-   JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and
-   [developer’s mailing list]({{ site.baseurl }}/get-started/support/) to 
identify
-   good areas for contribution.
-2. **Design:** More complicated contributions will likely benefit from some
-   early discussion in order to scope and design them well.
-
-**For all contributions:**
-
-1. **Code:** The best part ;-)
-2. **Review:** Submit a pull request with your contribution to our [GitHub
-   mirror](https://github.com/apache/beam/). Work with a committer to review
-and iterate on the code, if needed.
-3. **Commit:** A Beam committer merges the pull request into our [Apache
-   repository](https://gitbox.apache.org/repos/asf/beam.git).
-
-We look forward to working with you!
-
-## Engage
-
-### Mailing list(s)
-
-We discuss design and implementation issues on the `d...@beam.apache.org`
-mailing list, which is archived
-[here](https://lists.apache.org/list.html?d...@beam.apache.org). Join by
-emailing
-[`dev-subscr...@beam.apache.org`](mailto:dev-subscr...@beam.apache.org).
-Anyone is welcome to join, and we'd love to hear what you're interested in.
-
-If interested, you can also join the other [mailing lists]({{ site.baseurl
-}}/get-started/support/).
-
-### JIRA issue tracker
-
-We use the Apache Software Foundation's
-[JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue tracking and
-project management tool.
-
-If you do not already have an Apache JIRA account, sign up
-[here](https://issues.apache.org/jira/).
-
-If a quick
-[search](https://issues.apache.org/jira/issues/?jql=project%3DBEAM%20AND%20text%20~%20%22the%20thing%20I%20want%20to%20contribute%22)
-doesn’t turn up an existing JIRA issue for the work you want to contribute,
-feel free to create one.
-
-If there’s 

[beam-site] 01/06: Move Release Guide to committers section

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 67b9a07da07c4e0f256c4441baa68ea3efe8e409
Author: Kenneth Knowles 
AuthorDate: Fri May 11 14:47:07 2018 -0700

Move Release Guide to committers section
---
 src/_includes/section-menu/contribute.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/_includes/section-menu/contribute.html 
b/src/_includes/section-menu/contribute.html
index e07f004..bf74612 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -15,6 +15,7 @@
   
 Become a 
committer
 Committer 
guide
+Release 
guide
   
 
 
@@ -30,7 +31,6 @@
 
   
 Testing guide
-Release 
guide
 PTransform style guide
 Runner authoring 
guide
   

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 06/06: This closes #438

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 21a7749f8a4d0633a4860fa722d06791e8740228
Merge: 7a3 25a50fe
Author: Mergebot 
AuthorDate: Mon May 14 21:48:47 2018 -0700

This closes #438

 Rakefile   |   6 +-
 src/_includes/section-menu/contribute.html |  34 +-
 .../2016-05-20-where-is-my-pcollection-dot-map.md  |   5 +-
 src/contribute/contribution-guide.md   | 387 -
 src/contribute/design-principles.md|  53 ---
 src/contribute/eclipse.md  |  18 +-
 src/contribute/index.md| 187 +-
 src/contribute/source-repository.md|  29 --
 src/contribute/work-in-progress.md |  33 --
 9 files changed, 195 insertions(+), 557 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 04/06: Google has scrapped research, now only AI

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4284b6af60b5fd82c960c6593051bdb4d39a1505
Author: Kenneth Knowles 
AuthorDate: Sat May 12 20:02:49 2018 -0700

Google has scrapped research, now only AI
---
 src/_posts/2016-05-20-where-is-my-pcollection-dot-map.md | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/src/_posts/2016-05-20-where-is-my-pcollection-dot-map.md 
b/src/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
index 5fc13f0..f4d431a 100644
--- a/src/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
+++ b/src/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
@@ -11,7 +11,10 @@ Have you ever wondered why Beam has PTransforms for 
everything instead of having
 
 
 
-Though Beam is relatively new, its design draws heavily on many years of 
experience with real-world pipelines. One of the primary inspirations is 
[FlumeJava](http://research.google.com/pubs/pub35650.html), which is Google's 
internal successor to MapReduce first introduced in 2009.
+Though Beam is relatively new, its design draws heavily on many years of
+experience with real-world pipelines. One of the primary inspirations is
+[FlumeJava](https://ai.google/research/pubs/pub35650), which is Google's
+internal successor to MapReduce first introduced in 2009.
 
 The original FlumeJava API has methods like `count` and `parallelDo` on the 
PCollections. Though slightly more succinct, this approach has many 
disadvantages to extensibility. Every new user to FlumeJava wanted to add 
transforms, and adding them as methods to PCollection simply doesn't scale 
well. In contrast, a PCollection in Beam has a single `apply` method which 
takes any PTransform as an argument.
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/06: Move committer section to the bottom (we can find it :-)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 333740a484a9581d6bec36b539deea1159d91d1b
Author: Kenneth Knowles 
AuthorDate: Fri May 11 14:48:52 2018 -0700

Move committer section to the bottom (we can find it :-)
---
 src/_includes/section-menu/contribute.html | 18 +-
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/src/_includes/section-menu/contribute.html 
b/src/_includes/section-menu/contribute.html
index bf74612..a9bea28 100644
--- a/src/_includes/section-menu/contribute.html
+++ b/src/_includes/section-menu/contribute.html
@@ -10,15 +10,6 @@
   
 
 
-  Committers
-
-  
-Become a 
committer
-Committer 
guide
-Release 
guide
-  
-
-
   IDE tips
 
   
@@ -46,3 +37,12 @@
 Docker 
images
   
 
+
+  Committers
+
+  
+Become a 
committer
+Committer 
guide
+Release 
guide
+  
+

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (4605bf7 -> 21a7749)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4605bf7  This closes #410
 add ae67bad  Prepare repository for deployment.
 add bef6e32  Add top-level Community section
 add 712fd02  Remove obsolete Maturity Model section
 add 94ceaae  Regenerate site content
 add 7a3  Merge #436: Add Community top-level section
 new 67b9a07  Move Release Guide to committers section
 new 333740a  Move committer section to the bottom (we can find it :-)
 new a4f2e5c  Streamline the contribution section
 new 4284b6a  Google has scrapped research, now only AI
 new 25a50fe  Temporarily suppress validation of ai.google links
 new 21a7749  This closes #438

The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 Rakefile   |6 +-
 .../capability/2016/03/17/capability-matrix.html   |3 +
 .../2016/04/03/presentation-materials.html |3 +
 .../sdk/2016/02/25/python-sdk-now-public.html  |3 +
 content/beam/release/2016/06/15/first-release.html |3 +
 .../2016/10/11/strata-hadoop-world-and-beam.html   |3 +
 .../update/website/2016/02/22/beam-has-a-logo.html |3 +
 .../blog/2016/05/18/splitAtFraction-method.html|3 +
 .../05/27/where-is-my-pcollection-dot-map.html |3 +
 .../2016/06/13/flink-batch-runner-milestone.html   |3 +
 content/blog/2016/08/03/six-months.html|3 +
 content/blog/2016/10/20/test-stream.html   |3 +
 content/blog/2017/01/09/added-apex-runner.html |3 +
 content/blog/2017/01/10/beam-graduates.html|3 +
 .../blog/2017/02/01/graduation-media-recap.html|3 +
 content/blog/2017/02/13/stateful-processing.html   |3 +
 content/blog/2017/03/16/python-sdk-release.html|3 +
 .../blog/2017/05/17/beam-first-stable-release.html |3 +
 content/blog/2017/08/16/splittable-do-fn.html  |3 +
 content/blog/2017/08/28/timely-processing.html |3 +
 content/blog/2018/01/09/beam-a-look-back.html  |3 +
 content/blog/2018/02/19/beam-2.3.0.html|3 +
 content/blog/index.html|3 +
 content/coming-soon.html   |3 +
 .../contact-us}/index.html |   88 +-
 content/{contribute => community}/logos/index.html |   67 +-
 .../presentation-materials/index.html  |   67 +-
 content/contribute/become-a-committer/index.html   |   15 +-
 content/contribute/committer-guide/index.html  |   15 +-
 content/contribute/contribution-guide/index.html   |   15 +-
 content/contribute/design-principles/index.html|   15 +-
 content/contribute/docker-images/index.html|   15 +-
 content/contribute/eclipse/index.html  |   15 +-
 content/contribute/feature-branches/index.html |   15 +-
 content/contribute/index.html  |   15 +-
 content/contribute/intellij/index.html |   15 +-
 content/contribute/logos/index.html|  537 +-
 content/contribute/maturity-model/index.html   | 6894 
 content/contribute/portability/index.html  |   15 +-
 .../contribute/presentation-materials/index.html   |  253 +-
 .../contribute/ptransform-style-guide/index.html   |   15 +-
 content/contribute/release-guide/index.html|   15 +-
 content/contribute/runner-guide/index.html |   15 +-
 content/contribute/source-repository/index.html|   15 +-
 content/contribute/testing/index.html  |   15 +-
 .../contribute/website-contributions/index.html|   15 +-
 content/contribute/work-in-progress/index.html |   15 +-
 content/documentation/dsls/sql/index.html  |4 +
 content/documentation/execution-model/index.html   |3 +
 content/documentation/index.html   |3 +
 content/documentation/io/authoring-java/index.html |3 +
 .../documentation/io/authoring-overview/index.html |3 +
 .../documentation/io/authoring-python/index.html   |3 +
 .../documentation/io/built-in/hadoop/index.html|3 +
 content/documentation/io/built-in/index.html   |3 +
 content/documentation/io/contributing/index.html   |3 +
 content/documentation/io/io-toc/index.html |3 +
 content/documentation/io/testing/index.html|3 +
 .../pipelines/create-your-pipeline/index.html  |3 +
 .../pipelines/design-your-pipeline/index.html  |3 +
 .../pipelines/test-your-pipeline/index.html|3 +
 content/documentation/programming-guide/index.html |  150 +-
 content/documentation/resources/index.html |3 +
 

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=102000=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102000
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 15/May/18 04:43
Start Date: 15/May/18 04:43
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5347: [BEAM-4201][SQL] 
Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-389041025
 
 
   I may be out of date - previously ITs were run only as postcommit. Gradle 
makes it more fluid.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 102000)
Time Spent: 2h 40m  (was: 2.5h)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101999
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 15/May/18 04:42
Start Date: 15/May/18 04:42
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5347: [BEAM-4201][SQL] 
Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-389040981
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101999)
Time Spent: 2.5h  (was: 2h 20m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101998
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 15/May/18 04:38
Start Date: 15/May/18 04:38
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #5347: [BEAM-4201][SQL] 
Add integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-389040463
 
 
   Python precommit is on its own for this one. If we had fancier 
infrastructure it wouldn't even have run.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101998)
Time Spent: 2h 20m  (was: 2h 10m)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1641

[Apache Jenkins Server]

See 


--
[...truncated 43.12 MB...]
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229003137-a5f5744f/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229014851-397f3d67/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229020742-59ba9d2d/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229062629-e2484b99/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229065845-b5611253/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229072631-3655ccd9/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229074624-99235257/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229075040-f38e1595/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229092605-20e4793/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229123428-bade7939/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229144353-655f1b72/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229155654-98634bc5/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229162440-fe5bd213/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229181543-5a65137c/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229182130-1751965b/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229183154-85feeb63/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229184039-c32894ef/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229191702-65b947d6/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229223352-8afa5a37/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229223945-d1ffeed1/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229230124-831bc37c/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229230506-fd73a9ff/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1229234147-84c1de4a/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230002351-fee27251/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230003323-63bd4636/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230004633-30c06454/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230005623-257f8b0f/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230011622-c50dc72c/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230022103-9e17d86e/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230024107-233d78d7/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230031120-159c5d8f/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230063213-70b148c4/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230072623-4de9bcdd/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230084327-19f9323e/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230092854-8eeabb75/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230121526-244f46b7/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230122602-238eb604/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230123129-3a875c07/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230124542-ab2a90c/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230135237-55f093f0/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230183534-9c81c906/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1230230753-c4d3565a/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231003500-8102c7aa/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231063216-6eaa537d/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231071036-84eff8d0/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231072855-74e0e964/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231123234-164ff220/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-jenkins-1231183448-b931cab0/
gs://temp-storage-for-end-to-end-tests/wordcountit0teste2ewordcount-sisk-0224195036-90f851c8/

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #264

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add extra documentation to RemoteBundle#close

[tgroh] Add a static constructor to GrpcStateService

[kirpichov] fromQuery().withTemplateCompatibility() should do the same thing as

--
[...truncated 10.32 MB...]
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow)
 as step s8
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
May 15, 2018 4:11:28 AM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
May 15, 2018 4:11:28 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101995
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 04:04
Start Date: 15/May/18 04:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389036091
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101995)
Time Spent: 22h  (was: 21h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 22h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101996
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 04:04
Start Date: 15/May/18 04:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389036123
 
 
   Run Python Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101996)
Time Spent: 22h 10m  (was: 22h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 22h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101994
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 03:59
Start Date: 15/May/18 03:59
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389035498
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101994)
Time Spent: 21h 50m  (was: 21h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4971

[Apache Jenkins Server]

See 


--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #151

[Apache Jenkins Server]

See 


Changes:

[iemejia] [BEAM-3813] Refactor constructors to use only S3Options instead of 
each

[iemejia] [BEAM-3813] Update aws-java-sdk version to version 1.11.319

[iemejia] [BEAM-3813] Add support for SSE-C in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-S3 in S3 Filesystem

[iemejia] [BEAM-3813] Add support for SSE-KMS in S3 Filesystem

[iemejia] [BEAM-3813] Fix error-prone and other warnings for the AWS module

[tgroh] Add extra documentation to RemoteBundle#close

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[github] Use preinstalled virtualenv on Jenkins nodes.

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

[iemejia] Fix error-prone warnings for HCatalogIO

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

[tgroh] Add a static constructor to GrpcStateService

[kirpichov] fromQuery().withTemplateCompatibility() should do the same thing as

--
[...truncated 1.41 KB...]
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PostCommit_Python_ValidatesContainer_Dataflow] $ /bin/bash -xe 
/tmp/jenkins1991804530416006633.sh
+ cd src
+ bash sdks/python/container/run_validatescontainer.sh

# pip install --user installation location.
LOCAL_PATH=$HOME/.local/bin/

# Where to store integration test outputs.
GCS_LOCATION=gs://temp-storage-for-end-to-end-tests

# Project for the container and integration test
PROJECT=apache-beam-testing

# Verify in the root of the repository
test -d sdks/python/container

# Verify docker and gcloud commands exist
command -v docker
/usr/bin/docker
command -v gcloud
/usr/bin/gcloud
docker -v
Docker version 17.05.0-ce, build 89658be
gcloud -v
Google Cloud SDK 191.0.0
alpha 2018.02.23
beta 2018.02.23
bq 2.0.29
core 2018.02.23
gsutil 4.28

# ensure gcloud is version 186 or above
TMPDIR=$(mktemp -d)
mktemp -d
gcloud_ver=$(gcloud -v | head -1 | awk '{print $4}')
gcloud -v | head -1 | awk '{print $4}'
if [[ "$gcloud_ver" < "186" ]]
then
  pushd $TMPDIR
  curl 
https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-186.0.0-linux-x86_64.tar.gz
 --output gcloud.tar.gz
  tar xf gcloud.tar.gz
  ./google-cloud-sdk/install.sh --quiet
  . ./google-cloud-sdk/path.bash.inc
  popd
  gcloud components update --quiet || echo 'gcloud components update failed'
  gcloud -v
fi

# Build the container
TAG=$(date +%Y%m%d-%H%M%S)
date +%Y%m%d-%H%M%S
CONTAINER=us.gcr.io/$PROJECT/$USER/python
echo "Using container $CONTAINER"
Using container us.gcr.io/apache-beam-testing/jenkins/python
./gradlew :beam-sdks-python-container:docker 
-Pdocker-repository-root=us.gcr.io/$PROJECT/$USER -Pdocker-tag=$TAG
Parallel execution with configuration on demand is an incubating feature.

> Configure project :
Applying build_rules.gradle to beam
createPerformanceTestHarness with default configuration for project beam
Adding 47 .gitignore exclusions to Apache Rat

> Configure project :beam-sdks-python-container
Applying build_rules.gradle to beam-sdks-python-container
applyGoNature with default configuration for project beam-sdks-python-container
applyDockerNature with default configuration for project 
beam-sdks-python-container
containerImageName with [name:python] for project beam-sdks-python-container

> Configure project :beam-sdks-go
Applying build_rules.gradle to beam-sdks-go
applyGoNature with default configuration for project beam-sdks-go

> Task :beam-sdks-go:prepare
Use project GOPATH: 


> Task :beam-sdks-python-container:prepare
Use project GOPATH: 


> Task :beam-sdks-python-container:dockerClean UP-TO-DATE

> Task :beam-sdks-python:setupVirtualenv
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
Collecting 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101987
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 03:02
Start Date: 15/May/18 03:02
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389027957
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101987)
Time Spent: 21h 40m  (was: 21.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101986
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:58
Start Date: 15/May/18 02:58
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389027362
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101986)
Time Spent: 21.5h  (was: 21h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101985
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:51
Start Date: 15/May/18 02:51
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389026314
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101985)
Time Spent: 21h 20m  (was: 21h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101984=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101984
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:46
Start Date: 15/May/18 02:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389025496
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101984)
Time Spent: 21h 10m  (was: 21h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #394

[Apache Jenkins Server]

See 


Changes:

[kirpichov] fromQuery().withTemplateCompatibility() should do the same thing as

--
[...truncated 19.00 MB...]
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-14_19_33_49-9441561379442316196
May 15, 2018 2:33:50 AM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-14_19_33_49-9441561379442316196 with 0 
expected assertions.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:49.810Z: Autoscaling is enabled for job 
2018-05-14_19_33_49-9441561379442316196. The number of workers will be between 
1 and 1000.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:49.852Z: Autoscaling was automatically enabled for 
job 2018-05-14_19_33_49-9441561379442316196.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:52.580Z: Checking required Cloud APIs are enabled.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:52.766Z: Checking permissions granted to controller 
Service Account.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:56.795Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.250Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.510Z: Expanding GroupByKey operations into 
optimizable parts.
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.550Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.832Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.880Z: Elided trivial flatten 
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.928Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:57.966Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:58.002Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:58.048Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:58.092Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 15, 2018 2:34:01 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-15T02:33:58.133Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4970

[Apache Jenkins Server]

See 


Changes:

[kirpichov] fromQuery().withTemplateCompatibility() should do the same thing as

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101979
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:23
Start Date: 15/May/18 02:23
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389022141
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101979)
Time Spent: 21h  (was: 20h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 21h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101978=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101978
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:18
Start Date: 15/May/18 02:18
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389021338
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101978)
Time Spent: 20h 50m  (was: 20h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101977
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:07
Start Date: 15/May/18 02:07
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389019584
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101977)
Time Spent: 20h 40m  (was: 20.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101976
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 02:03
Start Date: 15/May/18 02:03
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389018952
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101976)
Time Spent: 20.5h  (was: 20h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3827) Add Go SDK integration tests

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3827?focusedWorklogId=101975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101975
 ]

ASF GitHub Bot logged work on BEAM-3827:


Author: ASF GitHub Bot
Created on: 15/May/18 01:56
Start Date: 15/May/18 01:56
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #5326: [BEAM-3827] Add Go 
integration tests
URL: https://github.com/apache/beam/pull/5326#issuecomment-389017845
 
 
   Any comments?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101975)
Time Spent: 0.5h  (was: 20m)

> Add Go SDK integration tests
> 
>
> Key: BEAM-3827
> URL: https://issues.apache.org/jira/browse/BEAM-3827
> Project: Beam
>  Issue Type: Task
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> We should add post-commit testing similar to ValidateRunner tests to ensure 
> that the model is implemented correctly.
> Proposal: 
> https://docs.google.com/document/d/1jy6EE7D4RjgfNV0FhD3rMsT1YKhnUfcHRZMAlC6ygXw/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5360: fromQuery().withTemplateCompatibility() should do the same thing as without

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d0aafa9217db6e459b022580a5be4a50c4eb8254
Merge: 0def450 2fb76f1
Author: Eugene Kirpichov 
AuthorDate: Mon May 14 18:51:50 2018 -0700

Merge pull request #5360: fromQuery().withTemplateCompatibility() should do 
the same thing as without

fromQuery().withTemplateCompatibility() should do the same thing as without

 .../main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (0def450 -> d0aafa9)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0def450  Merge pull request #5291: Add extra documentation to 
RemoteBundle#close
 add 2fb76f1  fromQuery().withTemplateCompatibility() should do the same 
thing as without
 new d0aafa9  Merge pull request #5360: 
fromQuery().withTemplateCompatibility() should do the same thing as without

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java | 8 ++--
 1 file changed, 6 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Created] (BEAM-4293) Add mobile gaming example

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-4293:
---

 Summary: Add mobile gaming example
 Key: BEAM-4293
 URL: https://issues.apache.org/jira/browse/BEAM-4293
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Henning Rohde


It is one of the main common examples on the Beam website.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101974
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:34
Start Date: 15/May/18 01:34
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389014309
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101974)
Time Spent: 20h 20m  (was: 20h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4292) Add streaming wordcount example

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-4292:
---

 Summary: Add streaming wordcount example
 Key: BEAM-4292
 URL: https://issues.apache.org/jira/browse/BEAM-4292
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Henning Rohde


It is referenced on the Beam website as part of the Wordcount progression.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101973
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:30
Start Date: 15/May/18 01:30
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389013696
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101973)
Time Spent: 20h 10m  (was: 20h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101972=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101972
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:26
Start Date: 15/May/18 01:26
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389012910
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101972)
Time Spent: 20h  (was: 19h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 20h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101970
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 15/May/18 01:23
Start Date: 15/May/18 01:23
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5359: 
[BEAM-4286] Implement pooled artifact source
URL: https://github.com/apache/beam/pull/5359#discussion_r188142354
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/ArtifactSourcePool.java
 ##
 @@ -39,23 +44,134 @@
 @ThreadSafe
 public class ArtifactSourcePool implements ArtifactSource {
 
+  public static ArtifactSourcePool create() {
+return new ArtifactSourcePool();
+  }
+
+  private final Object lock = new Object();
+  private final Map artifactSources = 
Maps.newLinkedHashMap();
+
   private ArtifactSourcePool() {}
 
   /**
* Adds a new cache to the pool. When the returned {@link AutoCloseable} is 
closed, the given
-   * cache will be removed from the pool.
+   * cache will be removed from the pool. The call to {@link 
AutoCloseable#close()} will block until
+   * the artifact source is no longer being used.
*/
   public AutoCloseable addToPool(ArtifactSource artifactSource) {
-throw new UnsupportedOperationException();
+synchronized (lock) {
+  checkState(!artifactSources.containsKey(artifactSource));
+  artifactSources.put(artifactSource, new ArtifactSourceLock());
+  return () -> {
+synchronized (lock) {
+  ArtifactSourceLock innerLock = 
artifactSources.remove(artifactSource);
+  checkState(innerLock != null);
+  innerLock.close();
+}
+  };
+}
   }
 
   @Override
   public Manifest getManifest() throws IOException {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
+  Map.Entry entry =
+  artifactSources.entrySet().iterator().next();
+  source = entry.getKey();
+  sourceHandle = entry.getValue().open();
+}
+try {
+  return source.getManifest();
+} finally {
+  sourceHandle.close();
+}
   }
 
   @Override
   public void getArtifact(String name, StreamObserver 
responseObserver) {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
 
 Review comment:
   Seems that you can deduplicate this w.r.t. the previous method into a method 
`SourceHandle getAny()`, if you make `SourceHandle` contain the 
`ArtifactSource` too.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101970)
Time Spent: 50m  (was: 40m)

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101971
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 15/May/18 01:23
Start Date: 15/May/18 01:23
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5359: 
[BEAM-4286] Implement pooled artifact source
URL: https://github.com/apache/beam/pull/5359#discussion_r188141986
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/ArtifactSourcePool.java
 ##
 @@ -39,23 +44,134 @@
 @ThreadSafe
 public class ArtifactSourcePool implements ArtifactSource {
 
+  public static ArtifactSourcePool create() {
 
 Review comment:
   Hmm the javadoc says we need this for DistributedCache - but we're not 
planning on using DistributedCache just yet. Is this class still relevant then?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101971)
Time Spent: 50m  (was: 40m)

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101968=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101968
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 15/May/18 01:23
Start Date: 15/May/18 01:23
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5359: 
[BEAM-4286] Implement pooled artifact source
URL: https://github.com/apache/beam/pull/5359#discussion_r188142721
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/ArtifactSourcePool.java
 ##
 @@ -39,23 +44,134 @@
 @ThreadSafe
 public class ArtifactSourcePool implements ArtifactSource {
 
+  public static ArtifactSourcePool create() {
+return new ArtifactSourcePool();
+  }
+
+  private final Object lock = new Object();
+  private final Map artifactSources = 
Maps.newLinkedHashMap();
+
   private ArtifactSourcePool() {}
 
   /**
* Adds a new cache to the pool. When the returned {@link AutoCloseable} is 
closed, the given
-   * cache will be removed from the pool.
+   * cache will be removed from the pool. The call to {@link 
AutoCloseable#close()} will block until
+   * the artifact source is no longer being used.
*/
   public AutoCloseable addToPool(ArtifactSource artifactSource) {
-throw new UnsupportedOperationException();
+synchronized (lock) {
+  checkState(!artifactSources.containsKey(artifactSource));
+  artifactSources.put(artifactSource, new ArtifactSourceLock());
+  return () -> {
+synchronized (lock) {
+  ArtifactSourceLock innerLock = 
artifactSources.remove(artifactSource);
+  checkState(innerLock != null);
+  innerLock.close();
+}
+  };
+}
   }
 
   @Override
   public Manifest getManifest() throws IOException {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
+  Map.Entry entry =
+  artifactSources.entrySet().iterator().next();
+  source = entry.getKey();
+  sourceHandle = entry.getValue().open();
+}
+try {
+  return source.getManifest();
+} finally {
+  sourceHandle.close();
+}
   }
 
   @Override
   public void getArtifact(String name, StreamObserver 
responseObserver) {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
+  Map.Entry entry =
+  artifactSources.entrySet().iterator().next();
+  source = entry.getKey();
+  sourceHandle = entry.getValue().open();
+}
+try {
+  source.getArtifact(name, responseObserver);
+} finally {
+  sourceHandle.close();
+}
+  }
+
+  /** Manages the state of a composed artifact source. */
 
 Review comment:
   On second thought, do we even need this reference counting? Is this to 
ensure that removing from the pool will block on completion of all currently 
running operations on this ArtifactSource? That doesn't seem necessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101968)
Time Spent: 0.5h  (was: 20m)

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101969=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101969
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 15/May/18 01:23
Start Date: 15/May/18 01:23
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #5359: 
[BEAM-4286] Implement pooled artifact source
URL: https://github.com/apache/beam/pull/5359#discussion_r188141811
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/ArtifactSourcePool.java
 ##
 @@ -39,23 +44,134 @@
 @ThreadSafe
 public class ArtifactSourcePool implements ArtifactSource {
 
+  public static ArtifactSourcePool create() {
+return new ArtifactSourcePool();
+  }
+
+  private final Object lock = new Object();
+  private final Map artifactSources = 
Maps.newLinkedHashMap();
+
   private ArtifactSourcePool() {}
 
   /**
* Adds a new cache to the pool. When the returned {@link AutoCloseable} is 
closed, the given
-   * cache will be removed from the pool.
+   * cache will be removed from the pool. The call to {@link 
AutoCloseable#close()} will block until
+   * the artifact source is no longer being used.
*/
   public AutoCloseable addToPool(ArtifactSource artifactSource) {
-throw new UnsupportedOperationException();
+synchronized (lock) {
+  checkState(!artifactSources.containsKey(artifactSource));
+  artifactSources.put(artifactSource, new ArtifactSourceLock());
+  return () -> {
+synchronized (lock) {
+  ArtifactSourceLock innerLock = 
artifactSources.remove(artifactSource);
+  checkState(innerLock != null);
+  innerLock.close();
+}
+  };
+}
   }
 
   @Override
   public Manifest getManifest() throws IOException {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
+  Map.Entry entry =
+  artifactSources.entrySet().iterator().next();
+  source = entry.getKey();
+  sourceHandle = entry.getValue().open();
+}
+try {
+  return source.getManifest();
+} finally {
+  sourceHandle.close();
+}
   }
 
   @Override
   public void getArtifact(String name, StreamObserver 
responseObserver) {
-throw new UnsupportedOperationException();
+ArtifactSource source;
+SourceHandle sourceHandle;
+synchronized (lock) {
+  checkState(!artifactSources.isEmpty());
+  Map.Entry entry =
+  artifactSources.entrySet().iterator().next();
+  source = entry.getKey();
+  sourceHandle = entry.getValue().open();
+}
+try {
+  source.getArtifact(name, responseObserver);
+} finally {
+  sourceHandle.close();
+}
+  }
+
+  /** Manages the state of a composed artifact source. */
 
 Review comment:
   For a blocking reference counter, I think you can use Phaser - see 
https://stackoverflow.com/a/32442754/278042


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101969)
Time Spent: 40m  (was: 0.5h)

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101959
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141417
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
 
 Review comment:
   
   This method will only be used  to download files from GCS to local folder, 
the part that handles uploads to GCS would be a dead code, let's remove it. 
Uploads will be handled by stage_artifact.
   
   I think it would be easier to reason about the code if we have 
`_download_file` method and GCS / HTTP helper methods as follows:
   ```
   def _download_file(from_path, to_path):
 if from_path starts.with('gcs://'):
   _download_from_gcs(from_path, to_path)
   ...
 else if from_path starts.with('http'):
   _download_from_http(from_path, to_path)
   ...  
 else: 
   # used in tests 
   ...
   ```
   
   Also since 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101955=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101955
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389006329
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101955)
Time Spent: 19h 40m  (was: 19.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101967=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101967
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141414
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101960
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141423
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101966
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141429
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/dependency.py
 ##
 @@ -607,65 +151,9 @@ def get_sdk_name_and_version():
 def get_sdk_package_name():
 
 Review comment:
   
   Let's implement this in LegacyDataflowStager (see another comment).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101966)
Time Spent: 9h 10m  (was: 9h)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101963=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101963
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141413
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101961
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141416
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101964
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141422
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101962=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101962
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141415
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101956=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101956
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389007022
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101956)
Time Spent: 19h 50m  (was: 19h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101965
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141420
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101958
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141421
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/stager.py
 ##
 @@ -0,0 +1,573 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Support for installing custom code and required dependencies.
+
+Workflows, with the exception of very simple ones, are organized in multiple
+modules and packages. Typically, these modules and packages have
+dependencies on other standard libraries. Beam relies on the Python
+setuptools package to handle these scenarios. For further details please read:
+https://pythonhosted.org/an_example_pypi_project/setuptools.html
+
+When a runner tries to run a pipeline it will check for a --requirements_file
+and a --setup_file option.
+
+If --setup_file is present then it is assumed that the folder containing the
+file specified by the option has the typical layout required by setuptools and
+it will run 'python setup.py sdist' to produce a source distribution. The
+resulting tarball (a .tar or .tar.gz file) will be staged at the staging
+location specified as job option. When a worker starts it will check for the
+presence of this file and will run 'easy_install tarball' to install the
+package in the worker.
+
+If --requirements_file is present then the file specified by the option will be
+staged in the staging location.  When a worker starts it will check for the
+presence of this file and will run 'pip install -r requirements.txt'. A
+requirements file can be easily generated by running 'pip freeze -r
+requirements.txt'. The reason a runner does not run this automatically is
+because quite often only a small fraction of the dependencies present in a
+requirements.txt file are actually needed for remote execution and therefore a
+one-time manual trimming is desirable.
+
+TODO(silviuc): Should we allow several setup packages?
+TODO(silviuc): We should allow customizing the exact command for setup build.
+"""
+import functools
+import glob
+import logging
+import os
+import shutil
+import subprocess
+import sys
+import tempfile
+
+import pkg_resources
+
+from apache_beam.internal import pickler
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.options.pipeline_options import SetupOptions
+# TODO(angoenka): Remove reference to dataflow internal names
+from apache_beam.runners.dataflow.internal import names
+from apache_beam.utils import processes
+
+# All constants are for internal use only; no backwards-compatibility
+# guarantees.
+
+# Standard file names used for staging files.
+WORKFLOW_TARBALL_FILE = 'workflow.tar.gz'
+REQUIREMENTS_FILE = 'requirements.txt'
+EXTRA_PACKAGES_FILE = 'extra_packages.txt'
+
+# Package names for distributions
+BEAM_PACKAGE_NAME = 'apache-beam'
+
+
+class Stager(object):
+  """Stager identifies and copies the appropriate artifacts to the staging
+  location."""
+
+  def _copy_file(self, from_path, to_path):
+"""Copies a local file to a GCS file or vice versa."""
+logging.info('file copy from %s to %s.', from_path, to_path)
+if from_path.startswith('gs://') or to_path.startswith('gs://'):
+  from apache_beam.io.gcp import gcsio
+  if from_path.startswith('gs://') and to_path.startswith('gs://'):
+# Both files are GCS files so copy.
+gcsio.GcsIO().copy(from_path, to_path)
+  elif to_path.startswith('gs://'):
+# Only target is a GCS file, read local file and upload.
+with open(from_path, 'rb') as f:
+  with gcsio.GcsIO().open(to_path, mode='wb') as g:
+pfun = functools.partial(f.read, gcsio.WRITE_CHUNK_SIZE)
+for chunk in iter(pfun, ''):
+  

[jira] [Work logged] (BEAM-3883) Python SDK stages artifacts when talking to job server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3883?focusedWorklogId=101957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101957
 ]

ASF GitHub Bot logged work on BEAM-3883:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5251: 
[BEAM-3883] Refactor and clean dependency.py to make it reusable with artifact 
service
URL: https://github.com/apache/beam/pull/5251#discussion_r188141411
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -731,6 +746,18 @@ def translate_scalar_counter_float(accumulator, 
metric_update_proto):
 metric_update_proto.floatingPoint = accumulator.value
 
 
+class _ParameterizedStager(Stager):
 
 Review comment:
   
   Let's call this `_LegacyDataflowStager`, and take an instance of 
`DataflowApplicationClient` in the constructor, then let's call 
`dataflow_application_client._gcs_file_copy()` in stage_artifact.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101957)
Time Spent: 7h 50m  (was: 7h 40m)

> Python SDK stages artifacts when talking to job server
> --
>
> Key: BEAM-3883
> URL: https://issues.apache.org/jira/browse/BEAM-3883
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Ben Sidhom
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> The Python SDK does not currently stage its user-defined functions or 
> dependencies when talking to the job API. Artifacts that need to be staged 
> include the user code itself, any SDK components not included in the 
> container image, and the list of Python packages that must be installed at 
> runtime.
>  
> Artifacts that are currently expected can be found in the harness boot code: 
> [https://github.com/apache/beam/blob/58e3b06bee7378d2d8db1c8dd534b415864f63e1/sdks/python/container/boot.go#L52.]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4291) ArtifactRetrievalService that retrieves artifacts from a distributed filesystem

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-4291:
--

 Summary: ArtifactRetrievalService that retrieves artifacts from a 
distributed filesystem
 Key: BEAM-4291
 URL: https://issues.apache.org/jira/browse/BEAM-4291
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core
Reporter: Eugene Kirpichov
Assignee: Axel Magnuson


In agreement with how they are staged in BEAM-4290.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101954
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:11
Start Date: 15/May/18 01:11
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389010582
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101954)
Time Spent: 19.5h  (was: 19h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4290) ArtifactStagingService that stages to a distributed filesystem

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-4290:
--

 Summary: ArtifactStagingService that stages to a distributed 
filesystem
 Key: BEAM-4290
 URL: https://issues.apache.org/jira/browse/BEAM-4290
 Project: Beam
  Issue Type: Sub-task
  Components: runner-core
Reporter: Eugene Kirpichov
Assignee: Ankur Goenka


Using the job's staging directory from PipelineOptions.

Physical layout on the distributed filesystem is TBD but it should allow for 
arbitrary filenames and ideally for eventually avoiding uploading artifacts 
that are already there.

Handling credentials is TBD.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101952=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101952
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 01:05
Start Date: 15/May/18 01:05
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389009670
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101952)
Time Spent: 19h 20m  (was: 19h 10m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4969

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add extra documentation to RemoteBundle#close

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 

[jira] [Created] (BEAM-4289) How to express "Job Update" in portable Beam

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-4289:
--

 Summary: How to express "Job Update" in portable Beam
 Key: BEAM-4289
 URL: https://issues.apache.org/jira/browse/BEAM-4289
 Project: Beam
  Issue Type: Improvement
  Components: beam-model
Reporter: Ankur Goenka
Assignee: Ankur Goenka


Based in the discussion here 
[https://docs.google.com/a/google.com/document/d/1xOaEEJrMmiSHprd-WiYABegfT129qqF-idUBINjxz8s/edit?disco=B9MqC-A]

For portable beam, how can user express intent to update the a running job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101949
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 00:47
Start Date: 15/May/18 00:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389007022
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101949)
Time Spent: 19h 10m  (was: 19h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101948=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101948
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 15/May/18 00:42
Start Date: 15/May/18 00:42
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-389006329
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101948)
Time Spent: 19h  (was: 18h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 19h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101946
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 15/May/18 00:34
Start Date: 15/May/18 00:34
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188137062
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
 
 Review comment:
   Tested exceptions in the `close` methods.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101946)
Time Spent: 9h 20m  (was: 9h 10m)

> Execute a Stage via the portability framework in the ReferenceRunner
> 
>
> Key: BEAM-3326
> URL: https://issues.apache.org/jira/browse/BEAM-3326
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> This is the supertask for remote execution in the Universal Local Runner 
> (BEAM-2899).
> This executes a stage remotely via portability framework APIs



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101947
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 15/May/18 00:34
Start Date: 15/May/18 00:34
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188137160
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
+  private final EnvironmentFactory environmentFactory;
+
+  private final GrpcFnServer dataService;
+  private final GrpcFnServer stateService;
+
+  private final ConcurrentMap 
stageBundleFactories =
+  new ConcurrentHashMap<>();
+  private final ConcurrentMap environments =
+  new ConcurrentHashMap<>();
+
+  DirectJobBundleFactory(
+  EnvironmentFactory environmentFactory,
+  GrpcFnServer dataService,
+  GrpcFnServer stateService) {
+this.environmentFactory = environmentFactory;
+this.dataService = dataService;
+this.stateService = stateService;
+  }
+
+  @Override
+  public  StageBundleFactory forStage(ExecutableStage executableStage) {
+return (StageBundleFactory)
+stageBundleFactories.computeIfAbsent(executableStage, 
this::createBundleFactory);
+  }
+
+  private final AtomicLong idgen = new AtomicLong();
+
+  private  StageBundleFactory createBundleFactory(ExecutableStage stage) 
{
+RemoteEnvironment remoteEnv =
+environments.computeIfAbsent(
+stage.getEnvironment(),
+env -> {
+  try {
+return environmentFactory.createEnvironment(env);
+  } catch (Exception e) {
+throw new RuntimeException(e);
+  }
+});
+SdkHarnessClient sdkHarnessClient =
+

[jira] [Assigned] (BEAM-4267) Implement a reusable library that can run an ExecutableStage with a given Environment

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-4267:
--

Assignee: Ben Sidhom  (was: Axel Magnuson)

> Implement a reusable library that can run an ExecutableStage with a given 
> Environment
> -
>
> Key: BEAM-4267
> URL: https://issues.apache.org/jira/browse/BEAM-4267
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Axel Magnuson
>Assignee: Ben Sidhom
>Priority: Major
>
> Build off of the interfaces introduced in 
> [BEAM-3327|https://github.com/apache/beam/pull/5152] to provide a reusable 
> execution library to runners.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4288) SplittableDoFn: splitAtFraction() API for Python

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-4288:
--

 Summary: SplittableDoFn: splitAtFraction() API for Python
 Key: BEAM-4288
 URL: https://issues.apache.org/jira/browse/BEAM-4288
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


SDF currently only has a checkpoint() API. This Jira is about adding the 
splitAtFraction() API and its support in runners that support the respective 
feature for sources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4287) SplittableDoFn: splitAtFraction() API

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-4287:
--

 Summary: SplittableDoFn: splitAtFraction() API
 Key: BEAM-4287
 URL: https://issues.apache.org/jira/browse/BEAM-4287
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


SDF currently only has a checkpoint() API. This Jira is about adding the 
splitAtFraction() API and its support in runners that support the respective 
feature for sources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4287) SplittableDoFn: splitAtFraction() API for Java

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-4287:
---
Summary: SplittableDoFn: splitAtFraction() API for Java  (was: 
SplittableDoFn: splitAtFraction() API)

> SplittableDoFn: splitAtFraction() API for Java
> --
>
> Key: BEAM-4287
> URL: https://issues.apache.org/jira/browse/BEAM-4287
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Major
>
> SDF currently only has a checkpoint() API. This Jira is about adding the 
> splitAtFraction() API and its support in runners that support the respective 
> feature for sources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4968

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a static constructor to GrpcStateService

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #169

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[github] Use preinstalled virtualenv on Jenkins nodes.

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

[tgroh] Add a static constructor to GrpcStateService

--
[...truncated 647.69 KB...]
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:138)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn$1.output(SimpleParDoFn.java:271)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.outputWindowedValue(SimpleDoFnRunner.java:211)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #176

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[github] Use preinstalled virtualenv on Jenkins nodes.

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

[tgroh] Add a static constructor to GrpcStateService

--
[...truncated 666.63 KB...]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.net.ConnectException: Call From 
textioit0writethenreadall-05141706-sn0d-harness-h2pn.c.apache-beam-testing.internal/10.128.0.39
 to 50.45.225.35.bc.googleusercontent.com:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see:  
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy65.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy66.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:249)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:236)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #171

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[github] Use preinstalled virtualenv on Jenkins nodes.

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

[tgroh] Add a static constructor to GrpcStateService

--
[...truncated 449.67 KB...]
at 
org.apache.beam.runners.core.SimpleDoFnRunner.access$700(SimpleDoFnRunner.java:66)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:436)
at 
org.apache.beam.runners.core.SimpleDoFnRunner$DoFnProcessContext.output(SimpleDoFnRunner.java:424)
at 
org.apache.beam.sdk.transforms.MapElements$1.processElement(MapElements.java:129)
at 
org.apache.beam.sdk.transforms.MapElements$1$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:177)
at 
org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:141)
at 
com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at 
com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at 
com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at 
com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at 
com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:383)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:355)
at 
com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:286)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is {type=UNKNOWN, servers=[{address=35.224.69.199:27017, type=UNKNOWN, 
state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception 
opening socket}, caused by {java.net.SocketTimeoutException: connect timed 
out}}]
at 
com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369)
at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:75)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.(ClusterBinding.java:71)
at 
com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:168)
at 
com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:74)
at com.mongodb.Mongo.execute(Mongo.java:781)
at com.mongodb.Mongo$2.execute(Mongo.java:764)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:323)
at 
com.mongodb.MongoCollectionImpl.insertMany(MongoCollectionImpl.java:311)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.flush(MongoDbIO.java:667)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn.processElement(MongoDbIO.java:652)
com.mongodb.MongoTimeoutException: Timed out after 3 ms while waiting 
for a server that matches WritableServerSelector. Client view of cluster state 
is 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #170

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_JDBC #573

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[github] Use preinstalled virtualenv on Jenkins nodes.

[tgroh] Add a FlattenRunnerTest for a multi-consumed input

[tgroh] Use ListMultimap when creating Transform Runners

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

[tgroh] Add a static constructor to GrpcStateService

--
[...truncated 91.57 KB...]
Skipping task ':beam-sdks-java-io-google-cloud-platform:compileTestJava' as it 
is up-to-date.
:beam-sdks-java-io-google-cloud-platform:compileTestJava (Thread[Task worker 
for ':' Thread 5,5,main]) completed. Took 0.07 secs.

> Task :beam-runners-google-cloud-dataflow-java:classes UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:classes' as it has no 
actions.
:beam-runners-google-cloud-dataflow-java:classes (Thread[Task worker for ':' 
Thread 3,5,main]) completed. Took 0.0 secs.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:testClasses UP-TO-DATE
Skipping task ':beam-sdks-java-io-google-cloud-platform:testClasses' as it has 
no actions.
:beam-sdks-java-io-google-cloud-platform:testClasses (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 3,5,main]) started.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) started.

> Task :beam-sdks-java-io-google-cloud-platform:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar' is 
61adf85f2830f45f4ab9cc9f63929a8b
Caching disabled for task 
':beam-sdks-java-io-google-cloud-platform:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-sdks-java-io-google-cloud-platform:shadowTestJar' as it is 
up-to-date.
:beam-sdks-java-io-google-cloud-platform:shadowTestJar (Thread[Task worker for 
':' Thread 5,5,main]) completed. Took 0.035 secs.

> Task :beam-runners-google-cloud-dataflow-java:shadowJar UP-TO-DATE
Build cache key for task ':beam-runners-google-cloud-dataflow-java:shadowJar' 
is 1e29a0b4834c1cf2a495f272776ebb1f
Caching disabled for task ':beam-runners-google-cloud-dataflow-java:shadowJar': 
Caching has not been enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowJar (Thread[Task worker for ':' 
Thread 3,5,main]) completed. Took 0.037 secs.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 3,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:compileTestJava UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:compileTestJava' is 
cd38d0c52c1f76da3f41516d4f23ae76
Skipping task ':beam-runners-google-cloud-dataflow-java:compileTestJava' as it 
is up-to-date.
:beam-runners-google-cloud-dataflow-java:compileTestJava (Thread[Task worker 
for ':' Thread 3,5,main]) completed. Took 0.054 secs.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 3,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:testClasses UP-TO-DATE
Skipping task ':beam-runners-google-cloud-dataflow-java:testClasses' as it has 
no actions.
:beam-runners-google-cloud-dataflow-java:testClasses (Thread[Task worker for 
':' Thread 3,5,main]) completed. Took 0.0 secs.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 3,5,main]) started.

> Task :beam-runners-google-cloud-dataflow-java:shadowTestJar UP-TO-DATE
Build cache key for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar' is 
5a6337ea971edb357a1982c7d4d17e5f
Caching disabled for task 
':beam-runners-google-cloud-dataflow-java:shadowTestJar': Caching has not been 
enabled for the task
Skipping task ':beam-runners-google-cloud-dataflow-java:shadowTestJar' as it is 
up-to-date.
:beam-runners-google-cloud-dataflow-java:shadowTestJar (Thread[Task worker for 
':' Thread 3,5,main]) completed. Took 0.069 secs.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' Thread 
3,5,main]) started.

> Task :beam-sdks-java-io-jdbc:compileTestJava UP-TO-DATE
Build cache key for task ':beam-sdks-java-io-jdbc:compileTestJava' is 
6b848940525fbab8bf7a576ae3b27134
Skipping task ':beam-sdks-java-io-jdbc:compileTestJava' as it is up-to-date.
:beam-sdks-java-io-jdbc:compileTestJava (Thread[Task worker for ':' 

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101942=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101942
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 14/May/18 23:52
Start Date: 14/May/18 23:52
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-388998501
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101942)
Time Spent: 18h 50m  (was: 18h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101937=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101937
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 14/May/18 23:33
Start Date: 14/May/18 23:33
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-388995266
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101937)
Time Spent: 18h 40m  (was: 18.5h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101936
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 14/May/18 23:31
Start Date: 14/May/18 23:31
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5359: [BEAM-4286] Implement 
pooled artifact source
URL: https://github.com/apache/beam/pull/5359#issuecomment-388995009
 
 
   R: @jkff @axelmagn 
   CC: @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101936)
Time Spent: 20m  (was: 10m)

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4286) Pooled artifact source

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4286?focusedWorklogId=101935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101935
 ]

ASF GitHub Bot logged work on BEAM-4286:


Author: ASF GitHub Bot
Created on: 14/May/18 23:31
Start Date: 14/May/18 23:31
Worklog Time Spent: 10m 
  Work Description: bsidhom opened a new pull request #5359: [BEAM-4286] 
Implement pooled artifact source
URL: https://github.com/apache/beam/pull/5359
 
 
   
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `./gradlew build` to make sure basic checks pass. A more thorough 
check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101935)
Time Spent: 10m
Remaining Estimate: 0h

> Pooled artifact source
> --
>
> Key: BEAM-4286
> URL: https://issues.apache.org/jira/browse/BEAM-4286
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
> need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
> job-scoped and should be the same throughout a job's lifetime. For this 
> reason, it is safe to pool artifact sources and serve artifacts from an 
> arbitrary pooled source as long as the underlying source is still in scope.
> We need a pooled source in order to satisfy the bundle factory interfaces. 
> Using the job-scoped and stage-scoped bundle factories allows us to cache and 
> reuse different components that serve SDK harnesses. Because the distributed 
> cache lifetimes are specific to Flink, the pooled artifact source should 
> probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4286) Pooled artifact source

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4286:


 Summary: Pooled artifact source
 Key: BEAM-4286
 URL: https://issues.apache.org/jira/browse/BEAM-4286
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Ben Sidhom


Because DistributeCache lifetimes are tied to operator lifetimes in Flink, we 
need a way to wrap operator-scoped artifact sources. Artifacts are inherently 
job-scoped and should be the same throughout a job's lifetime. For this reason, 
it is safe to pool artifact sources and serve artifacts from an arbitrary 
pooled source as long as the underlying source is still in scope.

We need a pooled source in order to satisfy the bundle factory interfaces. 
Using the job-scoped and stage-scoped bundle factories allows us to cache and 
reuse different components that serve SDK harnesses. Because the distributed 
cache lifetimes are specific to Flink, the pooled artifact source should 
probably live in a runner-specific directory.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101929=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101929
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 14/May/18 23:24
Start Date: 14/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-382933717
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101929)
Time Spent: 18h 10m  (was: 18h)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101928
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 14/May/18 23:24
Start Date: 14/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-382923316
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101928)
Time Spent: 18h  (was: 17h 50m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 18h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=101927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101927
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 14/May/18 23:24
Start Date: 14/May/18 23:24
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-382874662
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101927)
Time Spent: 17h 50m  (was: 17h 40m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 17h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3326) Execute a Stage via the portability framework in the ReferenceRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3326?focusedWorklogId=101925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101925
 ]

ASF GitHub Bot logged work on BEAM-3326:


Author: ASF GitHub Bot
Created on: 14/May/18 23:22
Start Date: 14/May/18 23:22
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5348: 
[BEAM-3326] Add a Direct Job Bundle Factory
URL: https://github.com/apache/beam/pull/5348#discussion_r188127187
 
 

 ##
 File path: 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectJobBundleFactory.java
 ##
 @@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.direct.portable;
+
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.Target;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.control.JobBundleFactory;
+import org.apache.beam.runners.fnexecution.control.OutputReceiverFactory;
+import org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors;
+import 
org.apache.beam.runners.fnexecution.control.ProcessBundleDescriptors.ExecutableProcessBundleDescriptor;
+import org.apache.beam.runners.fnexecution.control.RemoteBundle;
+import org.apache.beam.runners.fnexecution.control.RemoteOutputReceiver;
+import org.apache.beam.runners.fnexecution.control.SdkHarnessClient;
+import 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient.ActiveBundle;
+import org.apache.beam.runners.fnexecution.control.StageBundleFactory;
+import org.apache.beam.runners.fnexecution.data.GrpcDataService;
+import org.apache.beam.runners.fnexecution.data.RemoteInputDestination;
+import org.apache.beam.runners.fnexecution.environment.EnvironmentFactory;
+import org.apache.beam.runners.fnexecution.environment.RemoteEnvironment;
+import org.apache.beam.runners.fnexecution.state.GrpcStateService;
+import org.apache.beam.runners.fnexecution.state.StateRequestHandler;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.fn.data.FnDataReceiver;
+import org.apache.beam.sdk.util.MoreFutures;
+import org.apache.beam.sdk.util.WindowedValue;
+
+/** A {@link JobBundleFactory} for the ReferenceRunner. */
+class DirectJobBundleFactory implements JobBundleFactory {
+  private final EnvironmentFactory environmentFactory;
+
+  private final GrpcFnServer dataService;
+  private final GrpcFnServer stateService;
+
+  private final ConcurrentMap 
stageBundleFactories =
+  new ConcurrentHashMap<>();
+  private final ConcurrentMap environments =
+  new ConcurrentHashMap<>();
+
+  DirectJobBundleFactory(
+  EnvironmentFactory environmentFactory,
+  GrpcFnServer dataService,
+  GrpcFnServer stateService) {
+this.environmentFactory = environmentFactory;
+this.dataService = dataService;
+this.stateService = stateService;
+  }
+
+  @Override
+  public  StageBundleFactory forStage(ExecutableStage executableStage) {
+return (StageBundleFactory)
+stageBundleFactories.computeIfAbsent(executableStage, 
this::createBundleFactory);
+  }
+
+  private final AtomicLong idgen = new AtomicLong();
 
 Review comment:
   Done.
   
   Done everywhere, specifically.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101925)
Time Spent: 

[jira] [Work logged] (BEAM-2588) Portable Flink Runner Job API

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2588?focusedWorklogId=101921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101921
 ]

ASF GitHub Bot logged work on BEAM-2588:


Author: ASF GitHub Bot
Created on: 14/May/18 22:51
Start Date: 14/May/18 22:51
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #5262: 
[BEAM-2588] WIP Portability Runner Job Service
URL: https://github.com/apache/beam/pull/5262#discussion_r188121725
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/JobService.java
 ##
 @@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.jobsubmission;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.protobuf.Struct;
+import io.grpc.Status;
+import io.grpc.StatusException;
+import io.grpc.StatusRuntimeException;
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.CancelJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.GetJobStateResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessage;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobMessagesResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.JobState;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.PrepareJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobRequest;
+import org.apache.beam.model.jobmanagement.v1.JobApi.RunJobResponse;
+import org.apache.beam.model.jobmanagement.v1.JobServiceGrpc;
+import org.apache.beam.runners.fnexecution.FnService;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService;
+import 
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingServiceProvider;
+import org.apache.beam.sdk.fn.stream.SynchronizedStreamObserver;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A JobService that prepares and runs jobs on behalf of a client using a 
{@link JobInvoker}.
+ *
+ * Job management is handled in-memory rather than any persistent storage, 
running the risk of
+ * leaking jobs if the JobService crashes.
+ *
+ * TODO: replace in-memory job management state with persistent solution.
+ */
+public class JobService extends JobServiceGrpc.JobServiceImplBase implements 
FnService {
+  private static final Logger LOG = LoggerFactory.getLogger(JobService.class);
+
+  public static JobService create(
+  ArtifactStagingServiceProvider artifactStagingServiceProvider, 
JobInvoker invoker) {
+return new JobService(artifactStagingServiceProvider, invoker);
+  }
+
+  private final ConcurrentMap preparations;
+  private final ConcurrentMap invocations;
+  private final ArtifactStagingServiceProvider artifactStagingServiceProvider;
+  private final JobInvoker invoker;
+
+  private JobService(
+  ArtifactStagingServiceProvider artifactStagingServiceProvider, 
JobInvoker invoker) {
+this.artifactStagingServiceProvider = artifactStagingServiceProvider;
+this.invoker = invoker;
+
+this.preparations = new ConcurrentHashMap<>();
+this.invocations = new ConcurrentHashMap<>();
+  }
+
+  @Override
+  public void prepare(
+  PrepareJobRequest request,
+  StreamObserver responseObserver) {
+try {
+  LOG.trace("{} {}", PrepareJobRequest.class.getSimpleName(), request);
+  // insert preparation
+  String 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4967

[Apache Jenkins Server]

See 


Changes:

[iemejia] Fix error-prone warnings and multiple cleanups based on static 
analysis

[tgroh] Enable MapFnRunner to provide full WindowedValues

[tgroh] Implement AssignWindows in the Java SDK Harness

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4966

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Splitting Job Service from ULR

[ankurgoenka] Renaming ULR to PortableRunner and JobService to LocalJobService

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

--
[...truncated 1.01 MB...]
copying apache_beam/runners/dataflow/dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/dataflow_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/ptransform_overrides.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/template_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/test_dataflow_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow
copying apache_beam/runners/dataflow/internal/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/apiclient_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/dependency_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/names.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal
copying apache_beam/runners/dataflow/internal/clients/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients
copying apache_beam/runners/dataflow/internal/clients/dataflow/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying 

[jira] [Work logged] (BEAM-4201) Integration Tests for PubsubIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4201?focusedWorklogId=101908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101908
 ]

ASF GitHub Bot logged work on BEAM-4201:


Author: ASF GitHub Bot
Created on: 14/May/18 22:10
Start Date: 14/May/18 22:10
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #5347: [BEAM-4201][SQL] Add 
integration test for PubsubIO JSON -> SQL
URL: https://github.com/apache/beam/pull/5347#issuecomment-388979163
 
 
   run python precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 101908)
Time Spent: 2h 10m  (was: 2h)

> Integration Tests for PubsubIO
> --
>
> Key: BEAM-4201
> URL: https://issues.apache.org/jira/browse/BEAM-4201
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> Add integration tests for PubsubIO



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #388

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3970) Java SDK harness supports window_into

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3970.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Java SDK harness supports window_into
> -
>
> Key: BEAM-3970
> URL: https://issues.apache.org/jira/browse/BEAM-3970
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-harness
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Major
> Fix For: 2.5.0
>
>
> The Java SDK harness does not currently register a PTransformRunnerFactory 
> for beam:transform:window_into:v1. We need this functionality for GroupByKey 
> transforms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4269) Implement Assign Windows in the Java SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4269?focusedWorklogId=101907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101907
 ]

ASF GitHub Bot logged work on BEAM-4269:


Author: ASF GitHub Bot
Created on: 14/May/18 22:06
Start Date: 14/May/18 22:06
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #5329: [BEAM-4269, 
BEAM-3970] Implement AssignWindows in the Java SDK harness
URL: https://github.com/apache/beam/pull/5329
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
new file mode 100644
index 000..84a2771d3b0
--- /dev/null
+++ 
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.fn.harness;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Map;
+import org.apache.beam.fn.harness.MapFnRunners.WindowedValueMapFnFactory;
+import org.apache.beam.model.pipeline.v1.RunnerApi.PTransform;
+import org.apache.beam.model.pipeline.v1.RunnerApi.WindowIntoPayload;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.sdk.fn.function.ThrowingFunction;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.joda.time.Instant;
+
+/** The Java SDK Harness implementation of the {@link Window.Assign} 
primitive. */
+class AssignWindowsRunner {
+
+  /** A registrar which provides a factory to handle Java {@link WindowFn 
WindowFns}. */
+  @AutoService(PTransformRunnerFactory.Registrar.class)
+  public static class Registrar implements PTransformRunnerFactory.Registrar {
+@Override
+public Map getPTransformRunnerFactories() 
{
+  return ImmutableMap.of(
+  PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN,
+  MapFnRunners.forWindowedValueMapFnFactory(new 
AssignWindowsMapFnFactory<>()));
+}
+  }
+
+  @VisibleForTesting
+  static class AssignWindowsMapFnFactory implements 
WindowedValueMapFnFactory {
+@Override
+public ThrowingFunction forPTransform(
+String ptransformId, PTransform ptransform) throws IOException {
+  checkArgument(
+  
PTransformTranslation.ASSIGN_WINDOWS_TRANSFORM_URN.equals(ptransform.getSpec().getUrn()));
+  checkArgument(ptransform.getInputsCount() == 1, "Expected only one 
input");
+  checkArgument(ptransform.getOutputsCount() == 1, "Expected only one 
output");
+  WindowIntoPayload payload = 
WindowIntoPayload.parseFrom(ptransform.getSpec().getPayload());
+
+  WindowFn windowFn =
+  (WindowFn) 
WindowingStrategyTranslation.windowFnFromProto(payload.getWindowFn());
+
+  return AssignWindowsRunner.create(windowFn)::assignWindows;
+}
+  }
+
+  

+
+  static  AssignWindowsRunner create(
+  WindowFn windowFn) {
+// Safe contravariant cast
+WindowFn typedWindowFn = (WindowFn) windowFn;
+return new 

[beam] branch master updated (060c512 -> f0aa75f)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 060c512  Merge pull request #5355: Fix error-prone warnings and 
multiple cleanups based on static analysis for CassandraIO
 add f4ecff1  Enable MapFnRunner to provide full WindowedValues
 add 0530cb7  Implement AssignWindows in the Java SDK Harness
 new f0aa75f  [BEAM-4269, BEAM-3970] Implement AssignWindows in the Java 
SDK harness

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/fn/harness/AssignWindowsRunner.java   | 111 +++
 .../org/apache/beam/fn/harness/MapFnRunner.java| 120 ---
 .../org/apache/beam/fn/harness/MapFnRunners.java   | 180 ++
 .../beam/fn/harness/WindowMappingFnRunner.java |  19 +-
 .../beam/fn/harness/AssignWindowsRunnerTest.java   | 369 +
 .../apache/beam/fn/harness/MapFnRunnerTest.java|  97 --
 .../apache/beam/fn/harness/MapFnRunnersTest.java   | 185 +++
 7 files changed, 855 insertions(+), 226 deletions(-)
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/AssignWindowsRunner.java
 delete mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunner.java
 create mode 100644 
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/MapFnRunners.java
 create mode 100644 
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/AssignWindowsRunnerTest.java
 delete mode 100644 
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnerTest.java
 create mode 100644 
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/MapFnRunnersTest.java

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: [BEAM-4269, BEAM-3970] Implement AssignWindows in the Java SDK harness

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f0aa75fbd8e07f438d0662f297bed18194adcc8b
Merge: 060c512 0530cb7
Author: Lukasz Cwik 
AuthorDate: Mon May 14 15:06:54 2018 -0700

[BEAM-4269, BEAM-3970] Implement AssignWindows in the Java SDK harness

 .../beam/fn/harness/AssignWindowsRunner.java   | 111 +++
 .../org/apache/beam/fn/harness/MapFnRunner.java| 120 ---
 .../org/apache/beam/fn/harness/MapFnRunners.java   | 180 ++
 .../beam/fn/harness/WindowMappingFnRunner.java |  19 +-
 .../beam/fn/harness/AssignWindowsRunnerTest.java   | 369 +
 .../apache/beam/fn/harness/MapFnRunnerTest.java|  97 --
 .../apache/beam/fn/harness/MapFnRunnersTest.java   | 185 +++
 7 files changed, 855 insertions(+), 226 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #387

[Apache Jenkins Server]

See 


Changes:

[lcwik] [BEAM-4280] Prevent DirectStreamObserver from blocking indefinitely if

--
[...truncated 18.90 MB...]
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-05-14_14_59_51-16022520619779808760
May 14, 2018 9:59:52 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-05-14_14_59_51-16022520619779808760 with 0 
expected assertions.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:51.454Z: Autoscaling is enabled for job 
2018-05-14_14_59_51-16022520619779808760. The number of workers will be between 
1 and 1000.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:51.488Z: Autoscaling was automatically enabled for 
job 2018-05-14_14_59_51-16022520619779808760.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:53.982Z: Checking required Cloud APIs are enabled.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:54.157Z: Checking permissions granted to controller 
Service Account.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:58.373Z: Worker configuration: n1-standard-1 in 
us-central1-b.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:58.740Z: Expanding CoGroupByKey operations into 
optimizable parts.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:58.909Z: Expanding GroupByKey operations into 
optimizable parts.
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:58.942Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.150Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.175Z: Elided trivial flatten 
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.205Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.240Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.272Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.298Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.322Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
May 14, 2018 10:00:01 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-05-14T21:59:59.346Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4127?focusedWorklogId=101905=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101905
 ]

ASF GitHub Bot logged work on BEAM-4127:


Author: ASF GitHub Bot
Created on: 14/May/18 21:58
Start Date: 14/May/18 21:58
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5358: 
[BEAM-4127] Flink streaming translation for portable pipeline.
URL: https://github.com/apache/beam/pull/5358#discussion_r188110775
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import com.google.common.collect.Iterables;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItem;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItemCoder;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.util.Collector;
+
+/**
+ * Translate an unbounded portable pipeline representation into a Flink 
pipeline representation.
+ */
+public class FlinkStreamingPortablePipelineTranslator implements 
FlinkPortablePipelineTranslator<
+FlinkStreamingPortablePipelineTranslator.StreamingTranslationContext> {
+
+  /**
+   * Creates a streaming translation context. The resulting Flink execution 
dag will live in a new
+   * {@link 

[jira] [Work logged] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4127?focusedWorklogId=101901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101901
 ]

ASF GitHub Bot logged work on BEAM-4127:


Author: ASF GitHub Bot
Created on: 14/May/18 21:58
Start Date: 14/May/18 21:58
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5358: 
[BEAM-4127] Flink streaming translation for portable pipeline.
URL: https://github.com/apache/beam/pull/5358#discussion_r188109842
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import com.google.common.collect.Iterables;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItem;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItemCoder;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.util.Collector;
+
+/**
+ * Translate an unbounded portable pipeline representation into a Flink 
pipeline representation.
+ */
+public class FlinkStreamingPortablePipelineTranslator implements 
FlinkPortablePipelineTranslator<
+FlinkStreamingPortablePipelineTranslator.StreamingTranslationContext> {
+
+  /**
+   * Creates a streaming translation context. The resulting Flink execution 
dag will live in a new
+   * {@link 

[jira] [Work logged] (BEAM-4127) Flink portable runner translates streaming pipelines by proto

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4127?focusedWorklogId=101903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-101903
 ]

ASF GitHub Bot logged work on BEAM-4127:


Author: ASF GitHub Bot
Created on: 14/May/18 21:58
Start Date: 14/May/18 21:58
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #5358: 
[BEAM-4127] Flink streaming translation for portable pipeline.
URL: https://github.com/apache/beam/pull/5358#discussion_r188107869
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingPortablePipelineTranslator.java
 ##
 @@ -0,0 +1,443 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import com.google.common.collect.Iterables;
+import com.google.protobuf.InvalidProtocolBufferException;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import org.apache.beam.runners.core.SystemReduceFn;
+import org.apache.beam.runners.core.construction.PTransformTranslation;
+import org.apache.beam.runners.core.construction.PipelineOptionsTranslation;
+import org.apache.beam.runners.core.construction.RehydratedComponents;
+import org.apache.beam.runners.core.construction.WindowingStrategyTranslation;
+import org.apache.beam.runners.core.construction.graph.ExecutableStage;
+import org.apache.beam.runners.core.construction.graph.PipelineNode;
+import org.apache.beam.runners.core.construction.graph.QueryablePipeline;
+import org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
+import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItem;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItemCoder;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WorkItemKeySelector;
+import org.apache.beam.runners.fnexecution.provisioning.JobInfo;
+import org.apache.beam.runners.fnexecution.wire.WireCoders;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.IterableCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.VoidCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowedValue.WindowedValueCoder;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.WindowingStrategy;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.DataStreamSource;
+import org.apache.flink.streaming.api.datastream.KeyedStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.util.Collector;
+
+/**
+ * Translate an unbounded portable pipeline representation into a Flink 
pipeline representation.
+ */
+public class FlinkStreamingPortablePipelineTranslator implements 
FlinkPortablePipelineTranslator<
+FlinkStreamingPortablePipelineTranslator.StreamingTranslationContext> {
+
+  /**
+   * Creates a streaming translation context. The resulting Flink execution 
dag will live in a new
+   * {@link