[GitHub] incubator-beam-site pull request #48: Stage pull requests in Google Cloud St...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam-site/pull/48

Stage pull requests in Google Cloud Storage

This is work-in-progress. Several aspects need to be taken care of.

R: @francesperry 
CC: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site buildpr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/48.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #48


commit 7e294ba42056e47e729b31354d21eeed89302084
Author: Davor Bonaci 
Date:   2016-10-19T03:41:29Z

Stage pull requests in Google Cloud Storage

This is work-in-progress. Several aspects need to be taken care of.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-777) KafkaIO Test should handle reader.start() better

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-777:
-

 Summary: KafkaIO Test should handle reader.start() better
 Key: BEAM-777
 URL: https://issues.apache.org/jira/browse/BEAM-777
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Minor


KafkaIOTest currently expects reader.start() to return true. It can return 
false as well. It should handle it better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[4/8] incubator-beam-site git commit: Add Design Principles (take from the original Beam technical vision document).

[frances]

Add Design Principles (take from the original Beam technical vision document).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/99783418
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/99783418
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/99783418

Branch: refs/heads/asf-site
Commit: 997834188ecf29b307e195c9c7e8d31fa60b34ff
Parents: 7f234a5
Author: Frances Perry 
Authored: Mon Oct 3 19:00:03 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:39 2016 -0700

--
 _includes/header.html   |  5 ++--
 contribute/design-principles.md | 53 
 2 files changed, 56 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/99783418/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 182b30a..67631a9 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -63,12 +63,13 @@
  
  Basics
  Contribution Guide
- Testing
  Mailing Lists
   Source Repository
   Issue 
Tracking
   
- Technical Resources
+ Technical 
References
+ Testing
+  Design Principles
  https://goo.gl/nk5OM0;>Technical 
Vision
  


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/99783418/contribute/design-principles.md
--
diff --git a/contribute/design-principles.md b/contribute/design-principles.md
new file mode 100644
index 000..87ddd24
--- /dev/null
+++ b/contribute/design-principles.md
@@ -0,0 +1,53 @@
+---
+layout: default
+title: 'Design Principles in Beam'
+permalink: /contribute/design-principles/
+---
+
+# Design Principles in the Apache Beam Project
+
+Joshua Bloch’s [API Design Bumper 
Stickers](https://www.infoq.com/articles/API-Design-Joshua-Bloch) are a great 
list of what makes for good API design. In addition, we have specific design 
principles we follow in Beam.
+
+* TOC
+{:toc}
+
+## Use cases
+
+### Unify the model
+Provide one model that works over both bounded (aka. batch) and unbounded 
(aka. streaming) datasets. Pay special attention to windows / triggers / state 
/ timers, which often trip up folks used to a batch world.  Provide users with 
the right abstractions to adjust latency and completeness guarantees to cover 
both traditional batch and streaming use cases. 
+
+### Separate data shapes and runtime requirements
+The model should focus on letting users describe their data and processing, 
without exposing any details of a specific runtime system. For example, bounded 
and unbounded describe the shape of data, but batch and streaming describe the 
behavior of specific runtime systems. Good test cases are to imagine a mythical 
micro-batching runner that sits somewhere between batch and streaming or a 
engine that dynamically switches between streaming and batch depending on the 
backlog.
+
+### Make efficient things easy, rather than make easy things efficient
+Don’t prevent efficiency for ease of use. Design APIs that provide the 
information necessary for efficiently executing at scale. Provide class 
hierarchies and wrappers to make the common cases simpler.
+
+## Usability
+
+### Validate Early
+Validate constraints on graph shape, runner requirements, etc as early in the 
compile time - construction time - submission time - execution time spectrum as 
reasonably possible in order to provide a smoother user experience.
+
+### Public APIs, like diamonds, are forever (at least until the next major 
version)
+Backwards incompatible changes can only be made in the next major version. 
Because of the burden major versions place on users (code has to be modified, 
conflicting dependency nightmares, etc), we aim to do this infrequently. 
Clearly mark APIs that are considered experimental (may change at any point) 
and deprecated (will be removed in the next major version). Consider what APIs 
are more amenable to future changes (abstract classes vs. interfaces, etc.)
+
+### Examples should be pedagogical
+Canonical examples help people ingrain the principles. Design examples that 
teach complex concepts in modular chunks. If you can’t explain the concept 
easily, then the API isn’t right. Examples should withstand random 
copy-pasting. 
+
+## Extensibility
+
+### Use PTransforms for modularity
+Composite transformations 

[GitHub] incubator-beam-site pull request #42: Update Contribution Guide and add Work...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/8] incubator-beam-site git commit: Update mailing lists to use new lists site.

[frances]

Update mailing lists to use new lists site.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/7f234a5a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/7f234a5a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/7f234a5a

Branch: refs/heads/asf-site
Commit: 7f234a5a9314bbaf747e81ac069d3fcf6d3f0557
Parents: 29cc4a6
Author: Frances Perry 
Authored: Mon Oct 3 18:52:37 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:30 2016 -0700

--
 use/mailing-lists.md | 33 -
 1 file changed, 16 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/7f234a5a/use/mailing-lists.md
--
diff --git a/use/mailing-lists.md b/use/mailing-lists.md
index 0e28d71..5807781 100644
--- a/use/mailing-lists.md
+++ b/use/mailing-lists.md
@@ -7,17 +7,13 @@ redirect_from: /mailing_lists/
 
 # Apache Beam Mailing Lists
 
-These are the mailing lists that have been established for this project. For 
each list, there is a subscribe, unsubscribe, and an archive link.
-
-Name   Subscribe   Unsubscribe PostArchive
-   Subscribe   Unsubscribe Post
[mail-archives.apache.org](http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/)
-   Subscribe   Unsubscribe Postmail-archives.apache.org
-beam-user  Subscribe   Unsubscribe Postmail-archives.apache.org
+These are the mailing lists that have been established for this project.
 
 
   
 
   Name
+  Purpose
   Subscribe
   Unsubscribe
   Post
@@ -26,25 +22,28 @@ beam-user   Subscribe   Unsubscribe Post
mail-archives.apache.org
   
   
 
-  beam-dev
+  user@
+  User support and questions
+  mailto:user-subscr...@beam.incubator.apache.org;>Subscribe
+  mailto:user-unsubscr...@beam.incubator.apache.org;>Unsubscribe
+  mailto:u...@beam.incubator.apache.org;>Post
+  https://lists.apache.org/list.html?u...@beam.apache.org;>Archives
+
+
+  dev@
+  Development-related discussions
   mailto:dev-subscr...@beam.incubator.apache.org;>Subscribe
   mailto:dev-unsubscr...@beam.incubator.apache.org;>Unsubscribe
   mailto:d...@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/;>mail-archives.apache.org
+  https://lists.apache.org/list.html?d...@beam.apache.org;>Archives
 
 
-  beam-commits
+  commits@
+  JIRA updates, pull requests, etc.
   mailto:commits-subscr...@beam.incubator.apache.org;>Subscribe
   mailto:commits-unsubscr...@beam.incubator.apache.org;>Unsubscribe
   mailto:commits@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/;>mail-archives.apache.org
-
-
-  beam-user
-  mailto:user-subscr...@beam.incubator.apache.org;>Subscribe
-  mailto:user-unsubscr...@beam.incubator.apache.org;>Unsubscribe
-  mailto:u...@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-user/;>mail-archives.apache.org
+  https://lists.apache.org/list.html?comm...@beam.apache.org;>Archives
 
 
 

[5/8] incubator-beam-site git commit: Updated Contribution Guide & added Work In Progress * Added section on feature branches. * Revamped website section. * Standardized code formatting. * Added a "wo

[frances]

Updated Contribution Guide & added Work In Progress
* Added section on feature branches.
* Revamped website section.
* Standardized code formatting.
* Added a "work in progress" page to help make new and ongoing work more 
visible to the community.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c765b7ed
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c765b7ed
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c765b7ed

Branch: refs/heads/asf-site
Commit: c765b7edfcee3044ed21517a244b74ea8d8eded9
Parents: 229020c
Author: Frances Perry 
Authored: Mon Oct 3 20:57:24 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 21:03:25 2016 -0700

--
 README.md|   1 -
 _includes/header.html|   3 +-
 contribute/contribution-guide.md | 166 --
 contribute/work-in-progress.md   |  32 +++
 4 files changed, 150 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/README.md
--
diff --git a/README.md b/README.md
index 80819e2..40172ac 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,3 @@
-
 # Apache Beam (incubating) website
 
 This is the website for [Apache Beam](http://beam.incubator.apache.org)

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 67631a9..99d59eb 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -63,11 +63,12 @@
  
  Basics
  Contribution Guide
+ Work In Progress
  Mailing Lists
   Source Repository
   Issue 
Tracking
   
- Technical 
References
+ Technical References
  Testing
   Design Principles
  https://goo.gl/nk5OM0;>Technical 
Vision

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/contribute/contribution-guide.md
--
diff --git a/contribute/contribution-guide.md b/contribute/contribution-guide.md
index a0dcb50..61c762b 100644
--- a/contribute/contribution-guide.md
+++ b/contribute/contribution-guide.md
@@ -18,7 +18,7 @@ We use a review-then-commit workflow in Beam for all 
contributions.
 
 **For larger contributions or those that affect multiple components:**
 
-1. **Engage**: We encourage you to work with the Beam community on the [Apache 
JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and 
[developer’s mailing list](http://beam.incubator.apache.org/mailing_lists/) 
to identify good areas for contribution.
+1. **Engage**: We encourage you to work with the Beam community on the [Apache 
JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and 
[developer’s mailing 
list](http://beam.incubator.apache.org/use/mailing-lists/) to identify good 
areas for contribution.
 1. **Design:** More complicated contributions will likely benefit from some 
early discussion in order to scope and design them well.
 
 **For all contributions:**
@@ -32,27 +32,28 @@ We look forward to working with you!
 ## Engage
 
 ### Mailing list(s)
-We discuss design and implementation issues on d...@beam.incubator.apache.org 
mailing list, which is archived 
[here](http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/). Join by 
emailing 
[`dev-subscr...@beam.incubator.apache.org`](mailto:dev-subscr...@beam.incubator.apache.org).
+We discuss design and implementation issues on d...@beam.incubator.apache.org 
mailing list, which is archived 
[here](https://lists.apache.org/list.html?d...@beam.apache.org). Join by 
emailing 
[`dev-subscr...@beam.incubator.apache.org`](mailto:dev-subscr...@beam.incubator.apache.org).
 
-If interested, you can also join 
[`u...@beam.incubator.apache.org`](http://mail-archives.apache.org/mod_mbox/incubator-beam-user/)
 and 
[`commits@beam.incubator.apache.org`](http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/)
 too.
+If interested, you can also join the other [mailing 
lists](http://beam.incubator.apache.org/use/mailing-lists/) too.
 
 ### Apache JIRA
 We use [Apache JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue 
tracking and project management tool, as well as a way to communicate among a 
very diverse and distributed set of contributors. To be able to gather 
feedback, avoid 

[3/8] incubator-beam-site git commit: Remove page title from TOCs

[frances]

Remove page title from TOCs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/229020ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/229020ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/229020ce

Branch: refs/heads/asf-site
Commit: 229020ceac325fb5cb40f91dcd1f9d8458510999
Parents: 9978341
Author: Frances Perry 
Authored: Mon Oct 3 21:46:23 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:39 2016 -0700

--
 _config.yml | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/229020ce/_config.yml
--
diff --git a/_config.yml b/_config.yml
index 39fdbcd..39cf260 100644
--- a/_config.yml
+++ b/_config.yml
@@ -41,6 +41,10 @@ exclude: ['README.md', 'Gemfile.lock', 'Gemfile', 'Rakefile']
 # Downloads directory
 downloads: downloads
 
+# Don't use the page title in the table of contents
+kramdown:
+  toc_levels: 2..6
+
 gems:
   - jekyll-redirect-from
   
\ No newline at end of file

[jira] [Assigned] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian reassigned BEAM-659:


Assignee: Neelesh Srinivas Salian

> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: easy, easyfix, starter
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_PythonVerify #567

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-774) Implement Metrics support for Spark runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-774:
-
Summary: Implement Metrics support for Spark runner  (was: Implement 
Metrics support for Spark runenr)

> Implement Metrics support for Spark runner
> --
>
> Key: BEAM-774
> URL: https://issues.apache.org/jira/browse/BEAM-774
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Ben Chambers
>Assignee: Amit Sela
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1574

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1574

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1132: Close threadpools when finished with them

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/1132

Close threadpools when finished with them

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This avoids building up an arbitrary number of dangling threads,
which can cause issues in testing and is undesirable in production.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam threadpools

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1132


commit 4adccb28d84fda8791f58e5e59ef24509f3dc81f
Author: Robert Bradshaw 
Date:   2016-10-19T00:08:18Z

Close threadpools when finished with them

This avoids building up an arbitrary number of dangling threads,
which can cause issues in testing and is undesirable in production.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-517) Check versions of pip and cython

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-517.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Check versions of pip and cython
> 
>
> Key: BEAM-517
> URL: https://issues.apache.org/jira/browse/BEAM-517
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>
> Python SDK depends on pip and cython however it does not check the versions 
> of these.
> Some of the pip flags does not exist in older versions:
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/28#issuecomment-236382953
> (Note: Even though the above issue was reported by the user in a different 
> repo it is related to the apache beam sdk)
> Similarly with cython, SDK supports running with or without Cython. Because 
> of that reason it is not list it as a requirement in the setup.py file. 
> However, with an old version of cython SDK might fail.
> To avoid the above problem: In the SDK check the version of these packages 
> and show a warning to upgrade.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-562) DoFn Reuse: Add new methods to DoFn

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-562:
-
Labels: sdk-consistency  (was: )

> DoFn Reuse: Add new methods to DoFn
> ---
>
> Key: BEAM-562
> URL: https://issues.apache.org/jira/browse/BEAM-562
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
>
> Java SDK added setup and teardown methods to the DoFns. This makes DoFns 
> reusable and provide performance improvements. Python SDK should add support 
> for these new DoFn methods:
> Proposal doc: 
> https://docs.google.com/document/d/1LLQqggSePURt3XavKBGV7SZJYQ4NW8yCu63lBchzMRk/edit?ts=5771458f#



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-681) DoFns should be serialized at apply time and deserialized when executing

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-681:
-
Labels: sdk-consistency  (was: )

> DoFns should be serialized at apply time and deserialized when executing
> 
>
> Key: BEAM-681
> URL: https://issues.apache.org/jira/browse/BEAM-681
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ben Chambers
>Assignee: Frances Perry
>  Labels: sdk-consistency
>
> 1. Serializing DoFns at application time ensures that any modifications of 
> fields within the DoFn after application do not accidentally pollute the 
> execution. This mirrors the approach taken in Java to provide an 
> approximation of lexical-closure (eg., you only need to know the state of the 
> DoFn at the time it was applied, not afterwards, to understand its behavior).
> 2. Based on 1, the DIrectRunner should also be deserializing DoFns before 
> running them, which should also detect other classes of errors such as using 
> the pipeline object (which is not pickleable) within the DoFn



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-759:
-
Labels: sdk-consistency  (was: )

> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-753) Pin versions of all dependencies

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-753:
-
Labels:   (was: sdk-consistency)

> Pin versions of all dependencies
> 
>
> Key: BEAM-753
> URL: https://issues.apache.org/jira/browse/BEAM-753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: Not applicable
>
>
> ERROR: Failure: ImportError (cannot import name locked_file)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
>  line 31, in 
> from apache_beam.io import concat_source
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
>  line 24, in 
> from apache_beam.io import iobase
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
>  line 818, in 
> from apache_beam.runners.dataflow.native_io.iobase import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
>  line 23, in 
> from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 43, in 
> from apache_beam.internal.clients import dataflow as dataflow_api
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
>  line 23, in 
> from apitools.base.py import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 22, in 
> from apitools.base.py.credentials_lib import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
>  line 50, in 
> from oauth2client import locked_file
> ImportError: cannot import name locked_file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-753) Pin versions of all dependencies

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-753:
-
Labels: sdk-consistency  (was: )

> Pin versions of all dependencies
> 
>
> Key: BEAM-753
> URL: https://issues.apache.org/jira/browse/BEAM-753
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
> Fix For: Not applicable
>
>
> ERROR: Failure: ImportError (cannot import name locked_file)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/nose-1.3.7-py2.7.egg/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/__init__.py",
>  line 78, in 
> from apache_beam import io
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/__init__.py",
>  line 21, in 
> from apache_beam.io.avroio import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/avroio.py",
>  line 29, in 
> from apache_beam.io import filebasedsource
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/filebasedsource.py",
>  line 31, in 
> from apache_beam.io import concat_source
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/concat_source.py",
>  line 24, in 
> from apache_beam.io import iobase
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/io/iobase.py",
>  line 818, in 
> from apache_beam.runners.dataflow.native_io.iobase import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/__init__.py",
>  line 23, in 
> from apache_beam.runners.dataflow_runner import DataflowPipelineRunner
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/runners/dataflow_runner.py",
>  line 43, in 
> from apache_beam.internal.clients import dataflow as dataflow_api
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/apache_beam/internal/clients/dataflow/__init__.py",
>  line 23, in 
> from apitools.base.py import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/__init__.py",
>  line 22, in 
> from apitools.base.py.credentials_lib import *
>   File 
> "/usr/local/google/home/altay/Desktop/beam/test/incubator-beam/sdks/python/.tox/py27/local/lib/python2.7/site-packages/apitools/base/py/credentials_lib.py",
>  line 50, in 
> from oauth2client import locked_file
> ImportError: cannot import name locked_file



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1131: Better error for missing job name

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/incubator-beam/pull/1131

Better error for missing job name

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam job-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1131.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1131


commit 30056332d3de72b5c23df0219cbbd7e8a218f5c1
Author: Robert Bradshaw 
Date:   2016-10-18T23:15:12Z

Better error for missing job name




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-775) Remove Aggregators from the Java SDK

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-775:
-

 Summary: Remove Aggregators from the Java SDK
 Key: BEAM-775
 URL: https://issues.apache.org/jira/browse/BEAM-775
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Ben Chambers
Assignee: Ben Chambers






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-774) Implement Metrics support for Spark runenr

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-774:
-

 Summary: Implement Metrics support for Spark runenr
 Key: BEAM-774
 URL: https://issues.apache.org/jira/browse/BEAM-774
 Project: Beam
  Issue Type: Sub-task
  Components: runner-spark
Reporter: Ben Chambers
Assignee: Amit Sela






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-458) Support for Flink Metrics

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers closed BEAM-458.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> Support for Flink Metrics 
> --
>
> Key: BEAM-458
> URL: https://issues.apache.org/jira/browse/BEAM-458
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Sumit Chawla
> Fix For: Not applicable
>
>
> Flink has added support for CodeHale Metrics 
> (https://ci.apache.org/projects/flink/flink-docs-master/monitoring/metrics.html)
> These metrics are more advanced then the current Accumulators. 
> Adding support for these to Beam level should be a good addition.
> https://github.com/apache/flink/pull/1947#issuecomment-233029166



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-773) Implement Metrics support for Flink runner

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-773:
-

 Summary: Implement Metrics support for Flink runner
 Key: BEAM-773
 URL: https://issues.apache.org/jira/browse/BEAM-773
 Project: Beam
  Issue Type: Sub-task
  Components: runner-flink
Reporter: Ben Chambers






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-772) Implement Metrics support for Dataflow Runner

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-772:
-

 Summary: Implement Metrics support for Dataflow Runner
 Key: BEAM-772
 URL: https://issues.apache.org/jira/browse/BEAM-772
 Project: Beam
  Issue Type: Sub-task
  Components: runner-dataflow
Reporter: Ben Chambers
Assignee: Ben Chambers






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-764) Remove cloneAs from PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586910#comment-15586910
 ] 

ASF GitHub Bot commented on BEAM-764:
-

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1120


> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>  Labels: codehealth
> Fix For: 0.3.0-incubating
>
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1120: [BEAM-764] Remove cloneAs from PipelineOp...

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/incubator-beam/pull/1120


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-764) Remove cloneAs from PipelineOptions

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586902#comment-15586902
 ] 

Luke Cwik commented on BEAM-764:


This was merged into master here:
https://github.com/apache/incubator-beam/commit/71c69b31b6894064bf8111007f947150ff725528

> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>  Labels: codehealth
> Fix For: 0.3.0-incubating
>
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-764) Remove cloneAs from PipelineOptions

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-764:
---
Priority: Minor  (was: Major)

> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>  Labels: codehealth
> Fix For: 0.3.0-incubating
>
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-764) Remove cloneAs from PipelineOptions

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-764.

   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>  Labels: codehealth
> Fix For: 0.3.0-incubating
>
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-27) Add user-ready API for interacting with timers

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586870#comment-15586870
 ] 

ASF GitHub Bot commented on BEAM-27:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1130

[BEAM-27] Add parsing and validation of @StateId params to @ProcessElement

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers @jkff 

This is an initial draft to discuss some design decisions along the way.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam State-params

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1130


commit 33c884118ed04c0a008ef6018a06fa336a866de8
Author: Kenneth Knowles 
Date:   2016-10-14T17:51:57Z

Use the correct State class in DoFnSignatures

commit 81b6994bf8162bfc3aaba40a708ce8385be76d80
Author: Kenneth Knowles 
Date:   2016-10-14T18:07:55Z

Add analysis and validation of State parameters to DoFn.ProcessElement




> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1130: [BEAM-27] Add parsing and validation of @...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1130

[BEAM-27] Add parsing and validation of @StateId params to @ProcessElement

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers @jkff 

This is an initial draft to discuss some design decisions along the way.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam State-params

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1130


commit 33c884118ed04c0a008ef6018a06fa336a866de8
Author: Kenneth Knowles 
Date:   2016-10-14T17:51:57Z

Use the correct State class in DoFnSignatures

commit 81b6994bf8162bfc3aaba40a708ce8385be76d80
Author: Kenneth Knowles 
Date:   2016-10-14T18:07:55Z

Add analysis and validation of State parameters to DoFn.ProcessElement




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_PythonVerify #566

[Apache Jenkins Server]

See 

--
[...truncated 2850 lines...]
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/WriteImpl/ViewAsIterable(write|WriteImpl|FlatMap().None)/CreatePCollectionView.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 
"write/WriteImpl/ViewAsIterable(write|WriteImpl|FlatMap().None)/CreatePCollectionView"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"non_parallel_inputs": {
  "s15": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "s15"
  }, 
  "s9": {
"@type": "OutputReference", 
"output_name": "out", 
"step_name": "s9"
  }
}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/WriteImpl/finalize_write.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s7"
}, 
"serialized_fn": "", 
"user_name": "write/WriteImpl/finalize_write"
  }
}
  ], 
  "type": "JOB_TYPE_BATCH"
}
INFO:root:Create job: 
INFO:root:Created job with id: [2016-10-18_14_04_17-7862377147606629765]
INFO:root:To access the Dataflow monitoring console, please navigate to 
https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2016-10-18_14_04_17-7862377147606629765
INFO:root:Job 2016-10-18_14_04_17-7862377147606629765 is in state 
JOB_STATE_RUNNING
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf356: 
2016-10-18T21:04:18.774Z: JOB_MESSAGE_DETAILED: (2787e482e4005483): Checking 
required Cloud APIs are enabled.
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf467: 
2016-10-18T21:04:19.047Z: JOB_MESSAGE_DEBUG: (2787e482e4005f84): Combiner 
lifting skipped for step write/WriteImpl/GroupByKey: GroupByKey not followed by 
a combiner.
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf46a: 
2016-10-18T21:04:19.050Z: JOB_MESSAGE_DEBUG: (2787e482e40051f6): Combiner 
lifting skipped for step group: GroupByKey not followed by a combiner.
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf46c: 
2016-10-18T21:04:19.052Z: JOB_MESSAGE_DETAILED: (2787e482e4005468): Expanding 
GroupByKey operations into optimizable parts.
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf46f: 
2016-10-18T21:04:19.055Z: JOB_MESSAGE_DETAILED: (2787e482e40056da): Lifting 
ValueCombiningMappingFns into MergeBucketsMappingFns
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf476: 
2016-10-18T21:04:19.062Z: JOB_MESSAGE_DETAILED: (2787e482e4005e30): Annotating 
graph with Autotuner information.
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf497: 
2016-10-18T21:04:19.095Z: JOB_MESSAGE_DETAILED: (2787e482e40057f8): Fusing 
adjacent ParDo, Read, Write, and Flatten operations
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf49b: 
2016-10-18T21:04:19.099Z: JOB_MESSAGE_DETAILED: (2787e482e4005a6a): Fusing 
consumer split into read
INFO:root:2016-10-18_14_04_17-7862377147606629765_0157d99bf49d: 
2016-10-18T21:04:19.101Z: 

[GitHub] incubator-beam pull request #1129: Pin oauth2client to <4.0.0dev.

[charlesccychen]

GitHub user charlesccychen opened a pull request:

https://github.com/apache/incubator-beam/pull/1129

Pin oauth2client to <4.0.0dev.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/charlesccychen/incubator-beam pin-oauth2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1129.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1129


commit 8a8811194fbc131f3fb627d9bb0992d68b6b173e
Author: Charles Chen 
Date:   2016-10-18T21:05:40Z

Pin oauth2client to <4.0.0dev.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Build failed in Jenkins: beam_PostCommit_PythonVerify #565

[Robert Bradshaw]

I'm looking into this.

On Tue, Oct 18, 2016 at 1:53 PM, Apache Jenkins Server
 wrote:
> See 
>
> Changes:
>
> [robertwb] Windowed side input test.
>
> [robertwb] Implement windowed side inputs for direct runner.
>
> [robertwb] Fix tests expecting list from AsIter.
>
> [robertwb] Implement windowed side inputs for InProcess runner.
>
> [robertwb] More complicated window tests.
>
> [robertwb] Optimize globally windowed side input case
>
> [robertwb] Minor fixups for better testing
>
> [robertwb] Rename from_iterable to avoid confusion.
>
> --
> [...truncated 2889 lines...]
> },
> {
>   "@type": 
> "FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>   "component_encodings": []
> }
>   ],
>   "is_pair_like": true
> },
> {
>   "@type": 
> "TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
>   "component_encodings": []
> },
> {
>   "@type": "SingletonCoder$",
>   "component_encodings": []
> }
>   ],
>   "is_wrapper": true
> },
> "output_name": "out",
> "user_name": "write/WriteImpl/finalize_write.out"
>   }
> ],
> "parallel_input": {
>   "@type": "OutputReference",
>   "output_name": "out",
>   "step_name": "s7"
> },
> "serialized_fn": "",
> "user_name": "write/WriteImpl/finalize_write"
>   }
> }
>   ],
>   "type": "JOB_TYPE_BATCH"
> }
> INFO:root:Create job:   id: u'2016-10-18_13_48_20-13116714018908792973'
>  projectId: u'apache-beam-testing'
>  steps: []
>  tempFiles: []
>  type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)>
> INFO:root:Created job with id: [2016-10-18_13_48_20-13116714018908792973]
> INFO:root:To access the Dataflow monitoring console, please navigate to 
> https://console.developers.google.com/project/apache-beam-testing/dataflow/job/2016-10-18_13_48_20-13116714018908792973
> INFO:root:Job 2016-10-18_13_48_20-13116714018908792973 is in state 
> JOB_STATE_RUNNING
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d55e6: 
> 2016-10-18T20:48:20.966Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade31d): Checking 
> required Cloud APIs are enabled.
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5700: 
> 2016-10-18T20:48:21.248Z: JOB_MESSAGE_DEBUG: (cc0f1c724dadedf5): Combiner 
> lifting skipped for step write/WriteImpl/GroupByKey: GroupByKey not followed 
> by a combiner.
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5702: 
> 2016-10-18T20:48:21.250Z: JOB_MESSAGE_DEBUG: (cc0f1c724dade8ab): Combiner 
> lifting skipped for step group: GroupByKey not followed by a combiner.
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5704: 
> 2016-10-18T20:48:21.252Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade361): Expanding 
> GroupByKey operations into optimizable parts.
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5707: 
> 2016-10-18T20:48:21.255Z: JOB_MESSAGE_DETAILED: (cc0f1c724dadee17): Lifting 
> ValueCombiningMappingFns into MergeBucketsMappingFns
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d570e: 
> 2016-10-18T20:48:21.262Z: JOB_MESSAGE_DETAILED: (cc0f1c724dadee39): 
> Annotating graph with Autotuner information.
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d573d: 
> 2016-10-18T20:48:21.309Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade911): Fusing 
> adjacent ParDo, Read, Write, and Flatten operations
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5740: 
> 2016-10-18T20:48:21.312Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade3c7): Fusing 
> consumer split into read
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5743: 
> 2016-10-18T20:48:21.315Z: JOB_MESSAGE_DETAILED: (cc0f1c724dadee7d): Fusing 
> consumer group/Reify into pair_with_one
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5746: 
> 2016-10-18T20:48:21.318Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade933): Fusing 
> consumer format into count
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d5749: 
> 2016-10-18T20:48:21.321Z: JOB_MESSAGE_DETAILED: (cc0f1c724dade3e9): Fusing 
> consumer write/WriteImpl/GroupByKey/GroupByWindow into 
> write/WriteImpl/GroupByKey/Read
> INFO:root:2016-10-18_13_48_20-13116714018908792973_0157d98d574b: 
> 2016-10-18T20:48:21.323Z: JOB_MESSAGE_DETAILED: (cc0f1c724dadee9f): Fusing 
> consumer write/WriteImpl/GroupByKey/Write into 
> write/WriteImpl/GroupByKey/Reify
> 

[jira] [Resolved] (BEAM-771) UnboundedReadEvaluator should reuse the input reader if it contains no elements

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-771.
--
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> UnboundedReadEvaluator should reuse the input reader if it contains no 
> elements
> ---
>
> Key: BEAM-771
> URL: https://issues.apache.org/jira/browse/BEAM-771
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: 0.3.0-incubating
>
>
> The UnboundedReadEvaluator checks, after it obtains the reader, that the 
> reader contains elements, and terminates without taking a new checkpoint if 
> not. It must also (but currently does not) add that reader and shard back as 
> a residual element.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1128: [BEAM-771] Add a Residual Shard for Empty...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1128


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-771) UnboundedReadEvaluator should reuse the input reader if it contains no elements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586628#comment-15586628
 ] 

ASF GitHub Bot commented on BEAM-771:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1128


> UnboundedReadEvaluator should reuse the input reader if it contains no 
> elements
> ---
>
> Key: BEAM-771
> URL: https://issues.apache.org/jira/browse/BEAM-771
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The UnboundedReadEvaluator checks, after it obtains the reader, that the 
> reader contains elements, and terminates without taking a new checkpoint if 
> not. It must also (but currently does not) add that reader and shard back as 
> a residual element.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1128

[dhalperi]

Closes #1128


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3b1c2a3c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3b1c2a3c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3b1c2a3c

Branch: refs/heads/master
Commit: 3b1c2a3cf46f79657f0e010868188abe44d708c8
Parents: a18c274 97c8949
Author: Dan Halperin 
Authored: Tue Oct 18 13:54:10 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 13:54:10 2016 -0700

--
 .../direct/UnboundedReadEvaluatorFactory.java   | 10 
 .../UnboundedReadEvaluatorFactoryTest.java  | 52 
 2 files changed, 62 insertions(+)
--

[8/9] incubator-beam git commit: Rename from_iterable to avoid confusion.

[robertwb]

Rename from_iterable to avoid confusion.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cf15ab51
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cf15ab51
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cf15ab51

Branch: refs/heads/python-sdk
Commit: cf15ab51efa9c8bb21f00eee255f0c57e41a8e7b
Parents: a50efcc
Author: Robert Bradshaw 
Authored: Tue Oct 18 12:31:30 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:31:30 2016 -0700

--
 sdks/python/apache_beam/pvalue.py| 8 
 sdks/python/apache_beam/transforms/sideinputs.py | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cf15ab51/sdks/python/apache_beam/pvalue.py
--
diff --git a/sdks/python/apache_beam/pvalue.py 
b/sdks/python/apache_beam/pvalue.py
index e40d746..4cfdfbe 100644
--- a/sdks/python/apache_beam/pvalue.py
+++ b/sdks/python/apache_beam/pvalue.py
@@ -268,7 +268,7 @@ class SingletonPCollectionView(PCollectionView):
   return base
 
   @staticmethod
-  def from_iterable(it, options):
+  def _from_runtime_iterable(it, options):
 head = list(itertools.islice(it, 2))
 if len(head) == 0:
   return options.get('default', EmptySideInput())
@@ -284,7 +284,7 @@ class IterablePCollectionView(PCollectionView):
   """A PCollectionView that can be treated as an iterable."""
 
   @staticmethod
-  def from_iterable(it, options):
+  def _from_runtime_iterable(it, options):
 return it
 
 
@@ -292,7 +292,7 @@ class ListPCollectionView(PCollectionView):
   """A PCollectionView that can be treated as a list."""
 
   @staticmethod
-  def from_iterable(it, options):
+  def _from_runtime_iterable(it, options):
 return list(it)
 
 
@@ -300,7 +300,7 @@ class DictPCollectionView(PCollectionView):
   """A PCollectionView that can be treated as a dict."""
 
   @staticmethod
-  def from_iterable(it, options):
+  def _from_runtime_iterable(it, options):
 return dict(it)
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cf15ab51/sdks/python/apache_beam/transforms/sideinputs.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs.py 
b/sdks/python/apache_beam/transforms/sideinputs.py
index f3a7178..05ba6ab 100644
--- a/sdks/python/apache_beam/transforms/sideinputs.py
+++ b/sdks/python/apache_beam/transforms/sideinputs.py
@@ -188,7 +188,7 @@ class SideInputMap(object):
   def __getitem__(self, window):
 if window not in self._cache:
   target_window = self._window_mapping_fn(window)
-  self._cache[window] = self._view_class.from_iterable(
+  self._cache[window] = self._view_class._from_runtime_iterable(
   _FilteringIterable(self._iterable, target_window), 
self._view_options)
 return self._cache[window]
 

[GitHub] incubator-beam pull request #1123: Add support for experiments flag

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1123


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[9/9] incubator-beam git commit: Closes #1100

[robertwb]

Closes #1100


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8e1793ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8e1793ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8e1793ca

Branch: refs/heads/python-sdk
Commit: 8e1793cafa1839456ae007937bc581764749a006
Parents: 6e6d89d cf15ab5
Author: Robert Bradshaw 
Authored: Tue Oct 18 13:12:56 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 13:12:56 2016 -0700

--
 .../examples/cookbook/bigquery_side_input.py|   3 +-
 sdks/python/apache_beam/pvalue.py   |  44 ++--
 sdks/python/apache_beam/runners/common.pxd  |   1 +
 sdks/python/apache_beam/runners/common.py   |  51 +++--
 .../python/apache_beam/runners/direct_runner.py |  30 +-
 .../inprocess/inprocess_evaluation_context.py   |  29 +
 .../python/apache_beam/transforms/sideinputs.py |  75 +++--
 .../apache_beam/transforms/sideinputs_test.py   | 105 +--
 sdks/python/apache_beam/transforms/timeutil.py  |   4 +
 sdks/python/apache_beam/transforms/util.py  |   5 +-
 sdks/python/apache_beam/transforms/window.py|   3 +
 .../transforms/write_ptransform_test.py |   2 +-
 12 files changed, 258 insertions(+), 94 deletions(-)
--

[1/9] incubator-beam git commit: Fix tests expecting list from AsIter.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 6e6d89d48 -> 8e1793caf


Fix tests expecting list from AsIter.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/66b4c2f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/66b4c2f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/66b4c2f1

Branch: refs/heads/python-sdk
Commit: 66b4c2f160b5fdce6aca9d5ea50b406fcb07bb51
Parents: 29a7378
Author: Robert Bradshaw 
Authored: Thu Oct 13 15:15:06 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:17:15 2016 -0700

--
 .../python/apache_beam/examples/cookbook/bigquery_side_input.py | 2 +-
 sdks/python/apache_beam/transforms/util.py  | 5 +++--
 sdks/python/apache_beam/transforms/write_ptransform_test.py | 2 +-
 3 files changed, 5 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/66b4c2f1/sdks/python/apache_beam/examples/cookbook/bigquery_side_input.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/bigquery_side_input.py 
b/sdks/python/apache_beam/examples/cookbook/bigquery_side_input.py
index 2099e48..ffba786 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigquery_side_input.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigquery_side_input.py
@@ -72,7 +72,7 @@ def create_groups(group_ids, corpus, word, ignore_corpus, 
ignore_word):
   | beam.FlatMap(
   'attach word',
   attach_word_fn,
-  AsIter(word),
+  AsList(word),
   AsSingleton(ignore_word)))
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/66b4c2f1/sdks/python/apache_beam/transforms/util.py
--
diff --git a/sdks/python/apache_beam/transforms/util.py 
b/sdks/python/apache_beam/transforms/util.py
index aeac0d9..ebe6ba9 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -20,7 +20,7 @@
 
 from __future__ import absolute_import
 
-from apache_beam.pvalue import AsIter as AllOf
+from apache_beam.pvalue import AsList
 from apache_beam.transforms import core
 from apache_beam.transforms import window
 from apache_beam.transforms.core import CombinePerKey, Create, Flatten, 
GroupByKey, Map
@@ -192,6 +192,7 @@ def equal_to(expected):
 
 def is_empty():
   def _empty(actual):
+actual = list(actual)
 if actual:
   raise DataflowAssertException(
   'Failed assert: [] == %r' % actual)
@@ -224,7 +225,7 @@ def assert_that(actual, matcher, label='assert_that'):
 def apply(self, pipeline):
   return pipeline | 'singleton' >> Create([None]) | Map(
   match,
-  AllOf(actual | core.WindowInto(window.GlobalWindows(
+  AsList(actual | core.WindowInto(window.GlobalWindows(
 
 def default_label(self):
   return label

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/66b4c2f1/sdks/python/apache_beam/transforms/write_ptransform_test.py
--
diff --git a/sdks/python/apache_beam/transforms/write_ptransform_test.py 
b/sdks/python/apache_beam/transforms/write_ptransform_test.py
index af3668c..e7cdbd4 100644
--- a/sdks/python/apache_beam/transforms/write_ptransform_test.py
+++ b/sdks/python/apache_beam/transforms/write_ptransform_test.py
@@ -100,7 +100,7 @@ class WriteTest(unittest.TestCase):
 write_to_test_sink = WriteToTestSink(return_init_result,
  return_write_results)
 p = Pipeline(options=PipelineOptions([]))
-result = p | 'start' >> beam.Create(data) | write_to_test_sink
+result = p | beam.Create(data) | write_to_test_sink | beam.Map(list)
 
 assert_that(result, is_empty())
 p.run()

[jira] [Commented] (BEAM-595) Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() in Spark EvaluationContext

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586588#comment-15586588
 ] 

ASF GitHub Bot commented on BEAM-595:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1096


> Support non-blocking run() in SparkRunner and cancel() and waitUntilFinish() 
> in Spark EvaluationContext
> ---
>
> Key: BEAM-595
> URL: https://issues.apache.org/jira/browse/BEAM-595
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-spark
>Reporter: Pei He
>Assignee: Amit Sela
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationExcedption in Spark runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] incubator-beam git commit: Keep throws in StreamingEvaluationContext cancel() and waitUntilFinish

[kenn]

Keep throws in StreamingEvaluationContext cancel() and waitUntilFinish


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b37526ea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b37526ea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b37526ea

Branch: refs/heads/master
Commit: b37526ea53f82f294eaed8e258c357a46ff01cd6
Parents: c92f986
Author: Pei He 
Authored: Mon Oct 17 11:41:41 2016 -0700
Committer: Pei He 
Committed: Tue Oct 18 10:57:49 2016 -0700

--
 .../spark/translation/EvaluationContext.java|  2 +-
 .../streaming/StreamingEvaluationContext.java   | 22 
 2 files changed, 23 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b37526ea/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
index 83ef3c5..c1c65dd 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
@@ -309,7 +309,7 @@ public class EvaluationContext implements EvaluationResult {
   @Override
   public State waitUntilFinish(Duration duration)
   throws IOException, InterruptedException {
-// This is no-op, since Spark runner is blocking.
+// This is no-op, since Spark runner in batch is blocking.
 // It needs to be updated once SparkRunner supports non-blocking execution:
 // https://issues.apache.org/jira/browse/BEAM-595
 return State.DONE;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b37526ea/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingEvaluationContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingEvaluationContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingEvaluationContext.java
index 0b32dfd..2652f2b 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingEvaluationContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/streaming/StreamingEvaluationContext.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.spark.translation.streaming;
 
 import com.google.common.collect.Iterables;
 
+import java.io.IOException;
 import java.util.LinkedHashMap;
 import java.util.LinkedHashSet;
 import java.util.Map;
@@ -45,6 +46,7 @@ import org.apache.spark.api.java.function.VoidFunction;
 import org.apache.spark.streaming.api.java.JavaDStream;
 import org.apache.spark.streaming.api.java.JavaDStreamLike;
 import org.apache.spark.streaming.api.java.JavaStreamingContext;
+import org.joda.time.Duration;
 
 
 /**
@@ -198,6 +200,26 @@ public class StreamingEvaluationContext extends 
EvaluationContext {
 return state;
   }
 
+  @Override
+  public State cancel() throws IOException {
+throw new UnsupportedOperationException(
+"Spark runner StreamingEvaluationContext does not support cancel.");
+  }
+
+  @Override
+  public State waitUntilFinish()
+  throws IOException, InterruptedException {
+throw new UnsupportedOperationException(
+"Spark runner StreamingEvaluationContext does not support 
waitUntilFinish.");
+  }
+
+  @Override
+  public State waitUntilFinish(Duration duration)
+  throws IOException, InterruptedException {
+throw new UnsupportedOperationException(
+"Spark runner StreamingEvaluationContext does not support 
waitUntilFinish.");
+  }
+
   // override in order to expose in package
   @Override
   protected  InputT getInput(PTransform 
transform) {

[GitHub] incubator-beam pull request #1096: [BEAM-595] avoid throwing Exception in wa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1096


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam git commit: [BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7f562cc10 -> a18c27488


[BEAM-595] avoid throwing Exception in waitUntilFinish in SparkRunner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c92f9861
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c92f9861
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c92f9861

Branch: refs/heads/master
Commit: c92f9861bad7f3352294cc2eb062f1e48be18445
Parents: b8e6eea
Author: Pei He 
Authored: Thu Oct 13 14:14:15 2016 -0700
Committer: Pei He 
Committed: Mon Oct 17 11:43:04 2016 -0700

--
 .../beam/runners/spark/translation/EvaluationContext.java  | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c92f9861/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
index 317c7be..83ef3c5 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
@@ -309,7 +309,9 @@ public class EvaluationContext implements EvaluationResult {
   @Override
   public State waitUntilFinish(Duration duration)
   throws IOException, InterruptedException {
-throw new UnsupportedOperationException(
-"Spark runner EvaluationContext does not support waitUntilFinish.");
+// This is no-op, since Spark runner is blocking.
+// It needs to be updated once SparkRunner supports non-blocking execution:
+// https://issues.apache.org/jira/browse/BEAM-595
+return State.DONE;
   }
 }

[3/3] incubator-beam git commit: This closes #1096

[kenn]

This closes #1096


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a18c2748
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a18c2748
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a18c2748

Branch: refs/heads/master
Commit: a18c27488666c02ace2731fb4a040ca903e43b7b
Parents: 7f562cc b37526e
Author: Kenneth Knowles 
Authored: Tue Oct 18 13:20:11 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Oct 18 13:20:11 2016 -0700

--
 .../spark/translation/EvaluationContext.java|  6 --
 .../streaming/StreamingEvaluationContext.java   | 22 
 2 files changed, 26 insertions(+), 2 deletions(-)
--

[GitHub] incubator-beam pull request #1109: "Intern" schemas and schema strings to pr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1109


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: "Intern" schemas and schema strings to prevent out of memory issues when dealing with many many files in Avro sources.

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6d686288e -> 7f562cc10


"Intern" schemas and schema strings to prevent out of memory issues when 
dealing with many many files in Avro sources.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ebc62025
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ebc62025
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ebc62025

Branch: refs/heads/master
Commit: ebc62025d26af5b5b3e2568e6e0e9e9df9d72546
Parents: 6d68628
Author: Luke Cwik 
Authored: Fri Oct 14 11:46:37 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 12:50:56 2016 -0700

--
 .../java/org/apache/beam/sdk/io/AvroSource.java | 97 +---
 .../org/apache/beam/sdk/io/AvroSourceTest.java  | 43 +
 2 files changed, 125 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ebc62025/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
index f7ce3c2..aaf72ac 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroSource.java
@@ -21,10 +21,13 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import static com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
 
+import com.google.common.annotations.VisibleForTesting;
 import java.io.ByteArrayInputStream;
 import java.io.EOFException;
 import java.io.IOException;
 import java.io.InputStream;
+import java.io.InvalidObjectException;
+import java.io.ObjectStreamException;
 import java.io.PushbackInputStream;
 import java.nio.ByteBuffer;
 import java.nio.channels.Channels;
@@ -32,6 +35,8 @@ import java.nio.channels.ReadableByteChannel;
 import java.nio.channels.SeekableByteChannel;
 import java.util.Arrays;
 import java.util.Collection;
+import java.util.Map;
+import java.util.WeakHashMap;
 import java.util.zip.Inflater;
 import java.util.zip.InflaterInputStream;
 import javax.annotation.concurrent.GuardedBy;
@@ -164,7 +169,7 @@ public class AvroSource extends BlockBasedSource {
* to read records of the given type from a file pattern.
*/
   public static  Read.Bounded readFromFileWithClass(String filePattern, 
Class clazz) {
-return Read.from(new AvroSource(filePattern, DEFAULT_MIN_BUNDLE_SIZE,
+return Read.from(new AvroSource<>(filePattern, DEFAULT_MIN_BUNDLE_SIZE,
 ReflectData.get().getSchema(clazz).toString(), clazz, null, null));
   }
 
@@ -218,14 +223,14 @@ public class AvroSource extends BlockBasedSource {
* Does not modify this object.
*/
   public AvroSource withMinBundleSize(long minBundleSize) {
-return new AvroSource(
+return new AvroSource<>(
 getFileOrPatternSpec(), minBundleSize, readSchemaString, type, codec, 
syncMarker);
   }
 
   private AvroSource(String fileNameOrPattern, long minBundleSize, String 
schema, Class type,
   String codec, byte[] syncMarker) {
 super(fileNameOrPattern, minBundleSize);
-this.readSchemaString = schema;
+this.readSchemaString = internSchemaString(schema);
 this.codec = codec;
 this.syncMarker = syncMarker;
 this.type = type;
@@ -235,11 +240,11 @@ public class AvroSource extends BlockBasedSource {
   private AvroSource(String fileName, long minBundleSize, long startOffset, 
long endOffset,
   String schema, Class type, String codec, byte[] syncMarker, String 
fileSchema) {
 super(fileName, minBundleSize, startOffset, endOffset);
-this.readSchemaString = schema;
+this.readSchemaString = internSchemaString(schema);
 this.codec = codec;
 this.syncMarker = syncMarker;
 this.type = type;
-this.fileSchemaString = fileSchema;
+this.fileSchemaString = internSchemaString(fileSchema);
   }
 
   @Override
@@ -277,13 +282,18 @@ public class AvroSource extends BlockBasedSource {
 readSchemaString = metadata.getSchemaString();
   }
 }
-return new AvroSource(fileName, getMinBundleSize(), start, end, 
readSchemaString, type,
+// Note that if the fileSchemaString is equivalent to the 
readSchemaString, "intern"ing
+// the string will occur within the constructor and return the same 
reference as the
+// readSchemaString. This allows for Java to have an efficient 
serialization since it
+// will only encode the schema once while just storing pointers to the 
encoded version
+// within this source.
+return new 

[2/2] incubator-beam git commit: "Intern" schemas and schema strings to prevent out of memory issues

[lcwik]

"Intern" schemas and schema strings to prevent out of memory issues

This closes #1109


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7f562cc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7f562cc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7f562cc1

Branch: refs/heads/master
Commit: 7f562cc10753147bdd424e0b237c50b1b33f852c
Parents: 6d68628 ebc6202
Author: Luke Cwik 
Authored: Tue Oct 18 12:51:19 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 12:51:19 2016 -0700

--
 .../java/org/apache/beam/sdk/io/AvroSource.java | 97 +---
 .../org/apache/beam/sdk/io/AvroSourceTest.java  | 43 +
 2 files changed, 125 insertions(+), 15 deletions(-)
--

[jira] [Commented] (BEAM-738) Update sourcetestutils to fail when BoundedSources maintain state

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586492#comment-15586492
 ] 

Chamikara Jayalath commented on BEAM-738:
-

https://github.com/apache/incubator-beam/pull/1090 fixes this.

> Update sourcetestutils to fail when BoundedSources maintain state
> -
>
> Key: BEAM-738
> URL: https://issues.apache.org/jira/browse/BEAM-738
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Maintaining state in BoundedSource implementations is problematic and can 
> lead to hard to debug errors. For example (1) pickling errors (2) errors due 
> to a runner reusing a BoundedSource object with state.
> We can try to prevent users from adding state to BoundedSource 
> implementations in following two ways.
> (1) Clearly mention in BoundedSource API that objects should not maintain 
> transient state.
> (2) Update sourcetesutils to catch source objects that maintain local state.
> (2) can be done by adding a check that verifies that a source produces 
> expected output in the presence of a re-entrant read.
> i = s.read_records()
> i.next()
> i.next()
> read the whole thing from s.read_records()
> i.next() some more
> Verify that 'i' produced correct output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-737) Update TextSource to not maintain local state

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-737.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Update TextSource to not maintain local state
> -
>
> Key: BEAM-737
> URL: https://issues.apache.org/jira/browse/BEAM-737
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: Not applicable
>
>
> Currently, TextSource [1] maintains a temporary buffer as local state. This 
> could be problematic if a runner implementation tries to pickle the source. 
> We could update TextSource to not maintain any temporary state.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/textio.py#L32



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] incubator-beam git commit: Adds an assertion to source_test_utils for testing reentrancy.

[robertwb]

Adds an assertion to source_test_utils for testing reentrancy.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bdcb04cb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bdcb04cb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bdcb04cb

Branch: refs/heads/python-sdk
Commit: bdcb04cb9146d035339f02559127a810166721ab
Parents: 2ab8d62
Author: Chamikara Jayalath 
Authored: Sat Oct 15 17:49:13 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:09:19 2016 -0700

--
 sdks/python/apache_beam/io/avroio_test.py   | 30 ---
 sdks/python/apache_beam/io/iobase.py|  5 +-
 sdks/python/apache_beam/io/source_test_utils.py | 55 
 sdks/python/apache_beam/io/textio.py| 29 +--
 sdks/python/apache_beam/io/textio_test.py   | 34 +++-
 5 files changed, 99 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index eb2c81c..f72c3f3 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -110,7 +110,7 @@ class TestAvro(unittest.TestCase):
 return file_name_prefix + os.path.sep + 'mytemp*'
 
   def _run_avro_test(self, pattern, desired_bundle_size, perform_splitting,
- expected_result, test_reentrancy=False):
+ expected_result):
 source = AvroSource(pattern)
 
 read_records = []
@@ -128,23 +128,9 @@ class TestAvro(unittest.TestCase):
   (split.source, split.start_position, split.stop_position)
   for split in splits
   ]
-  if test_reentrancy:
-for source_info in sources_info:
-  reader_iter = source_info[0].read(source_info[0].get_range_tracker(
-  source_info[1], source_info[2]))
-  try:
-next(reader_iter)
-  except StopIteration:
-# Ignoring empty bundle
-pass
-
   source_test_utils.assertSourcesEqualReferenceSource((source, None, None),
   sources_info)
 else:
-  if test_reentrancy:
-reader_iter = source.read(source.get_range_tracker(None, None))
-next(reader_iter)
-
   read_records = source_test_utils.readFromSource(source, None, None)
   self.assertItemsEqual(expected_result, read_records)
 
@@ -160,15 +146,17 @@ class TestAvro(unittest.TestCase):
 
   def test_read_reentrant_without_splitting(self):
 file_name = self._write_data()
-expected_result = self.RECORDS
-self._run_avro_test(file_name, None, False, expected_result,
-test_reentrancy=True)
+source = AvroSource(file_name)
+source_test_utils.assertReentrantReadsSucceed((source, None, None))
 
   def test_read_reantrant_with_splitting(self):
 file_name = self._write_data()
-expected_result = self.RECORDS
-self._run_avro_test(file_name, 100, True, expected_result,
-test_reentrancy=True)
+source = AvroSource(file_name)
+splits = [
+split for split in source.split(desired_bundle_size=10)]
+assert len(splits) == 1
+source_test_utils.assertReentrantReadsSucceed(
+(splits[0].source, splits[0].start_position, splits[0].stop_position))
 
   def test_read_without_splitting_multiple_blocks(self):
 file_name = self._write_data(count=12000)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index edd3524..9701964 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -94,11 +94,10 @@ class BoundedSource(object):
 
   **Mutability**
 
-  A ``BoundedSource`` object should be fully mutated before being submitted
-  for reading. A ``BoundedSource`` object should not be mutated while
+  A ``BoundedSource`` object should not be mutated while
   its methods (for example, ``read()``) are being invoked by a runner. Runner
   implementations may invoke methods of ``BoundedSource`` objects through
-  multi-threaded and/or re-entrant execution modes.
+  multi-threaded and/or reentrant execution modes.
   """
 
   def estimate_size(self):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/source_test_utils.py

[1/3] incubator-beam git commit: Document that source objects should not be mutated.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk fe1f39609 -> 6e6d89d48


Document that source objects should not be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentracy tests for text and Avro sources.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2ab8d62a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2ab8d62a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2ab8d62a

Branch: refs/heads/python-sdk
Commit: 2ab8d62ac48481a52fa04c704491f3a5889de27c
Parents: fe1f396
Author: Chamikara Jayalath 
Authored: Wed Oct 12 16:51:20 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:08:41 2016 -0700

--
 sdks/python/apache_beam/io/avroio_test.py   | 28 ++-
 sdks/python/apache_beam/io/iobase.py|  8 ++
 sdks/python/apache_beam/io/source_test_utils.py | 13 +---
 sdks/python/apache_beam/io/textio.py| 80 +++-
 sdks/python/apache_beam/io/textio_test.py   | 37 +
 5 files changed, 121 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2ab8d62a/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index 1c96d72..eb2c81c 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -110,7 +110,7 @@ class TestAvro(unittest.TestCase):
 return file_name_prefix + os.path.sep + 'mytemp*'
 
   def _run_avro_test(self, pattern, desired_bundle_size, perform_splitting,
- expected_result):
+ expected_result, test_reentrancy=False):
 source = AvroSource(pattern)
 
 read_records = []
@@ -128,9 +128,23 @@ class TestAvro(unittest.TestCase):
   (split.source, split.start_position, split.stop_position)
   for split in splits
   ]
+  if test_reentrancy:
+for source_info in sources_info:
+  reader_iter = source_info[0].read(source_info[0].get_range_tracker(
+  source_info[1], source_info[2]))
+  try:
+next(reader_iter)
+  except StopIteration:
+# Ignoring empty bundle
+pass
+
   source_test_utils.assertSourcesEqualReferenceSource((source, None, None),
   sources_info)
 else:
+  if test_reentrancy:
+reader_iter = source.read(source.get_range_tracker(None, None))
+next(reader_iter)
+
   read_records = source_test_utils.readFromSource(source, None, None)
   self.assertItemsEqual(expected_result, read_records)
 
@@ -144,6 +158,18 @@ class TestAvro(unittest.TestCase):
 expected_result = self.RECORDS
 self._run_avro_test(file_name, 100, True, expected_result)
 
+  def test_read_reentrant_without_splitting(self):
+file_name = self._write_data()
+expected_result = self.RECORDS
+self._run_avro_test(file_name, None, False, expected_result,
+test_reentrancy=True)
+
+  def test_read_reantrant_with_splitting(self):
+file_name = self._write_data()
+expected_result = self.RECORDS
+self._run_avro_test(file_name, 100, True, expected_result,
+test_reentrancy=True)
+
   def test_read_without_splitting_multiple_blocks(self):
 file_name = self._write_data(count=12000)
 expected_result = self.RECORDS * 2000

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2ab8d62a/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index 8239e26..edd3524 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -91,6 +91,14 @@ class BoundedSource(object):
   positions passed to the method ``get_range_tracker()`` are ``None``
   (2) Method read() will be invoked with the ``RangeTracker`` obtained in the
   previous step.
+
+  **Mutability**
+
+  A ``BoundedSource`` object should be fully mutated before being submitted
+  for reading. A ``BoundedSource`` object should not be mutated while
+  its methods (for example, ``read()``) are being invoked by a runner. Runner
+  implementations may invoke methods of ``BoundedSource`` objects through
+  multi-threaded and/or re-entrant execution modes.
   """
 
   def 

[2/2] incubator-beam git commit: Closes #1123

[robertwb]

Closes #1123


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fe1f3960
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fe1f3960
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fe1f3960

Branch: refs/heads/python-sdk
Commit: fe1f39609fe262b48fc647d6af684f30c130b27b
Parents: afa5ebc b2c5f3a
Author: Robert Bradshaw 
Authored: Tue Oct 18 12:04:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:04:59 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py  |  6 ++
 sdks/python/apache_beam/utils/options.py   |  9 +
 sdks/python/apache_beam/utils/pipeline_options_test.py | 10 +-
 3 files changed, 24 insertions(+), 1 deletion(-)
--

[1/2] incubator-beam git commit: Add support for experiments

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk afa5ebc70 -> fe1f39609


Add support for experiments

Adds a new --experiment flag, to allow runners to optionally enable
experimental features.

This is similar to the same flag in the java sdk.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2c5f3a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2c5f3a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2c5f3a8

Branch: refs/heads/python-sdk
Commit: b2c5f3a8f227bdb2cf0092c489701f4e1d762b7b
Parents: afa5ebc
Author: Ahmet Altay 
Authored: Mon Oct 17 18:17:24 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:04:58 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py  |  6 ++
 sdks/python/apache_beam/utils/options.py   |  9 +
 sdks/python/apache_beam/utils/pipeline_options_test.py | 10 +-
 3 files changed, 24 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index 3f82f29..8c7cc29 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -37,6 +37,7 @@ from apache_beam.utils import retry
 from apache_beam.utils.dependency import get_required_container_version
 from apache_beam.utils.dependency import get_sdk_name_and_version
 from apache_beam.utils.names import PropertyNames
+from apache_beam.utils.options import DebugOptions
 from apache_beam.utils.options import GoogleCloudOptions
 from apache_beam.utils.options import StandardOptions
 from apache_beam.utils.options import WorkerOptions
@@ -107,6 +108,7 @@ class Environment(object):
 self.standard_options = options.view_as(StandardOptions)
 self.google_cloud_options = options.view_as(GoogleCloudOptions)
 self.worker_options = options.view_as(WorkerOptions)
+self.debug_options = options.view_as(DebugOptions)
 self.proto = dataflow.Environment()
 self.proto.clusterManagerApiService = COMPUTE_API_SERVICE
 self.proto.dataset = '%s/cloud_dataflow' % BIGQUERY_API_SERVICE
@@ -141,6 +143,10 @@ class Environment(object):
 value=to_json_value(job_type)),
 dataflow.Environment.VersionValue.AdditionalProperty(
 key='major', value=to_json_value(environment_version))])
+# Experiments
+if self.debug_options.experiments:
+  for experiment in self.debug_options.experiments:
+self.proto.experiments.append(experiment)
 # Worker pool(s) information.
 package_descriptors = []
 for package in packages:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/utils/options.py
--
diff --git a/sdks/python/apache_beam/utils/options.py 
b/sdks/python/apache_beam/utils/options.py
index a3cea1e..4a56ee2 100644
--- a/sdks/python/apache_beam/utils/options.py
+++ b/sdks/python/apache_beam/utils/options.py
@@ -364,6 +364,15 @@ class DebugOptions(PipelineOptions):
 parser.add_argument('--dataflow_job_file',
 default=None,
 help='Debug file to write the workflow specification.')
+parser.add_argument(
+'--experiment',
+dest='experiments',
+action='append',
+default=None,
+help=
+('Runners may provide a number of experimental features that can be '
+ 'enabled with this flag. Please sync with the owners of the runner '
+ 'before enabling any experiments.'))
 
 
 class ProfilingOptions(PipelineOptions):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/utils/pipeline_options_test.py
--
diff --git a/sdks/python/apache_beam/utils/pipeline_options_test.py 
b/sdks/python/apache_beam/utils/pipeline_options_test.py
index 249aa04..ddeecaf 100644
--- a/sdks/python/apache_beam/utils/pipeline_options_test.py
+++ b/sdks/python/apache_beam/utils/pipeline_options_test.py
@@ -80,7 +80,7 @@ class PipelineOptionsTest(unittest.TestCase):
   PipelineOptionsTest.MockOptions).mock_option,
case['expected']['mock_option'])
 
-  def test_option_with_spcae(self):
+  def test_option_with_space(self):
 options = PipelineOptions(flags=['--option with space= value with space'])
 self.assertEqual(
 getattr(options.view_as(PipelineOptionsTest.MockOptions),

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1570

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1570

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586335#comment-15586335
 ] 

Amit Sela commented on BEAM-769:


>From my experience those failures happen when a "bad" context is reused - 
>since the context is reused in Spark tests, if it failed and closed for some 
>reason, it will try to keep reusing that closed context, and that's how it 
>manifests.
I fixed this while working on BEAM-259, this is the fix: 
https://github.com/apache/incubator-beam/pull/1055/files#diff-cdc52a99e44ad1b6d1950ecab0928ecd.
I could patch this up as part of this ticket, because BEAM-259 is still in 
review, but it will be tomorrow morning (IDT).

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify #1569

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1569

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586248#comment-15586248
 ] 

Jason Kuster commented on BEAM-769:
---

This also seems to have caused a postcommit failure:

https://builds.apache.org/job/beam_PostCommit_MavenVerify/1567/

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify #1567

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1567

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Restore trigger-related tests missed in #1083

[lcwik]

Restore trigger-related tests missed in #1083

This closes #1127


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6d686288
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6d686288
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6d686288

Branch: refs/heads/master
Commit: 6d686288efd5fd64d43ba9802314f2cbbc8df72e
Parents: 71c69b3 8d43e8a
Author: Luke Cwik 
Authored: Tue Oct 18 11:01:14 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 11:01:14 2016 -0700

--
 .../beam/sdk/util/ExecutableTriggerTest.java| 127 +++
 .../sdk/util/FinishedTriggersBitSetTest.java|  55 
 .../sdk/util/FinishedTriggersProperties.java| 110 
 .../beam/sdk/util/FinishedTriggersSetTest.java  |  60 +
 4 files changed, 352 insertions(+)
--

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1568

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1127: Restore trigger-related tests missed in #...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1127


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Restore trigger-related tests missed in #1083

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 71c69b31b -> 6d686288e


Restore trigger-related tests missed in #1083


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d43e8aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d43e8aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d43e8aa

Branch: refs/heads/master
Commit: 8d43e8aa7ccb154e17d6840c25c7a72684c615aa
Parents: 71c69b3
Author: Kenneth Knowles 
Authored: Tue Oct 18 10:11:37 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 11:00:47 2016 -0700

--
 .../beam/sdk/util/ExecutableTriggerTest.java| 127 +++
 .../sdk/util/FinishedTriggersBitSetTest.java|  55 
 .../sdk/util/FinishedTriggersProperties.java| 110 
 .../beam/sdk/util/FinishedTriggersSetTest.java  |  60 +
 4 files changed, 352 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8d43e8aa/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
new file mode 100644
index 000..1e3a1ff
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertSame;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ExecutableTrigger}.
+ */
+@RunWith(JUnit4.class)
+public class ExecutableTriggerTest {
+
+  @Test
+  public void testIndexAssignmentLeaf() throws Exception {
+StubTrigger t1 = new StubTrigger();
+ExecutableTrigger executable = ExecutableTrigger.create(t1);
+assertEquals(0, executable.getTriggerIndex());
+  }
+
+  @Test
+  public void testIndexAssignmentOneLevel() throws Exception {
+StubTrigger t1 = new StubTrigger();
+StubTrigger t2 = new StubTrigger();
+StubTrigger t = new StubTrigger(t1, t2);
+
+ExecutableTrigger executable = ExecutableTrigger.create(t);
+
+assertEquals(0, executable.getTriggerIndex());
+assertEquals(1, executable.subTriggers().get(0).getTriggerIndex());
+assertSame(t1, executable.subTriggers().get(0).getSpec());
+assertEquals(2, executable.subTriggers().get(1).getTriggerIndex());
+assertSame(t2, executable.subTriggers().get(1).getSpec());
+  }
+
+  @Test
+  public void testIndexAssignmentTwoLevel() throws Exception {
+StubTrigger t11 = new StubTrigger();
+StubTrigger t12 = new StubTrigger();
+StubTrigger t13 = new StubTrigger();
+StubTrigger t14 = new StubTrigger();
+StubTrigger t21 = new StubTrigger();
+StubTrigger t22 = new StubTrigger();
+StubTrigger t1 = new StubTrigger(t11, t12, t13, t14);
+StubTrigger t2 = new StubTrigger(t21, t22);
+StubTrigger t = new StubTrigger(t1, t2);
+
+ExecutableTrigger executable = ExecutableTrigger.create(t);
+
+assertEquals(0, executable.getTriggerIndex());
+assertEquals(1, executable.subTriggers().get(0).getTriggerIndex());
+assertEquals(6, 
executable.subTriggers().get(0).getFirstIndexAfterSubtree());
+assertEquals(6, executable.subTriggers().get(1).getTriggerIndex());
+
+assertSame(t1, executable.getSubTriggerContaining(1).getSpec());
+assertSame(t2, executable.getSubTriggerContaining(6).getSpec());
+assertSame(t1, executable.getSubTriggerContaining(2).getSpec());
+assertSame(t1, 

[GitHub] incubator-beam pull request #1128: [BEAM-771] Add a Residual Shard for Empty...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1128

[BEAM-771] Add a Residual Shard for Empty Readers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam read_when_no_inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1128


commit 87f26972c02bb7640c1eee5069ad842a4a3ab1a0
Author: Thomas Groh 
Date:   2016-10-18T17:57:53Z

Add a Residual Shard for Empty Readers

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-771) UnboundedReadEvaluator should reuse the input reader if it contains no elements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586181#comment-15586181
 ] 

ASF GitHub Bot commented on BEAM-771:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1128

[BEAM-771] Add a Residual Shard for Empty Readers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam read_when_no_inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1128


commit 87f26972c02bb7640c1eee5069ad842a4a3ab1a0
Author: Thomas Groh 
Date:   2016-10-18T17:57:53Z

Add a Residual Shard for Empty Readers

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.




> UnboundedReadEvaluator should reuse the input reader if it contains no 
> elements
> ---
>
> Key: BEAM-771
> URL: https://issues.apache.org/jira/browse/BEAM-771
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The UnboundedReadEvaluator checks, after it obtains the reader, that the 
> reader contains elements, and terminates without taking a new checkpoint if 
> not. It must also (but currently does not) add that reader and shard back as 
> a residual element.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-744) UnboundedKafkaReader should return as soon as it can.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Description: 
KafkaIO has two "wait" properties:

{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

Instead, they should return as soon as they can, [~rangadi] mentioned here: 
https://github.com/apache/incubator-beam/pull/1071 that they were both 
originally set to 10 milliseconds, so {{START_NEW_RECORDS_POLL_TIMEOUT}} should 
probably be re-set to 10 milliseconds.

  was:
KafkaIO has two "wait" properties:

{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

[~rangadi] mentioned some of these were set to due to limitations of the 
DirectRunner, and I can add that they are now limiting the Spark runner (which 
reads in defined time frames, which may be smaller then the wait time and so 
never actually read).

This feels like defaults should be set for optimal read from Kafka, while a 
runner may override those if it needs to.

[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
may be passed when creating the reader. 


> UnboundedKafkaReader should return as soon as it can.
> -
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> Instead, they should return as soon as they can, [~rangadi] mentioned here: 
> https://github.com/apache/incubator-beam/pull/1071 that they were both 
> originally set to 10 milliseconds, so {{START_NEW_RECORDS_POLL_TIMEOUT}} 
> should probably be re-set to 10 milliseconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Labels:   (was: forward-incompatible)

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Comment: was deleted

(was: Mild compatibility change -- new feature.)

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [BEAM-764] Remove cloneAs from PipelineOptions

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 75f54682a -> 71c69b31b


[BEAM-764] Remove cloneAs from PipelineOptions


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/71c69b31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/71c69b31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/71c69b31

Branch: refs/heads/master
Commit: 71c69b31b6894064bf8111007f947150ff725528
Parents: 75f5468
Author: Pei He 
Authored: Mon Oct 17 14:13:42 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:56:43 2016 -0700

--
 .../beam/sdk/options/PipelineOptions.java   |  8 
 .../sdk/options/PipelineOptionsFactory.java |  1 -
 .../sdk/options/ProxyInvocationHandler.java | 22 --
 .../beam/sdk/options/PipelineOptionsTest.java   | 43 
 4 files changed, 74 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
index 3d6cad6..5588543 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
@@ -211,14 +211,6 @@ public interface PipelineOptions extends HasDisplayData {
T as(Class kls);
 
   /**
-   * Makes a deep clone of this object, and transforms the cloned object into 
the specified
-   * type {@code kls}. See {@link #as} for more information about the 
conversion.
-   *
-   * Properties that are marked with {@code @JsonIgnore} will not be cloned.
-   */
-   T cloneAs(Class kls);
-
-  /**
* The pipeline runner that will be used to execute the pipeline.
* For registered runners, the class name can be specified, otherwise the 
fully
* qualified name needs to be specified.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 1c8a835..7206b11 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -989,7 +989,6 @@ public class PipelineOptionsFactory {
 // Ignore methods on the base PipelineOptions interface.
 try {
   methods.add(iface.getMethod("as", Class.class));
-  methods.add(iface.getMethod("cloneAs", Class.class));
   methods.add(iface.getMethod("populateDisplayData", 
DisplayData.Builder.class));
 } catch (NoSuchMethodException | SecurityException e) {
   throw new RuntimeException(e);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
index 47d7cee..a77dcc6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
@@ -134,10 +134,6 @@ class ProxyInvocationHandler implements InvocationHandler, 
HasDisplayData {
   @SuppressWarnings("unchecked")
   Class clazz = (Class) args[0];
   return as(clazz);
-} else if (args != null && "cloneAs".equals(method.getName()) && args[0] 
instanceof Class) {
-  @SuppressWarnings("unchecked")
-  Class clazz = (Class) args[0];
-  return cloneAs(proxy, clazz);
 } else if (args != null && "populateDisplayData".equals(method.getName())
 && args[0] instanceof DisplayData.Builder) {
   @SuppressWarnings("unchecked")
@@ -223,24 +219,6 @@ class ProxyInvocationHandler implements InvocationHandler, 
HasDisplayData {
   }
 
   /**
-   * Backing implementation for {@link PipelineOptions#cloneAs(Class)}.
-   *
-   * @return A copy of the PipelineOptions.
-   */
-  synchronized  T cloneAs(Object proxy, Class 
iface) {
-PipelineOptions clonedOptions;
-try {
-  clonedOptions = 

[jira] [Commented] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586153#comment-15586153
 ] 

Daniel Halperin commented on BEAM-770:
--

Mild compatibility change -- new feature.

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Labels: forward-incompatible  (was: )

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: forward-incompatible
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-770) Improve registered runner resolution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586149#comment-15586149
 ] 

ASF GitHub Bot commented on BEAM-770:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1087


> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: forward-incompatible
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Resolve Registered Runners that don't end with Runner

[dhalperi]

Resolve Registered Runners that don't end with Runner

Resolve runners in a case-insensitive manner.

This reduces duplication in specifying a runner

e.g. the DirectRunner can be specified with (among others) any of
"--runner=direct", "--runner=directrunner", "--runner=DirectRunner",
"--runner=Direct", or "--runner=directRunner"


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e601410b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e601410b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e601410b

Branch: refs/heads/master
Commit: e601410b6597022c637be27cee86ab080274017a
Parents: 4469479
Author: Thomas Groh 
Authored: Wed Oct 12 13:07:52 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:52:59 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java | 44 +---
 .../sdk/options/PipelineOptionsFactoryTest.java | 23 +++---
 2 files changed, 48 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e601410b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index cd0c6b2..1c8a835 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -34,6 +34,7 @@ import com.google.common.collect.FluentIterable;
 import com.google.common.collect.ImmutableListMultimap;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Iterators;
 import com.google.common.collect.ListMultimap;
@@ -446,6 +447,7 @@ public class PipelineOptionsFactory {
   private static final Class[] EMPTY_CLASS_ARRAY = new Class[0];
   private static final ObjectMapper MAPPER = new ObjectMapper();
   private static final ClassLoader CLASS_LOADER;
+
   private static final Map> 
SUPPORTED_PIPELINE_RUNNERS;
 
   /** Classes that are used as the boundary in the stack trace to find the 
callers class name. */
@@ -514,16 +516,20 @@ public class PipelineOptionsFactory {
 
 CLASS_LOADER = findClassLoader();
 
-// Store the list of all available pipeline runners.
-ImmutableMap.Builder> builder =
-ImmutableMap.builder();
 Set pipelineRunnerRegistrars =
 Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
 pipelineRunnerRegistrars.addAll(
 Lists.newArrayList(ServiceLoader.load(PipelineRunnerRegistrar.class, 
CLASS_LOADER)));
+// Store the list of all available pipeline runners.
+ImmutableMap.Builder> builder =
+ImmutableMap.builder();
 for (PipelineRunnerRegistrar registrar : pipelineRunnerRegistrars) {
   for (Class> klass : 
registrar.getPipelineRunners()) {
-builder.put(klass.getSimpleName(), klass);
+String runnerName = klass.getSimpleName().toLowerCase();
+builder.put(runnerName, klass);
+if (runnerName.endsWith("runner")) {
+  builder.put(runnerName.substring(0, runnerName.length() - 
"Runner".length()), klass);
+}
   }
 }
 SUPPORTED_PIPELINE_RUNNERS = builder.build();
@@ -1420,24 +1426,25 @@ public class PipelineOptionsFactory {
 JavaType type = 
MAPPER.getTypeFactory().constructType(method.getGenericReturnType());
 if ("runner".equals(entry.getKey())) {
   String runner = Iterables.getOnlyElement(entry.getValue());
-  if (SUPPORTED_PIPELINE_RUNNERS.containsKey(runner)) {
-convertedOptions.put("runner", 
SUPPORTED_PIPELINE_RUNNERS.get(runner));
+  if (SUPPORTED_PIPELINE_RUNNERS.containsKey(runner.toLowerCase())) {
+convertedOptions.put("runner", 
SUPPORTED_PIPELINE_RUNNERS.get(runner.toLowerCase()));
   } else {
 try {
   Class runnerClass = Class.forName(runner);
-  checkArgument(
-  PipelineRunner.class.isAssignableFrom(runnerClass),
-  "Class '%s' does not implement PipelineRunner. Supported 
pipeline runners %s",
-  runner,
-  Sets.newTreeSet(SUPPORTED_PIPELINE_RUNNERS.keySet()));
+  if (!(PipelineRunner.class.isAssignableFrom(runnerClass))) {
+throw new IllegalArgumentException(
+String.format(
+

[jira] [Updated] (BEAM-744) UnboundedKafkaReader should return as soon as it can.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Summary: UnboundedKafkaReader should return as soon as it can.  (was: A 
runner should be able to override KafkaIO max wait properties.)

> UnboundedKafkaReader should return as soon as it can.
> -
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1087: [BEAM-770] Resolve Registered Runners tha...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1087


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #1087

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 446947947 -> 75f54682a


Closes #1087


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/75f54682
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/75f54682
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/75f54682

Branch: refs/heads/master
Commit: 75f54682a705aa0776c12bc9c2dcb8cb9a7c08fa
Parents: 4469479 e601410
Author: Dan Halperin 
Authored: Tue Oct 18 10:52:59 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:52:59 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java | 44 +---
 .../sdk/options/PipelineOptionsFactoryTest.java | 23 +++---
 2 files changed, 48 insertions(+), 19 deletions(-)
--

[jira] [Commented] (BEAM-733) Windowed side-inputs not supported in Python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586145#comment-15586145
 ] 

ASF GitHub Bot commented on BEAM-733:
-

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/1070


> Windowed side-inputs not supported in Python
> 
>
> Key: BEAM-733
> URL: https://issues.apache.org/jira/browse/BEAM-733
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1070: [BEAM-733] Prohibit windowed side inputs ...

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/1070


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Dissallow (unimplemented) windowed side inputs.

[robertwb]

Dissallow (unimplemented) windowed side inputs.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/552f6d7b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/552f6d7b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/552f6d7b

Branch: refs/heads/python-sdk
Commit: 552f6d7baa6e6205b290d21e642911a04ca259ec
Parents: 24b7bcc
Author: Robert Bradshaw 
Authored: Fri Oct 7 15:28:50 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:48:16 2016 -0700

--
 .../python/apache_beam/transforms/sideinputs.py |  4 ++
 .../apache_beam/transforms/sideinputs_test.py   | 40 
 2 files changed, 44 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/552f6d7b/sdks/python/apache_beam/transforms/sideinputs.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs.py 
b/sdks/python/apache_beam/transforms/sideinputs.py
index 6484a7c..6c698da 100644
--- a/sdks/python/apache_beam/transforms/sideinputs.py
+++ b/sdks/python/apache_beam/transforms/sideinputs.py
@@ -50,6 +50,10 @@ class CreatePCollectionView(PTransform):
 return input_type
 
   def apply(self, pcoll):
+if not pcoll.windowing.is_default():
+  raise ValueError(
+  "Side inputs only supported for global windows, default triggering. "
+  "Found %s" % pcoll.windowing)
 return self.view
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/552f6d7b/sdks/python/apache_beam/transforms/sideinputs_test.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py 
b/sdks/python/apache_beam/transforms/sideinputs_test.py
new file mode 100644
index 000..8e292e3
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/sideinputs_test.py
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+import logging
+import unittest
+
+import apache_beam as beam
+from apache_beam.transforms import window
+
+
+class SideInputsTest(unittest.TestCase):
+
+  # TODO(BEAM-733): Actually support this.
+  def test_no_sideinput_windowing(self):
+p = beam.Pipeline('DirectPipelineRunner')
+pc = p | beam.Create([0, 1]) | beam.WindowInto(window.FixedWindows(10))
+with self.assertRaises(ValueError):
+  # pylint: disable=expression-not-assigned
+  pc | beam.Map(lambda x, side: None, side=beam.pvalue.AsIter(pc))
+
+
+if __name__ == '__main__':
+  logging.getLogger().setLevel(logging.DEBUG)
+  unittest.main()

[3/3] incubator-beam git commit: Closes #1070

[robertwb]

Closes #1070


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/afa5ebc7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/afa5ebc7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/afa5ebc7

Branch: refs/heads/python-sdk
Commit: afa5ebc70aef3a7ad20b4e70a62e23a90bd7547a
Parents: 24b7bcc 5d9506d
Author: Robert Bradshaw 
Authored: Tue Oct 18 10:53:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:53:59 2016 -0700

--
 sdks/python/apache_beam/pipeline_test.py|  1 +
 .../python/apache_beam/transforms/sideinputs.py |  4 ++
 .../apache_beam/transforms/sideinputs_test.py   | 40 
 sdks/python/apache_beam/transforms/util.py  |  7 +++-
 4 files changed, 50 insertions(+), 2 deletions(-)
--

[1/3] incubator-beam git commit: Fix tests unnecessarily using windowed side inputs

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 24b7bcc26 -> afa5ebc70


Fix tests unnecessarily using windowed side inputs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5d9506df
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5d9506df
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5d9506df

Branch: refs/heads/python-sdk
Commit: 5d9506df44e5b2cd703702d4188c25c5bbb16e7f
Parents: 552f6d7
Author: Robert Bradshaw 
Authored: Fri Oct 7 16:17:47 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:48:16 2016 -0700

--
 sdks/python/apache_beam/pipeline_test.py   | 1 +
 sdks/python/apache_beam/transforms/util.py | 7 +--
 2 files changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d9506df/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index 8c1b3ba..dedd732 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -239,6 +239,7 @@ class PipelineTest(unittest.TestCase):
 'oom:combine/GroupByKey/group_by_key': 1,
 ('oom:check', None): 1,
 'assert_that/singleton': 1,
+('assert_that/WindowInto', None): 1,
 ('assert_that/Map(match)', None): 1,
 ('oom:combine/GroupByKey/group_by_window', None): 1,
 ('oom:combine/Combine/ParDo(CombineValuesDoFn)', None): 1})

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d9506df/sdks/python/apache_beam/transforms/util.py
--
diff --git a/sdks/python/apache_beam/transforms/util.py 
b/sdks/python/apache_beam/transforms/util.py
index 4564cf9..aeac0d9 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -21,6 +21,8 @@
 from __future__ import absolute_import
 
 from apache_beam.pvalue import AsIter as AllOf
+from apache_beam.transforms import core
+from apache_beam.transforms import window
 from apache_beam.transforms.core import CombinePerKey, Create, Flatten, 
GroupByKey, Map
 from apache_beam.transforms.ptransform import PTransform
 from apache_beam.transforms.ptransform import ptransform_fn
@@ -220,8 +222,9 @@ def assert_that(actual, matcher, label='assert_that'):
   class AssertThat(PTransform):
 
 def apply(self, pipeline):
-  return pipeline | 'singleton' >> Create([None]) | Map(match,
-AllOf(actual))
+  return pipeline | 'singleton' >> Create([None]) | Map(
+  match,
+  AllOf(actual | core.WindowInto(window.GlobalWindows(
 
 def default_label(self):
   return label

[jira] [Created] (BEAM-770) Improve registered runner resolution

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-770:


 Summary: Improve registered runner resolution
 Key: BEAM-770
 URL: https://issues.apache.org/jira/browse/BEAM-770
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


Runner is generally a duplicate word within the Pipeline args specification - 
specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
there's a registered {{FooRunner}} and use that.

Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1126: [BEAM-768] Dataflow: API is in beta now

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Dataflow: API is in beta now

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 006cb127e -> 446947947


Dataflow: API is in beta now


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/772fb3f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/772fb3f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/772fb3f1

Branch: refs/heads/master
Commit: 772fb3f1351bdad04adc6aeb3461daa0964cce92
Parents: 006cb12
Author: Dan Halperin 
Authored: Tue Oct 18 09:42:50 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:40:14 2016 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/772fb3f1/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index d014623..a6afccd 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -46,7 +46,7 @@ import org.slf4j.LoggerFactory;
  */
 public final class MonitoringUtil {
 
-  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud alpha dataflow";
+  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud beta dataflow";
   private static final String ENDPOINT_OVERRIDE_ENV_VAR =
   "CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW";
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/772fb3f1/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
index 98d8ab6..6c5a2be 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
@@ -133,7 +133,7 @@ public class MonitoringUtilTest {
 options.setProject(PROJECT_ID);
 options.setGcpCredential(new TestCredential());
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
-assertEquals("gcloud alpha dataflow jobs --project=someProject cancel 
1234", cancelCommand);
+assertEquals("gcloud beta dataflow jobs --project=someProject cancel 
1234", cancelCommand);
   }
 
   @Test
@@ -147,7 +147,7 @@ public class MonitoringUtilTest {
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
 assertEquals(
 
"CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW=https://dataflow.googleapis.com/v0neverExisted/
 "
-+ "gcloud alpha dataflow jobs --project=someProject cancel 1234",
++ "gcloud beta dataflow jobs --project=someProject cancel 1234",
 cancelCommand);
   }
 

[jira] [Resolved] (BEAM-768) Update Dataflow API to beta

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-768.
--
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Update Dataflow API to beta
> ---
>
> Key: BEAM-768
> URL: https://issues.apache.org/jira/browse/BEAM-768
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Dataflow's API is now in {{beta}}, so the {{gcloud}} command strings printed 
> need to be upgraded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-769) Spark translation tests failing with NullPointerException

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-769:
-
Description: 
https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/

{code}
org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?

  was:
{code}

org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?


> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-769) Spark translation tests failing with NullPointerException

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-769:


 Summary: Spark translation tests failing with NullPointerException
 Key: BEAM-769
 URL: https://issues.apache.org/jira/browse/BEAM-769
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Amit Sela


{code}

org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Closes #1126

[dhalperi]

Closes #1126


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/44694794
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/44694794
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/44694794

Branch: refs/heads/master
Commit: 446947947585ef8c7aa2b29ac1464465793abfe2
Parents: 006cb12 772fb3f
Author: Dan Halperin 
Authored: Tue Oct 18 10:40:15 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:40:15 2016 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--

[jira] [Commented] (BEAM-756) Checkstyle suppression for JavadocPackage not working on Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586076#comment-15586076
 ] 

ASF GitHub Bot commented on BEAM-756:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1114


> Checkstyle suppression for JavadocPackage not working on Windows
> 
>
> Key: BEAM-756
> URL: https://issues.apache.org/jira/browse/BEAM-756
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>
> Exclusions for test and other files don't consider '\' as separator. Hence 
> checkstyle complains about missing package-info files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1114: BEAM-756 fix JavadocPackage checkstyle su...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1114


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: BEAM-756 fix JavadocPackage checkstyle suppression

[lcwik]

BEAM-756 fix JavadocPackage checkstyle suppression

This closes #1114


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/006cb127
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/006cb127
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/006cb127

Branch: refs/heads/master
Commit: 006cb127e84e045d58f13705e81d4769ae0cbf89
Parents: 4e14ac2 6a69184
Author: Luke Cwik 
Authored: Tue Oct 18 10:07:59 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:07:59 2016 -0700

--
 .../src/main/resources/beam/suppressions.xml| 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--

[1/2] incubator-beam git commit: BEAM-756 fix JavadocPackage checkstyle suppression

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4e14ac2c2 -> 006cb127e


BEAM-756 fix JavadocPackage checkstyle suppression


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a691843
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a691843
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a691843

Branch: refs/heads/master
Commit: 6a691843460255abec96690778954990246a6b9d
Parents: 4e14ac2
Author: Thomas Weise 
Authored: Mon Oct 17 11:39:45 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:07:32 2016 -0700

--
 .../src/main/resources/beam/suppressions.xml| 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6a691843/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml 
b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
index 00d6729..9f60c25 100644
--- a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
@@ -17,13 +17,13 @@
 "http://www.puppycrawl.com/dtds/suppressions_1_1.dtd;>
 
 
-  
-  
-  
+  
+  
+  
   
   
-  
-  
-  
+  
+  
+  
 
 

[jira] [Updated] (BEAM-655) Rename @RunnableOnService to something more descriptive

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-655:
-
Description: 
There are currently two categories for unit tests in the Core Beam Java SDK: 
{{NeedsRunner}} and {{RunnableOnService}}. These categories are insufficiently 
descriptive for describing the tests that they categorize.

There are two goals for these tests:
* Demonstrate that a runner conforms with the Beam model and the Capability 
Matrix. These capabilities are described at [Capability Matrix 
Testing|https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2VlUyVtpi2WzzGM/edit]
* Demonstrate that a PTransform, when executed on a {{PipelineRunner}}, 
performs as specified

NeedsRunner tests should either:
* Be converted into a runner-independent test (for example, via use of 
DoFnTester to test that a DoFn throws on invalid input)
* Be converted into a RunnableOnService test

RunnableOnService tests that explicitly test the capabilities of a Model 
primitive should be categorized into that specific capability. Existing 
RunnableOnService tests that test the execution of a PTransform should remain 
RunnableOnService.


> Rename @RunnableOnService to something more descriptive
> ---
>
> Key: BEAM-655
> URL: https://issues.apache.org/jira/browse/BEAM-655
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>
> There are currently two categories for unit tests in the Core Beam Java SDK: 
> {{NeedsRunner}} and {{RunnableOnService}}. These categories are 
> insufficiently descriptive for describing the tests that they categorize.
> There are two goals for these tests:
> * Demonstrate that a runner conforms with the Beam model and the Capability 
> Matrix. These capabilities are described at [Capability Matrix 
> Testing|https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2VlUyVtpi2WzzGM/edit]
> * Demonstrate that a PTransform, when executed on a {{PipelineRunner}}, 
> performs as specified
> NeedsRunner tests should either:
> * Be converted into a runner-independent test (for example, via use of 
> DoFnTester to test that a DoFn throws on invalid input)
> * Be converted into a RunnableOnService test
> RunnableOnService tests that explicitly test the capabilities of a Model 
> primitive should be categorized into that specific capability. Existing 
> RunnableOnService tests that test the execution of a PTransform should remain 
> RunnableOnService.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1127: Restore trigger-related tests missed in #...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1127

Restore trigger-related tests missed in #1083

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 

This is a follow-up of #1118, restoring a couple more things missed in 
#1083.

The commit that merged #1083 is e969f3d. At the time I confirmed that 
`src/main` and `src/test/sdk/transforms/windowing` were unchanged, but 
neglected `util`. Here is a transcript demonstrating the genesis of this patch:

```
$ git diff --stat e969f3d^1 e969f3d^2 sdks/java/core/src/main
$ git diff --stat e969f3d^1 e969f3d^2 sdks/java/core/src/test
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
  | 127 
--
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersBitSetTest.java
 |  55 -
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersProperties.java
 | 110 
--
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersSetTest.java
|  60 --
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java 
  |  67 
 5 files changed, 419 deletions(-)
```

I then checked each of these out at `e969f3d^1`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam trigger-util-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1127


commit b8c48171c9593da2652ed40a221cda0fbe30d49a
Author: Kenneth Knowles 
Date:   2016-10-18T17:11:37Z

Restore trigger-related tests missed in #1083




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---