[jira] [Assigned] (BEAM-659) WindowFn#isCompatible should provide a meaningful reason

[Neelesh Srinivas Salian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neelesh Srinivas Salian reassigned BEAM-659:


Assignee: Neelesh Srinivas Salian

> WindowFn#isCompatible should provide a meaningful reason
> 
>
> Key: BEAM-659
> URL: https://issues.apache.org/jira/browse/BEAM-659
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Neelesh Srinivas Salian
>Priority: Minor
>  Labels: easy, easyfix, starter
>
> {{WindowFn#isCompatible(...)}} returns a true/false answer without any 
> explanation of why; it suffers from a form of [boolean 
> blindness|https://existentialtype.wordpress.com/2011/03/15/boolean-blindness/].
> We alleviated this for {{Coder#isDeterministic()}} by switching to 
> {{Coder#verifyDeterministic()}} which can raise an exception describing why 
> it is not deterministic (it could just as easily have been a return value).
> We should do the same for {{WindowFn#isCompatible(...)}}.
> See here: 
> http://stackoverflow.com/questions/39617897/inputs-to-flatten-had-incompatible-window-windowfns-when-cogroupbykey-with-calen
> (a decent {{toString}} would go a long way, for immediate and trivial 
> improvement)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/8] incubator-beam-site git commit: Update mailing lists to use new lists site.

[frances]

Update mailing lists to use new lists site.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/7f234a5a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/7f234a5a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/7f234a5a

Branch: refs/heads/asf-site
Commit: 7f234a5a9314bbaf747e81ac069d3fcf6d3f0557
Parents: 29cc4a6
Author: Frances Perry 
Authored: Mon Oct 3 18:52:37 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:30 2016 -0700

--
 use/mailing-lists.md | 33 -
 1 file changed, 16 insertions(+), 17 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/7f234a5a/use/mailing-lists.md
--
diff --git a/use/mailing-lists.md b/use/mailing-lists.md
index 0e28d71..5807781 100644
--- a/use/mailing-lists.md
+++ b/use/mailing-lists.md
@@ -7,17 +7,13 @@ redirect_from: /mailing_lists/
 
 # Apache Beam Mailing Lists
 
-These are the mailing lists that have been established for this project. For 
each list, there is a subscribe, unsubscribe, and an archive link.
-
-Name   Subscribe   Unsubscribe PostArchive
-   Subscribe   Unsubscribe Post
[mail-archives.apache.org](http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/)
-   Subscribe   Unsubscribe Postmail-archives.apache.org
-beam-user  Subscribe   Unsubscribe Postmail-archives.apache.org
+These are the mailing lists that have been established for this project.
 
 
   
 
   Name
+  Purpose
   Subscribe
   Unsubscribe
   Post
@@ -26,25 +22,28 @@ beam-user   Subscribe   Unsubscribe Post
mail-archives.apache.org
   
   
 
-  beam-dev
+  user@
+  User support and questions
+  mailto:user-subscr...@beam.incubator.apache.org;>Subscribe
+  mailto:user-unsubscr...@beam.incubator.apache.org;>Unsubscribe
+  mailto:u...@beam.incubator.apache.org;>Post
+  https://lists.apache.org/list.html?u...@beam.apache.org;>Archives
+
+
+  dev@
+  Development-related discussions
   mailto:dev-subscr...@beam.incubator.apache.org;>Subscribe
   mailto:dev-unsubscr...@beam.incubator.apache.org;>Unsubscribe
   mailto:d...@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/;>mail-archives.apache.org
+  https://lists.apache.org/list.html?d...@beam.apache.org;>Archives
 
 
-  beam-commits
+  commits@
+  JIRA updates, pull requests, etc.
   mailto:commits-subscr...@beam.incubator.apache.org;>Subscribe
   mailto:commits-unsubscr...@beam.incubator.apache.org;>Unsubscribe
   mailto:commits@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/;>mail-archives.apache.org
-
-
-  beam-user
-  mailto:user-subscr...@beam.incubator.apache.org;>Subscribe
-  mailto:user-unsubscr...@beam.incubator.apache.org;>Unsubscribe
-  mailto:u...@beam.incubator.apache.org;>Post
-  http://mail-archives.apache.org/mod_mbox/incubator-beam-user/;>mail-archives.apache.org
+  https://lists.apache.org/list.html?comm...@beam.apache.org;>Archives
 
 
 

[5/8] incubator-beam-site git commit: Updated Contribution Guide & added Work In Progress * Added section on feature branches. * Revamped website section. * Standardized code formatting. * Added a "wo

[frances]

Updated Contribution Guide & added Work In Progress
* Added section on feature branches.
* Revamped website section.
* Standardized code formatting.
* Added a "work in progress" page to help make new and ongoing work more 
visible to the community.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c765b7ed
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c765b7ed
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c765b7ed

Branch: refs/heads/asf-site
Commit: c765b7edfcee3044ed21517a244b74ea8d8eded9
Parents: 229020c
Author: Frances Perry 
Authored: Mon Oct 3 20:57:24 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 21:03:25 2016 -0700

--
 README.md|   1 -
 _includes/header.html|   3 +-
 contribute/contribution-guide.md | 166 --
 contribute/work-in-progress.md   |  32 +++
 4 files changed, 150 insertions(+), 52 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/README.md
--
diff --git a/README.md b/README.md
index 80819e2..40172ac 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,3 @@
-
 # Apache Beam (incubating) website
 
 This is the website for [Apache Beam](http://beam.incubator.apache.org)

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 67631a9..99d59eb 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -63,11 +63,12 @@
  
  Basics
  Contribution Guide
+ Work In Progress
  Mailing Lists
   Source Repository
   Issue 
Tracking
   
- Technical 
References
+ Technical References
  Testing
   Design Principles
  https://goo.gl/nk5OM0;>Technical 
Vision

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c765b7ed/contribute/contribution-guide.md
--
diff --git a/contribute/contribution-guide.md b/contribute/contribution-guide.md
index a0dcb50..61c762b 100644
--- a/contribute/contribution-guide.md
+++ b/contribute/contribution-guide.md
@@ -18,7 +18,7 @@ We use a review-then-commit workflow in Beam for all 
contributions.
 
 **For larger contributions or those that affect multiple components:**
 
-1. **Engage**: We encourage you to work with the Beam community on the [Apache 
JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and 
[developer’s mailing list](http://beam.incubator.apache.org/mailing_lists/) 
to identify good areas for contribution.
+1. **Engage**: We encourage you to work with the Beam community on the [Apache 
JIRA issue tracker](https://issues.apache.org/jira/browse/BEAM) and 
[developer’s mailing 
list](http://beam.incubator.apache.org/use/mailing-lists/) to identify good 
areas for contribution.
 1. **Design:** More complicated contributions will likely benefit from some 
early discussion in order to scope and design them well.
 
 **For all contributions:**
@@ -32,27 +32,28 @@ We look forward to working with you!
 ## Engage
 
 ### Mailing list(s)
-We discuss design and implementation issues on d...@beam.incubator.apache.org 
mailing list, which is archived 
[here](http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/). Join by 
emailing 
[`dev-subscr...@beam.incubator.apache.org`](mailto:dev-subscr...@beam.incubator.apache.org).
+We discuss design and implementation issues on d...@beam.incubator.apache.org 
mailing list, which is archived 
[here](https://lists.apache.org/list.html?d...@beam.apache.org). Join by 
emailing 
[`dev-subscr...@beam.incubator.apache.org`](mailto:dev-subscr...@beam.incubator.apache.org).
 
-If interested, you can also join 
[`u...@beam.incubator.apache.org`](http://mail-archives.apache.org/mod_mbox/incubator-beam-user/)
 and 
[`commits@beam.incubator.apache.org`](http://mail-archives.apache.org/mod_mbox/incubator-beam-commits/)
 too.
+If interested, you can also join the other [mailing 
lists](http://beam.incubator.apache.org/use/mailing-lists/) too.
 
 ### Apache JIRA
 We use [Apache JIRA](https://issues.apache.org/jira/browse/BEAM) as an issue 
tracking and project management tool, as well as a way to communicate among a 
very diverse and distributed set of contributors. To be able to gather 
feedback, avoid 

[3/8] incubator-beam-site git commit: Remove page title from TOCs

[frances]

Remove page title from TOCs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/229020ce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/229020ce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/229020ce

Branch: refs/heads/asf-site
Commit: 229020ceac325fb5cb40f91dcd1f9d8458510999
Parents: 9978341
Author: Frances Perry 
Authored: Mon Oct 3 21:46:23 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:39 2016 -0700

--
 _config.yml | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/229020ce/_config.yml
--
diff --git a/_config.yml b/_config.yml
index 39fdbcd..39cf260 100644
--- a/_config.yml
+++ b/_config.yml
@@ -41,6 +41,10 @@ exclude: ['README.md', 'Gemfile.lock', 'Gemfile', 'Rakefile']
 # Downloads directory
 downloads: downloads
 
+# Don't use the page title in the table of contents
+kramdown:
+  toc_levels: 2..6
+
 gems:
   - jekyll-redirect-from
   
\ No newline at end of file

[GitHub] incubator-beam-site pull request #42: Update Contribution Guide and add Work...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/42


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-777) KafkaIO Test should handle reader.start() better

[Raghu Angadi (JIRA)]

Raghu Angadi created BEAM-777:
-

 Summary: KafkaIO Test should handle reader.start() better
 Key: BEAM-777
 URL: https://issues.apache.org/jira/browse/BEAM-777
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Raghu Angadi
Assignee: Raghu Angadi
Priority: Minor


KafkaIOTest currently expects reader.start() to return true. It can return 
false as well. It should handle it better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #48: Stage pull requests in Google Cloud St...

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/incubator-beam-site/pull/48

Stage pull requests in Google Cloud Storage

This is work-in-progress. Several aspects need to be taken care of.

R: @francesperry 
CC: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site buildpr

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/48.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #48


commit 7e294ba42056e47e729b31354d21eeed89302084
Author: Davor Bonaci 
Date:   2016-10-19T03:41:29Z

Stage pull requests in Google Cloud Storage

This is work-in-progress. Several aspects need to be taken care of.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[4/8] incubator-beam-site git commit: Add Design Principles (take from the original Beam technical vision document).

[frances]

Add Design Principles (take from the original Beam technical vision document).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/99783418
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/99783418
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/99783418

Branch: refs/heads/asf-site
Commit: 997834188ecf29b307e195c9c7e8d31fa60b34ff
Parents: 7f234a5
Author: Frances Perry 
Authored: Mon Oct 3 19:00:03 2016 -0700
Committer: Frances Perry 
Committed: Tue Oct 18 20:56:39 2016 -0700

--
 _includes/header.html   |  5 ++--
 contribute/design-principles.md | 53 
 2 files changed, 56 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/99783418/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 182b30a..67631a9 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -63,12 +63,13 @@
  
  Basics
  Contribution Guide
- Testing
  Mailing Lists
   Source Repository
   Issue 
Tracking
   
- Technical Resources
+ Technical 
References
+ Testing
+  Design Principles
  https://goo.gl/nk5OM0;>Technical 
Vision
  


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/99783418/contribute/design-principles.md
--
diff --git a/contribute/design-principles.md b/contribute/design-principles.md
new file mode 100644
index 000..87ddd24
--- /dev/null
+++ b/contribute/design-principles.md
@@ -0,0 +1,53 @@
+---
+layout: default
+title: 'Design Principles in Beam'
+permalink: /contribute/design-principles/
+---
+
+# Design Principles in the Apache Beam Project
+
+Joshua Bloch’s [API Design Bumper 
Stickers](https://www.infoq.com/articles/API-Design-Joshua-Bloch) are a great 
list of what makes for good API design. In addition, we have specific design 
principles we follow in Beam.
+
+* TOC
+{:toc}
+
+## Use cases
+
+### Unify the model
+Provide one model that works over both bounded (aka. batch) and unbounded 
(aka. streaming) datasets. Pay special attention to windows / triggers / state 
/ timers, which often trip up folks used to a batch world.  Provide users with 
the right abstractions to adjust latency and completeness guarantees to cover 
both traditional batch and streaming use cases. 
+
+### Separate data shapes and runtime requirements
+The model should focus on letting users describe their data and processing, 
without exposing any details of a specific runtime system. For example, bounded 
and unbounded describe the shape of data, but batch and streaming describe the 
behavior of specific runtime systems. Good test cases are to imagine a mythical 
micro-batching runner that sits somewhere between batch and streaming or a 
engine that dynamically switches between streaming and batch depending on the 
backlog.
+
+### Make efficient things easy, rather than make easy things efficient
+Don’t prevent efficiency for ease of use. Design APIs that provide the 
information necessary for efficiently executing at scale. Provide class 
hierarchies and wrappers to make the common cases simpler.
+
+## Usability
+
+### Validate Early
+Validate constraints on graph shape, runner requirements, etc as early in the 
compile time - construction time - submission time - execution time spectrum as 
reasonably possible in order to provide a smoother user experience.
+
+### Public APIs, like diamonds, are forever (at least until the next major 
version)
+Backwards incompatible changes can only be made in the next major version. 
Because of the burden major versions place on users (code has to be modified, 
conflicting dependency nightmares, etc), we aim to do this infrequently. 
Clearly mark APIs that are considered experimental (may change at any point) 
and deprecated (will be removed in the next major version). Consider what APIs 
are more amenable to future changes (abstract classes vs. interfaces, etc.)
+
+### Examples should be pedagogical
+Canonical examples help people ingrain the principles. Design examples that 
teach complex concepts in modular chunks. If you can’t explain the concept 
easily, then the API isn’t right. Examples should withstand random 
copy-pasting. 
+
+## Extensibility
+
+### Use PTransforms for modularity
+Composite transformations 

[jira] [Updated] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Description: 
KafkaIO has two "wait" properties:

{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

[~rangadi] mentioned some of these were set to due to limitations of the 
DirectRunner, and I can add that they are now limiting the Spark runner (which 
reads in defined time frames, which may be smaller then the wait time and so 
never actually read).

This feels like defaults should be set for optimal read from Kafka, while a 
runner may override those if it needs to.

[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
may be passed when creating the reader. 

  was:
KafkaIO has three "wait" properties:

{{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

[~rangadi] mentioned some of these were set to due to limitations of the 
DirectRunner, and I can add that they are now limiting the Spark runner (which 
reads in defined time frames, which may be smaller then the wait time and so 
never actually read).

This feels like defaults should be set for optimal read from Kafka, while a 
runner may override those if it needs to.

[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
may be passed when creating the reader. 


> A runner should be able to override KafkaIO max wait properties.
> 
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1560

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1560

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Direct Java #1560

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1560

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1358

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585107#comment-15585107
 ] 

Amit Sela commented on BEAM-744:


That's Correct, I updated the JIRA, thanks!

> A runner should be able to override KafkaIO max wait properties.
> 
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585260#comment-15585260
 ] 

ASF GitHub Bot commented on BEAM-744:
-

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1125

[BEAM-744] A runner should be able to override KafkaIO max wait prope…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…rties.

Add KafkaOptions for the UnboundedKafkaReader.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-744

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1125.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1125


commit 627f50cc510783117b0642d4f699d4b4d9b342c7
Author: Sela 
Date:   2016-10-18T11:36:04Z

[BEAM-744] A runner should be able to override KafkaIO max wait properties.

Add KafkaOptions for the UnboundedKafkaReader.




> A runner should be able to override KafkaIO max wait properties.
> 
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1125: [BEAM-744] A runner should be able to ove...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/incubator-beam/pull/1125

[BEAM-744] A runner should be able to override KafkaIO max wait prope…

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

…rties.

Add KafkaOptions for the UnboundedKafkaReader.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/incubator-beam BEAM-744

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1125.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1125


commit 627f50cc510783117b0642d4f699d4b4d9b342c7
Author: Sela 
Date:   2016-10-18T11:36:04Z

[BEAM-744] A runner should be able to override KafkaIO max wait properties.

Add KafkaOptions for the UnboundedKafkaReader.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1561

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1561

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1561

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1561

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Direct Java #1561

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1359

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1563

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1563

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1360

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1122: [BEAM-755] SplittableParDoTest does not r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1122


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585879#comment-15585879
 ] 

ASF GitHub Bot commented on BEAM-755:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1122


> beam-runners-core-java NeedsRunner tests not executing
> --
>
> Key: BEAM-755
> URL: https://issues.apache.org/jira/browse/BEAM-755
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Kenneth Knowles
>
> org.apache.beam:beam-runners-core-java is not specified as an integration 
> test dependency to scan within runners/pom.xml
> There is also in runners/direct-java/pom.xml where its 
> org.apache.beam:beam-runners-java-core and should be 
> org.apache.beam:beam-runners-core-java
> Finally, even if these dependencies are added and the typo fixed. When 
> running the runnable on service integration tests, SplittableParDoTest which 
> contains @RunnableOnService tests (part of runners/core-java) doesn't execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: SplittableParDoTest does not require RunnableOnService

[dhalperi]

SplittableParDoTest does not require RunnableOnService


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7f5542e0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7f5542e0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7f5542e0

Branch: refs/heads/master
Commit: 7f5542e0cf21e21822e5a2ee91ed872c1824a4bf
Parents: ecc2abf
Author: Kenneth Knowles 
Authored: Mon Oct 17 15:51:11 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 09:18:14 2016 -0700

--
 .../java/org/apache/beam/runners/core/SplittableParDoTest.java   | 4 
 1 file changed, 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7f5542e0/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
index b7cdc64..29ff838 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
@@ -34,7 +34,6 @@ import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.SerializableCoder;
-import org.apache.beam.sdk.testing.RunnableOnService;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.DoFn;
@@ -53,7 +52,6 @@ import org.apache.beam.sdk.values.TimestampedValue;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.junit.Test;
-import org.junit.experimental.categories.Category;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
 
@@ -123,7 +121,6 @@ public class SplittableParDoTest {
   }
 
   @Test
-  @Category(RunnableOnService.class)
   public void testBoundednessForBoundedFn() {
 Pipeline pipeline = TestPipeline.create();
 DoFn boundedFn = new BoundedFakeFn();
@@ -142,7 +139,6 @@ public class SplittableParDoTest {
   }
 
   @Test
-  @Category(RunnableOnService.class)
   public void testBoundednessForUnboundedFn() {
 Pipeline pipeline = TestPipeline.create();
 DoFn unboundedFn = new UnboundedFakeFn();

[1/2] incubator-beam git commit: Closes #1122

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ecc2abfdb -> c81376bc7


Closes #1122


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c81376bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c81376bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c81376bc

Branch: refs/heads/master
Commit: c81376bc7c62ec1d29283a1cb36f6b1d257207cb
Parents: ecc2abf 7f5542e
Author: Dan Halperin 
Authored: Tue Oct 18 09:18:14 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 09:18:14 2016 -0700

--
 .../java/org/apache/beam/runners/core/SplittableParDoTest.java   | 4 
 1 file changed, 4 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1562

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-767) Add DisplayData builder '.nested(..)' operation

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-767:
-

 Summary: Add DisplayData builder '.nested(..)' operation
 Key: BEAM-767
 URL: https://issues.apache.org/jira/browse/BEAM-767
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Scott Wegner
Assignee: Scott Wegner


See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515

In the Java SDK, components can register display data directly, as well as 
include display data from sub-components via the 'include(..)' operation. In 
[BEAM-1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
to display data metadata and also include a 'delegate(..)' action.

During the change for paths, we noticed an emerging pattern where components 
will typically add the .getClass() of a subcomponent before including it. We 
can support this convention better by embedding it in a single configurable 
operation, '.nested(..)'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-767) Add DisplayData builder '.nested(..)' operation

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner updated BEAM-767:
--
Description: 
See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515

In the Java SDK, components can register display data directly, as well as 
include display data from sub-components via the 'include(..)' operation. In 
[PR #1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
to display data metadata and also include a 'delegate(..)' action.

During the change for paths, we noticed an emerging pattern where components 
will typically add the .getClass() of a subcomponent before including it. We 
can support this convention better by embedding it in a single configurable 
operation, '.nested(..)'

  was:
See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515

In the Java SDK, components can register display data directly, as well as 
include display data from sub-components via the 'include(..)' operation. In 
[BEAM-1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
to display data metadata and also include a 'delegate(..)' action.

During the change for paths, we noticed an emerging pattern where components 
will typically add the .getClass() of a subcomponent before including it. We 
can support this convention better by embedding it in a single configurable 
operation, '.nested(..)'


> Add DisplayData builder '.nested(..)' operation
> ---
>
> Key: BEAM-767
> URL: https://issues.apache.org/jira/browse/BEAM-767
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Scott Wegner
>Assignee: Scott Wegner
>
> See: https://github.com/apache/incubator-beam/pull/1088#discussion_r83332515
> In the Java SDK, components can register display data directly, as well as 
> include display data from sub-components via the 'include(..)' operation. In 
> [PR #1008|https://github.com/apache/incubator-beam/pull/1088], we add 'paths' 
> to display data metadata and also include a 'delegate(..)' action.
> During the change for paths, we noticed an emerging pattern where components 
> will typically add the .getClass() of a subcomponent before including it. We 
> can support this convention better by embedding it in a single configurable 
> operation, '.nested(..)'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-768) Update Dataflow API to beta

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-768:


 Summary: Update Dataflow API to beta
 Key: BEAM-768
 URL: https://issues.apache.org/jira/browse/BEAM-768
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Daniel Halperin
Priority: Minor


Dataflow's API is now in {{beta}}, so the {{gcloud}} command strings printed 
need to be upgraded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Update Dataflow container image for current version of Apache Beam.

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 6b5ff4c4a -> ecc2abfdb


Update Dataflow container image for current version of Apache Beam.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/760f30dd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/760f30dd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/760f30dd

Branch: refs/heads/master
Commit: 760f30ddaf186f6d7b0a1c84e1a580ee249bd344
Parents: 6b5ff4c
Author: Luke Cwik 
Authored: Mon Oct 17 21:58:26 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 08:27:28 2016 -0700

--
 .../java/org/apache/beam/runners/dataflow/DataflowRunner.java| 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/760f30dd/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
index 55a01f7..5f83788 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
@@ -208,9 +208,9 @@ public class DataflowRunner extends 
PipelineRunner {
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20160926";
+  "dataflow.gcr.io/v1beta3/beam-java-batch:beam-master-20161017";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE =
-  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20160926";
+  "dataflow.gcr.io/v1beta3/beam-java-streaming:beam-master-20161017";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[GitHub] incubator-beam pull request #1124: Update Dataflow container image for curre...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1124


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1562

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1562

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1562

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1361

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1564

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1564

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-768) Update Dataflow API to beta

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585952#comment-15585952
 ] 

ASF GitHub Bot commented on BEAM-768:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1126

[BEAM-768] Dataflow: API is in beta now



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam alpha-beta-gamma-delta

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1126


commit a51443d60001fe11fc9e9e8e99e8560128479912
Author: Dan Halperin 
Date:   2016-10-18T16:42:50Z

Dataflow: API is in beta now




> Update Dataflow API to beta
> ---
>
> Key: BEAM-768
> URL: https://issues.apache.org/jira/browse/BEAM-768
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> Dataflow's API is now in {{beta}}, so the {{gcloud}} command strings printed 
> need to be upgraded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1126: [BEAM-768] Dataflow: API is in beta now

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1126

[BEAM-768] Dataflow: API is in beta now



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam alpha-beta-gamma-delta

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1126.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1126


commit a51443d60001fe11fc9e9e8e99e8560128479912
Author: Dan Halperin 
Date:   2016-10-18T16:42:50Z

Dataflow: API is in beta now




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-655) Rename @RunnableOnService to something more descriptive

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-655:
-
Summary: Rename @RunnableOnService to something more descriptive  (was: 
Rename @RunnableonService to something more descriptive)

> Rename @RunnableOnService to something more descriptive
> ---
>
> Key: BEAM-655
> URL: https://issues.apache.org/jira/browse/BEAM-655
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1094

[jbonofre]

This closes #1094


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a2c342cf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a2c342cf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a2c342cf

Branch: refs/heads/master
Commit: a2c342cfd2c2f86593e6d4f30c25b8054fa2ffa2
Parents: c5329f9 957c545
Author: Jean-Baptiste Onofré 
Authored: Tue Oct 18 16:54:12 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Tue Oct 18 16:54:12 2016 +0200

--
 .travis.yml   | 2 ++
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java  | 3 ++-
 .../java/org/apache/beam/runners/direct/DirectMetrics.java| 7 +++
 .../beam/runners/direct/TestStreamEvaluatorFactory.java   | 2 +-
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java| 5 +++--
 .../main/java/org/apache/beam/runners/spark/io/SourceRDD.java | 6 --
 6 files changed, 19 insertions(+), 6 deletions(-)
--

[GitHub] incubator-beam pull request #1093: [BEAM-593] avoid throwing Exception in wa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1093


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-593) Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in FlinkRunnerResult

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585696#comment-15585696
 ] 

ASF GitHub Bot commented on BEAM-593:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1093


> Support unblocking run() in FlinkRunner and cancel() and waitUntilFinish() in 
> FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1094: Build in eclipse/eclipse-jdt

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1094


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Build in eclipse/eclipse-jdt

[jbonofre]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c5329f9b4 -> a2c342cfd


Build in eclipse/eclipse-jdt


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/957c545e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/957c545e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/957c545e

Branch: refs/heads/master
Commit: 957c545eaa33c861b561418b1c7dadf4c31f92f3
Parents: c5329f9
Author: Daniel Kulp 
Authored: Thu Oct 13 12:41:32 2016 -0400
Committer: Jean-Baptiste Onofré 
Committed: Tue Oct 18 16:53:10 2016 +0200

--
 .travis.yml   | 2 ++
 .../beam/runners/direct/BoundedReadEvaluatorFactory.java  | 3 ++-
 .../java/org/apache/beam/runners/direct/DirectMetrics.java| 7 +++
 .../beam/runners/direct/TestStreamEvaluatorFactory.java   | 2 +-
 .../beam/runners/direct/UnboundedReadEvaluatorFactory.java| 5 +++--
 .../main/java/org/apache/beam/runners/spark/io/SourceRDD.java | 6 --
 6 files changed, 19 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/957c545e/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 786b370..5133a43 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -39,6 +39,8 @@ matrix:
 - os: linux
   env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 - os: linux
+  env: CUSTOM_JDK="oraclejdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m' -Peclipse-jdt"
+- os: linux
   env: CUSTOM_JDK="openjdk7" 
MAVEN_OVERRIDE="-DbeamSurefireArgline='-Xmx512m'"
 
 before_install:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/957c545e/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
index 843dcd6..add1e8a 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java
@@ -34,6 +34,7 @@ import org.apache.beam.sdk.transforms.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -137,7 +138,7 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
 
 private 
 Collection> 
createInitialSplits(
-AppliedPTransform> transform, int 
targetParallelism)
+AppliedPTransform transform, int 
targetParallelism)
 throws Exception {
   BoundedSource source = transform.getTransform().getSource();
   PipelineOptions options = evaluationContext.getPipelineOptions();

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/957c545e/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
index a749a76..145326f 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectMetrics.java
@@ -224,6 +224,13 @@ class DirectMetrics extends MetricResults {
 
   @AutoValue
   abstract static class DirectMetricResult implements MetricResult {
+// need to define these here so they appear in the correct order
+// and the generated constructor is usable and consistent
+public abstract MetricName name();
+public abstract String step();
+public abstract T committed();
+public abstract T attempted();
+
 public static  MetricResult create(MetricName name, String scope,
 T committed, T attempted) {
   return new AutoValue_DirectMetrics_DirectMetricResult(

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/957c545e/runners/direct-java/src/main/java/org/apache/beam/runners/direct/TestStreamEvaluatorFactory.java

[2/2] incubator-beam git commit: This closes #1093

[mxm]

This closes #1093


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b5ff4c4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b5ff4c4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b5ff4c4

Branch: refs/heads/master
Commit: 6b5ff4c4aad5e4d1419b1a147153b0f8d72324ae
Parents: a2c342c 76434df
Author: Maximilian Michels 
Authored: Tue Oct 18 16:59:58 2016 +0200
Committer: Maximilian Michels 
Committed: Tue Oct 18 16:59:58 2016 +0200

--
 .../flink/FlinkDetachedRunnerResult.java| 76 
 .../apache/beam/runners/flink/FlinkRunner.java  |  9 ++-
 .../beam/runners/flink/FlinkRunnerResult.java   | 11 +--
 .../beam/runners/flink/TestFlinkRunner.java |  9 ++-
 4 files changed, 91 insertions(+), 14 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-593] avoid throwing Exception in waitUntilFinish

[mxm]

Repository: incubator-beam
Updated Branches:
  refs/heads/master a2c342cfd -> 6b5ff4c4a


[BEAM-593] avoid throwing Exception in waitUntilFinish

The current implementation of Flink's PipelineResult assumes that the
pipeline has already been processed. Hence, we can return State.Done
when wailUntilFinished is called.

Additionally, we introduce a PipelineResult for detached execution which
returns State.UNKNOWN for now.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/76434dff
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/76434dff
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/76434dff

Branch: refs/heads/master
Commit: 76434dff650196c74afdeac917d8ceddb2550079
Parents: a2c342c
Author: Maximilian Michels 
Authored: Thu Oct 13 14:01:06 2016 +0200
Committer: Maximilian Michels 
Committed: Tue Oct 18 16:59:49 2016 +0200

--
 .../flink/FlinkDetachedRunnerResult.java| 76 
 .../apache/beam/runners/flink/FlinkRunner.java  |  9 ++-
 .../beam/runners/flink/FlinkRunnerResult.java   | 11 +--
 .../beam/runners/flink/TestFlinkRunner.java |  9 ++-
 4 files changed, 91 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/76434dff/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java
new file mode 100644
index 000..6adcf07
--- /dev/null
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkDetachedRunnerResult.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.flink;
+
+import java.io.IOException;
+
+import org.apache.beam.sdk.AggregatorRetrievalException;
+import org.apache.beam.sdk.AggregatorValues;
+import org.apache.beam.sdk.PipelineResult;
+import org.apache.beam.sdk.metrics.MetricResults;
+import org.apache.beam.sdk.transforms.Aggregator;
+import org.apache.commons.lang.NotImplementedException;
+import org.joda.time.Duration;
+
+
+/**
+ * Result of a detached execution of a {@link org.apache.beam.sdk.Pipeline} 
with Flink.
+ * In detached execution, results and job execution are currently unavailable.
+ */
+public class FlinkDetachedRunnerResult implements PipelineResult {
+
+  FlinkDetachedRunnerResult() {}
+
+  @Override
+  public State getState() {
+return State.UNKNOWN;
+  }
+
+  @Override
+  public  AggregatorValues getAggregatorValues(final Aggregator 
aggregator)
+  throws AggregatorRetrievalException {
+throw new AggregatorRetrievalException(
+"Accumulators can't be retrieved for detached Job executions.",
+new NotImplementedException());
+  }
+
+  @Override
+  public MetricResults metrics() {
+throw new UnsupportedOperationException("The FlinkRunner does not 
currently support metrics.");
+  }
+
+  @Override
+  public State cancel() throws IOException {
+throw new UnsupportedOperationException("Cancelling is not yet 
supported.");
+  }
+
+  @Override
+  public State waitUntilFinish() {
+return State.UNKNOWN;
+  }
+
+  @Override
+  public State waitUntilFinish(Duration duration) {
+return State.UNKNOWN;
+  }
+
+  @Override
+  public String toString() {
+return "FlinkDetachedRunnerResult{}";
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/76434dff/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java
index 932952d..12e21c7 100644
--- 

Jenkins build is unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1563

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1127: Restore trigger-related tests missed in #...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1127

Restore trigger-related tests missed in #1083

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 

This is a follow-up of #1118, restoring a couple more things missed in 
#1083.

The commit that merged #1083 is e969f3d. At the time I confirmed that 
`src/main` and `src/test/sdk/transforms/windowing` were unchanged, but 
neglected `util`. Here is a transcript demonstrating the genesis of this patch:

```
$ git diff --stat e969f3d^1 e969f3d^2 sdks/java/core/src/main
$ git diff --stat e969f3d^1 e969f3d^2 sdks/java/core/src/test
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
  | 127 
--
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersBitSetTest.java
 |  55 -
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersProperties.java
 | 110 
--
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/FinishedTriggersSetTest.java
|  60 --
 
sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java 
  |  67 
 5 files changed, 419 deletions(-)
```

I then checked each of these out at `e969f3d^1`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam trigger-util-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1127


commit b8c48171c9593da2652ed40a221cda0fbe30d49a
Author: Kenneth Knowles 
Date:   2016-10-18T17:11:37Z

Restore trigger-related tests missed in #1083




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-756) Checkstyle suppression for JavadocPackage not working on Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586076#comment-15586076
 ] 

ASF GitHub Bot commented on BEAM-756:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1114


> Checkstyle suppression for JavadocPackage not working on Windows
> 
>
> Key: BEAM-756
> URL: https://issues.apache.org/jira/browse/BEAM-756
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>
> Exclusions for test and other files don't consider '\' as separator. Hence 
> checkstyle complains about missing package-info files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-768) Update Dataflow API to beta

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-768.
--
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> Update Dataflow API to beta
> ---
>
> Key: BEAM-768
> URL: https://issues.apache.org/jira/browse/BEAM-768
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Dataflow's API is now in {{beta}}, so the {{gcloud}} command strings printed 
> need to be upgraded.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-744) UnboundedKafkaReader should return as soon as it can.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Summary: UnboundedKafkaReader should return as soon as it can.  (was: A 
runner should be able to override KafkaIO max wait properties.)

> UnboundedKafkaReader should return as soon as it can.
> -
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Resolve Registered Runners that don't end with Runner

[dhalperi]

Resolve Registered Runners that don't end with Runner

Resolve runners in a case-insensitive manner.

This reduces duplication in specifying a runner

e.g. the DirectRunner can be specified with (among others) any of
"--runner=direct", "--runner=directrunner", "--runner=DirectRunner",
"--runner=Direct", or "--runner=directRunner"


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e601410b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e601410b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e601410b

Branch: refs/heads/master
Commit: e601410b6597022c637be27cee86ab080274017a
Parents: 4469479
Author: Thomas Groh 
Authored: Wed Oct 12 13:07:52 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:52:59 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java | 44 +---
 .../sdk/options/PipelineOptionsFactoryTest.java | 23 +++---
 2 files changed, 48 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e601410b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index cd0c6b2..1c8a835 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -34,6 +34,7 @@ import com.google.common.collect.FluentIterable;
 import com.google.common.collect.ImmutableListMultimap;
 import com.google.common.collect.ImmutableMap;
 import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.ImmutableSortedSet;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Iterators;
 import com.google.common.collect.ListMultimap;
@@ -446,6 +447,7 @@ public class PipelineOptionsFactory {
   private static final Class[] EMPTY_CLASS_ARRAY = new Class[0];
   private static final ObjectMapper MAPPER = new ObjectMapper();
   private static final ClassLoader CLASS_LOADER;
+
   private static final Map> 
SUPPORTED_PIPELINE_RUNNERS;
 
   /** Classes that are used as the boundary in the stack trace to find the 
callers class name. */
@@ -514,16 +516,20 @@ public class PipelineOptionsFactory {
 
 CLASS_LOADER = findClassLoader();
 
-// Store the list of all available pipeline runners.
-ImmutableMap.Builder> builder =
-ImmutableMap.builder();
 Set pipelineRunnerRegistrars =
 Sets.newTreeSet(ObjectsClassComparator.INSTANCE);
 pipelineRunnerRegistrars.addAll(
 Lists.newArrayList(ServiceLoader.load(PipelineRunnerRegistrar.class, 
CLASS_LOADER)));
+// Store the list of all available pipeline runners.
+ImmutableMap.Builder> builder =
+ImmutableMap.builder();
 for (PipelineRunnerRegistrar registrar : pipelineRunnerRegistrars) {
   for (Class> klass : 
registrar.getPipelineRunners()) {
-builder.put(klass.getSimpleName(), klass);
+String runnerName = klass.getSimpleName().toLowerCase();
+builder.put(runnerName, klass);
+if (runnerName.endsWith("runner")) {
+  builder.put(runnerName.substring(0, runnerName.length() - 
"Runner".length()), klass);
+}
   }
 }
 SUPPORTED_PIPELINE_RUNNERS = builder.build();
@@ -1420,24 +1426,25 @@ public class PipelineOptionsFactory {
 JavaType type = 
MAPPER.getTypeFactory().constructType(method.getGenericReturnType());
 if ("runner".equals(entry.getKey())) {
   String runner = Iterables.getOnlyElement(entry.getValue());
-  if (SUPPORTED_PIPELINE_RUNNERS.containsKey(runner)) {
-convertedOptions.put("runner", 
SUPPORTED_PIPELINE_RUNNERS.get(runner));
+  if (SUPPORTED_PIPELINE_RUNNERS.containsKey(runner.toLowerCase())) {
+convertedOptions.put("runner", 
SUPPORTED_PIPELINE_RUNNERS.get(runner.toLowerCase()));
   } else {
 try {
   Class runnerClass = Class.forName(runner);
-  checkArgument(
-  PipelineRunner.class.isAssignableFrom(runnerClass),
-  "Class '%s' does not implement PipelineRunner. Supported 
pipeline runners %s",
-  runner,
-  Sets.newTreeSet(SUPPORTED_PIPELINE_RUNNERS.keySet()));
+  if (!(PipelineRunner.class.isAssignableFrom(runnerClass))) {
+throw new IllegalArgumentException(
+String.format(
+

[jira] [Commented] (BEAM-770) Improve registered runner resolution

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586149#comment-15586149
 ] 

ASF GitHub Bot commented on BEAM-770:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1087


> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: forward-incompatible
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Restore trigger-related tests missed in #1083

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 71c69b31b -> 6d686288e


Restore trigger-related tests missed in #1083


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8d43e8aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8d43e8aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8d43e8aa

Branch: refs/heads/master
Commit: 8d43e8aa7ccb154e17d6840c25c7a72684c615aa
Parents: 71c69b3
Author: Kenneth Knowles 
Authored: Tue Oct 18 10:11:37 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 11:00:47 2016 -0700

--
 .../beam/sdk/util/ExecutableTriggerTest.java| 127 +++
 .../sdk/util/FinishedTriggersBitSetTest.java|  55 
 .../sdk/util/FinishedTriggersProperties.java| 110 
 .../beam/sdk/util/FinishedTriggersSetTest.java  |  60 +
 4 files changed, 352 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8d43e8aa/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
new file mode 100644
index 000..1e3a1ff
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ExecutableTriggerTest.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertSame;
+
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ExecutableTrigger}.
+ */
+@RunWith(JUnit4.class)
+public class ExecutableTriggerTest {
+
+  @Test
+  public void testIndexAssignmentLeaf() throws Exception {
+StubTrigger t1 = new StubTrigger();
+ExecutableTrigger executable = ExecutableTrigger.create(t1);
+assertEquals(0, executable.getTriggerIndex());
+  }
+
+  @Test
+  public void testIndexAssignmentOneLevel() throws Exception {
+StubTrigger t1 = new StubTrigger();
+StubTrigger t2 = new StubTrigger();
+StubTrigger t = new StubTrigger(t1, t2);
+
+ExecutableTrigger executable = ExecutableTrigger.create(t);
+
+assertEquals(0, executable.getTriggerIndex());
+assertEquals(1, executable.subTriggers().get(0).getTriggerIndex());
+assertSame(t1, executable.subTriggers().get(0).getSpec());
+assertEquals(2, executable.subTriggers().get(1).getTriggerIndex());
+assertSame(t2, executable.subTriggers().get(1).getSpec());
+  }
+
+  @Test
+  public void testIndexAssignmentTwoLevel() throws Exception {
+StubTrigger t11 = new StubTrigger();
+StubTrigger t12 = new StubTrigger();
+StubTrigger t13 = new StubTrigger();
+StubTrigger t14 = new StubTrigger();
+StubTrigger t21 = new StubTrigger();
+StubTrigger t22 = new StubTrigger();
+StubTrigger t1 = new StubTrigger(t11, t12, t13, t14);
+StubTrigger t2 = new StubTrigger(t21, t22);
+StubTrigger t = new StubTrigger(t1, t2);
+
+ExecutableTrigger executable = ExecutableTrigger.create(t);
+
+assertEquals(0, executable.getTriggerIndex());
+assertEquals(1, executable.subTriggers().get(0).getTriggerIndex());
+assertEquals(6, 
executable.subTriggers().get(0).getFirstIndexAfterSubtree());
+assertEquals(6, executable.subTriggers().get(1).getTriggerIndex());
+
+assertSame(t1, executable.getSubTriggerContaining(1).getSpec());
+assertSame(t2, executable.getSubTriggerContaining(6).getSpec());
+assertSame(t1, executable.getSubTriggerContaining(2).getSpec());
+assertSame(t1, 

[GitHub] incubator-beam pull request #1127: Restore trigger-related tests missed in #...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1127


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_MavenVerify #1569

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Restore ReshuffleTriggerTest to SDK

[lcwik]

Restore ReshuffleTriggerTest to SDK

This closes #1118


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4e14ac2c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4e14ac2c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4e14ac2c

Branch: refs/heads/master
Commit: 4e14ac2c2444f3d7f6e17286d0a4b9a500244aca
Parents: c81376b 619746b
Author: Luke Cwik 
Authored: Tue Oct 18 09:48:05 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 09:48:05 2016 -0700

--
 .../beam/sdk/util/ReshuffleTriggerTest.java | 67 
 1 file changed, 67 insertions(+)
--

[1/2] incubator-beam git commit: BEAM-756 fix JavadocPackage checkstyle suppression

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4e14ac2c2 -> 006cb127e


BEAM-756 fix JavadocPackage checkstyle suppression


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a691843
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a691843
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a691843

Branch: refs/heads/master
Commit: 6a691843460255abec96690778954990246a6b9d
Parents: 4e14ac2
Author: Thomas Weise 
Authored: Mon Oct 17 11:39:45 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:07:32 2016 -0700

--
 .../src/main/resources/beam/suppressions.xml| 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6a691843/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml 
b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
index 00d6729..9f60c25 100644
--- a/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/suppressions.xml
@@ -17,13 +17,13 @@
 "http://www.puppycrawl.com/dtds/suppressions_1_1.dtd;>
 
 
-  
-  
-  
+  
+  
+  
   
   
-  
-  
-  
+  
+  
+  
 
 

[GitHub] incubator-beam pull request #1114: BEAM-756 fix JavadocPackage checkstyle su...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1114


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: BEAM-756 fix JavadocPackage checkstyle suppression

[lcwik]

BEAM-756 fix JavadocPackage checkstyle suppression

This closes #1114


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/006cb127
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/006cb127
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/006cb127

Branch: refs/heads/master
Commit: 006cb127e84e045d58f13705e81d4769ae0cbf89
Parents: 4e14ac2 6a69184
Author: Luke Cwik 
Authored: Tue Oct 18 10:07:59 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:07:59 2016 -0700

--
 .../src/main/resources/beam/suppressions.xml| 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)
--

[GitHub] incubator-beam pull request #1070: [BEAM-733] Prohibit windowed side inputs ...

[robertwb]

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/1070


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Dissallow (unimplemented) windowed side inputs.

[robertwb]

Dissallow (unimplemented) windowed side inputs.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/552f6d7b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/552f6d7b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/552f6d7b

Branch: refs/heads/python-sdk
Commit: 552f6d7baa6e6205b290d21e642911a04ca259ec
Parents: 24b7bcc
Author: Robert Bradshaw 
Authored: Fri Oct 7 15:28:50 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:48:16 2016 -0700

--
 .../python/apache_beam/transforms/sideinputs.py |  4 ++
 .../apache_beam/transforms/sideinputs_test.py   | 40 
 2 files changed, 44 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/552f6d7b/sdks/python/apache_beam/transforms/sideinputs.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs.py 
b/sdks/python/apache_beam/transforms/sideinputs.py
index 6484a7c..6c698da 100644
--- a/sdks/python/apache_beam/transforms/sideinputs.py
+++ b/sdks/python/apache_beam/transforms/sideinputs.py
@@ -50,6 +50,10 @@ class CreatePCollectionView(PTransform):
 return input_type
 
   def apply(self, pcoll):
+if not pcoll.windowing.is_default():
+  raise ValueError(
+  "Side inputs only supported for global windows, default triggering. "
+  "Found %s" % pcoll.windowing)
 return self.view
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/552f6d7b/sdks/python/apache_beam/transforms/sideinputs_test.py
--
diff --git a/sdks/python/apache_beam/transforms/sideinputs_test.py 
b/sdks/python/apache_beam/transforms/sideinputs_test.py
new file mode 100644
index 000..8e292e3
--- /dev/null
+++ b/sdks/python/apache_beam/transforms/sideinputs_test.py
@@ -0,0 +1,40 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Unit tests for side inputs."""
+
+import logging
+import unittest
+
+import apache_beam as beam
+from apache_beam.transforms import window
+
+
+class SideInputsTest(unittest.TestCase):
+
+  # TODO(BEAM-733): Actually support this.
+  def test_no_sideinput_windowing(self):
+p = beam.Pipeline('DirectPipelineRunner')
+pc = p | beam.Create([0, 1]) | beam.WindowInto(window.FixedWindows(10))
+with self.assertRaises(ValueError):
+  # pylint: disable=expression-not-assigned
+  pc | beam.Map(lambda x, side: None, side=beam.pvalue.AsIter(pc))
+
+
+if __name__ == '__main__':
+  logging.getLogger().setLevel(logging.DEBUG)
+  unittest.main()

[jira] [Commented] (BEAM-733) Windowed side-inputs not supported in Python

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586145#comment-15586145
 ] 

ASF GitHub Bot commented on BEAM-733:
-

Github user robertwb closed the pull request at:

https://github.com/apache/incubator-beam/pull/1070


> Windowed side-inputs not supported in Python
> 
>
> Key: BEAM-733
> URL: https://issues.apache.org/jira/browse/BEAM-733
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Labels: forward-incompatible  (was: )

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>  Labels: forward-incompatible
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586153#comment-15586153
 ] 

Daniel Halperin commented on BEAM-770:
--

Mild compatibility change -- new feature.

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: [BEAM-764] Remove cloneAs from PipelineOptions

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 75f54682a -> 71c69b31b


[BEAM-764] Remove cloneAs from PipelineOptions


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/71c69b31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/71c69b31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/71c69b31

Branch: refs/heads/master
Commit: 71c69b31b6894064bf8111007f947150ff725528
Parents: 75f5468
Author: Pei He 
Authored: Mon Oct 17 14:13:42 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 10:56:43 2016 -0700

--
 .../beam/sdk/options/PipelineOptions.java   |  8 
 .../sdk/options/PipelineOptionsFactory.java |  1 -
 .../sdk/options/ProxyInvocationHandler.java | 22 --
 .../beam/sdk/options/PipelineOptionsTest.java   | 43 
 4 files changed, 74 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
index 3d6cad6..5588543 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptions.java
@@ -211,14 +211,6 @@ public interface PipelineOptions extends HasDisplayData {
T as(Class kls);
 
   /**
-   * Makes a deep clone of this object, and transforms the cloned object into 
the specified
-   * type {@code kls}. See {@link #as} for more information about the 
conversion.
-   *
-   * Properties that are marked with {@code @JsonIgnore} will not be cloned.
-   */
-   T cloneAs(Class kls);
-
-  /**
* The pipeline runner that will be used to execute the pipeline.
* For registered runners, the class name can be specified, otherwise the 
fully
* qualified name needs to be specified.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
index 1c8a835..7206b11 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/PipelineOptionsFactory.java
@@ -989,7 +989,6 @@ public class PipelineOptionsFactory {
 // Ignore methods on the base PipelineOptions interface.
 try {
   methods.add(iface.getMethod("as", Class.class));
-  methods.add(iface.getMethod("cloneAs", Class.class));
   methods.add(iface.getMethod("populateDisplayData", 
DisplayData.Builder.class));
 } catch (NoSuchMethodException | SecurityException e) {
   throw new RuntimeException(e);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/71c69b31/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
index 47d7cee..a77dcc6 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/options/ProxyInvocationHandler.java
@@ -134,10 +134,6 @@ class ProxyInvocationHandler implements InvocationHandler, 
HasDisplayData {
   @SuppressWarnings("unchecked")
   Class clazz = (Class) args[0];
   return as(clazz);
-} else if (args != null && "cloneAs".equals(method.getName()) && args[0] 
instanceof Class) {
-  @SuppressWarnings("unchecked")
-  Class clazz = (Class) args[0];
-  return cloneAs(proxy, clazz);
 } else if (args != null && "populateDisplayData".equals(method.getName())
 && args[0] instanceof DisplayData.Builder) {
   @SuppressWarnings("unchecked")
@@ -223,24 +219,6 @@ class ProxyInvocationHandler implements InvocationHandler, 
HasDisplayData {
   }
 
   /**
-   * Backing implementation for {@link PipelineOptions#cloneAs(Class)}.
-   *
-   * @return A copy of the PipelineOptions.
-   */
-  synchronized  T cloneAs(Object proxy, Class 
iface) {
-PipelineOptions clonedOptions;
-try {
-  clonedOptions = 

[jira] [Issue Comment Deleted] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Comment: was deleted

(was: Mild compatibility change -- new feature.)

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-770) Improve registered runner resolution

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-770:
-
Labels:   (was: forward-incompatible)

> Improve registered runner resolution
> 
>
> Key: BEAM-770
> URL: https://issues.apache.org/jira/browse/BEAM-770
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> Runner is generally a duplicate word within the Pipeline args specification - 
> specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
> PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
> there's a registered {{FooRunner}} and use that.
> Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1568

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify #1567

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1567

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1118: Restore ReshuffleTriggerTest to SDK

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1118


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-655) Rename @RunnableOnService to something more descriptive

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh updated BEAM-655:
-
Description: 
There are currently two categories for unit tests in the Core Beam Java SDK: 
{{NeedsRunner}} and {{RunnableOnService}}. These categories are insufficiently 
descriptive for describing the tests that they categorize.

There are two goals for these tests:
* Demonstrate that a runner conforms with the Beam model and the Capability 
Matrix. These capabilities are described at [Capability Matrix 
Testing|https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2VlUyVtpi2WzzGM/edit]
* Demonstrate that a PTransform, when executed on a {{PipelineRunner}}, 
performs as specified

NeedsRunner tests should either:
* Be converted into a runner-independent test (for example, via use of 
DoFnTester to test that a DoFn throws on invalid input)
* Be converted into a RunnableOnService test

RunnableOnService tests that explicitly test the capabilities of a Model 
primitive should be categorized into that specific capability. Existing 
RunnableOnService tests that test the execution of a PTransform should remain 
RunnableOnService.


> Rename @RunnableOnService to something more descriptive
> ---
>
> Key: BEAM-655
> URL: https://issues.apache.org/jira/browse/BEAM-655
> Project: Beam
>  Issue Type: Bug
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>
> There are currently two categories for unit tests in the Core Beam Java SDK: 
> {{NeedsRunner}} and {{RunnableOnService}}. These categories are 
> insufficiently descriptive for describing the tests that they categorize.
> There are two goals for these tests:
> * Demonstrate that a runner conforms with the Beam model and the Capability 
> Matrix. These capabilities are described at [Capability Matrix 
> Testing|https://docs.google.com/document/d/1fICxq32t9yWn9qXhmT07xpclHeHX2VlUyVtpi2WzzGM/edit]
> * Demonstrate that a PTransform, when executed on a {{PipelineRunner}}, 
> performs as specified
> NeedsRunner tests should either:
> * Be converted into a runner-independent test (for example, via use of 
> DoFnTester to test that a DoFn throws on invalid input)
> * Be converted into a RunnableOnService test
> RunnableOnService tests that explicitly test the capabilities of a Model 
> primitive should be categorized into that specific capability. Existing 
> RunnableOnService tests that test the execution of a PTransform should remain 
> RunnableOnService.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: Restore trigger-related tests missed in #1083

[lcwik]

Restore trigger-related tests missed in #1083

This closes #1127


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6d686288
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6d686288
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6d686288

Branch: refs/heads/master
Commit: 6d686288efd5fd64d43ba9802314f2cbbc8df72e
Parents: 71c69b3 8d43e8a
Author: Luke Cwik 
Authored: Tue Oct 18 11:01:14 2016 -0700
Committer: Luke Cwik 
Committed: Tue Oct 18 11:01:14 2016 -0700

--
 .../beam/sdk/util/ExecutableTriggerTest.java| 127 +++
 .../sdk/util/FinishedTriggersBitSetTest.java|  55 
 .../sdk/util/FinishedTriggersProperties.java| 110 
 .../beam/sdk/util/FinishedTriggersSetTest.java  |  60 +
 4 files changed, 352 insertions(+)
--

[jira] [Commented] (BEAM-769) Spark translation tests failing with NullPointerException

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586248#comment-15586248
 ] 

Jason Kuster commented on BEAM-769:
---

This also seems to have caused a postcommit failure:

https://builds.apache.org/job/beam_PostCommit_MavenVerify/1567/

> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Restore ReshuffleTriggerTest to SDK

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c81376bc7 -> 4e14ac2c2


Restore ReshuffleTriggerTest to SDK


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/619746b6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/619746b6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/619746b6

Branch: refs/heads/master
Commit: 619746b6cdbd46bc108f8b0d0214a9e8f266b6b1
Parents: c81376b
Author: Kenneth Knowles 
Authored: Mon Oct 17 12:44:29 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Oct 18 09:38:41 2016 -0700

--
 .../beam/sdk/util/ReshuffleTriggerTest.java | 67 
 1 file changed, 67 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/619746b6/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java
new file mode 100644
index 000..83077f4
--- /dev/null
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/util/ReshuffleTriggerTest.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.FixedWindows;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/**
+ * Tests for {@link ReshuffleTrigger}.
+ */
+@RunWith(JUnit4.class)
+public class ReshuffleTriggerTest {
+
+  /** Public so that other tests can instantiate {@link ReshuffleTrigger}. */
+  public static  ReshuffleTrigger forTest() {
+return new ReshuffleTrigger<>();
+  }
+
+  @Test
+  public void testShouldFire() throws Exception {
+TriggerTester tester = TriggerTester.forTrigger(
+new ReshuffleTrigger(), 
FixedWindows.of(Duration.millis(100)));
+IntervalWindow arbitraryWindow = new IntervalWindow(new Instant(300), new 
Instant(400));
+assertTrue(tester.shouldFire(arbitraryWindow));
+  }
+
+  @Test
+  public void testOnTimer() throws Exception {
+TriggerTester tester = TriggerTester.forTrigger(
+new ReshuffleTrigger(), 
FixedWindows.of(Duration.millis(100)));
+IntervalWindow arbitraryWindow = new IntervalWindow(new Instant(100), new 
Instant(200));
+tester.fireIfShouldFire(arbitraryWindow);
+assertFalse(tester.isMarkedFinished(arbitraryWindow));
+  }
+
+  @Test
+  public void testToString() {
+Trigger trigger = new ReshuffleTrigger<>();
+assertEquals("ReshuffleTrigger()", trigger.toString());
+  }
+}

[2/2] incubator-beam git commit: Closes #1126

[dhalperi]

Closes #1126


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/44694794
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/44694794
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/44694794

Branch: refs/heads/master
Commit: 446947947585ef8c7aa2b29ac1464465793abfe2
Parents: 006cb12 772fb3f
Author: Dan Halperin 
Authored: Tue Oct 18 10:40:15 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:40:15 2016 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--

[jira] [Updated] (BEAM-769) Spark translation tests failing with NullPointerException

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-769:
-
Description: 
https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/

{code}
org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?

  was:
{code}

org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?


> Spark translation tests failing with NullPointerException
> -
>
> Key: BEAM-769
> URL: https://issues.apache.org/jira/browse/BEAM-769
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Amit Sela
>
> https://builds.apache.org/job/beam_PreCommit_MavenVerify/4071/
> {code}
> org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
> org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
> org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
> org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
> {code}
> Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-769) Spark translation tests failing with NullPointerException

[Daniel Halperin (JIRA)]

Daniel Halperin created BEAM-769:


 Summary: Spark translation tests failing with NullPointerException
 Key: BEAM-769
 URL: https://issues.apache.org/jira/browse/BEAM-769
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Affects Versions: Not applicable
Reporter: Daniel Halperin
Assignee: Amit Sela


{code}

org.apache.beam.runners.spark.translation.streaming.EmptyStreamAssertionTest.testFixedWindows
org.apache.beam.runners.spark.translation.streaming.FlattenStreamingTest.testFlattenUnbounded
org.apache.beam.runners.spark.translation.streaming.KafkaStreamingTest.testRun
org.apache.beam.runners.spark.translation.streaming.SimpleStreamingWordCountTest.testFixedWindows
{code}

Amit, can you please take a look?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1126: [BEAM-768] Dataflow: API is in beta now

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1087: [BEAM-770] Resolve Registered Runners tha...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1087


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #1087

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 446947947 -> 75f54682a


Closes #1087


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/75f54682
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/75f54682
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/75f54682

Branch: refs/heads/master
Commit: 75f54682a705aa0776c12bc9c2dcb8cb9a7c08fa
Parents: 4469479 e601410
Author: Dan Halperin 
Authored: Tue Oct 18 10:52:59 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:52:59 2016 -0700

--
 .../sdk/options/PipelineOptionsFactory.java | 44 +---
 .../sdk/options/PipelineOptionsFactoryTest.java | 23 +++---
 2 files changed, 48 insertions(+), 19 deletions(-)
--

[jira] [Updated] (BEAM-744) UnboundedKafkaReader should return as soon as it can.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Description: 
KafkaIO has two "wait" properties:

{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

Instead, they should return as soon as they can, [~rangadi] mentioned here: 
https://github.com/apache/incubator-beam/pull/1071 that they were both 
originally set to 10 milliseconds, so {{START_NEW_RECORDS_POLL_TIMEOUT}} should 
probably be re-set to 10 milliseconds.

  was:
KafkaIO has two "wait" properties:

{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

[~rangadi] mentioned some of these were set to due to limitations of the 
DirectRunner, and I can add that they are now limiting the Spark runner (which 
reads in defined time frames, which may be smaller then the wait time and so 
never actually read).

This feels like defaults should be set for optimal read from Kafka, while a 
runner may override those if it needs to.

[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
may be passed when creating the reader. 


> UnboundedKafkaReader should return as soon as it can.
> -
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has two "wait" properties:
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> Instead, they should return as soon as they can, [~rangadi] mentioned here: 
> https://github.com/apache/incubator-beam/pull/1071 that they were both 
> originally set to 10 milliseconds, so {{START_NEW_RECORDS_POLL_TIMEOUT}} 
> should probably be re-set to 10 milliseconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1569

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Dataflow: API is in beta now

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 006cb127e -> 446947947


Dataflow: API is in beta now


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/772fb3f1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/772fb3f1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/772fb3f1

Branch: refs/heads/master
Commit: 772fb3f1351bdad04adc6aeb3461daa0964cce92
Parents: 006cb12
Author: Dan Halperin 
Authored: Tue Oct 18 09:42:50 2016 -0700
Committer: Dan Halperin 
Committed: Tue Oct 18 10:40:14 2016 -0700

--
 .../org/apache/beam/runners/dataflow/util/MonitoringUtil.java| 2 +-
 .../apache/beam/runners/dataflow/util/MonitoringUtilTest.java| 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/772fb3f1/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
index d014623..a6afccd 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/MonitoringUtil.java
@@ -46,7 +46,7 @@ import org.slf4j.LoggerFactory;
  */
 public final class MonitoringUtil {
 
-  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud alpha dataflow";
+  private static final String GCLOUD_DATAFLOW_PREFIX = "gcloud beta dataflow";
   private static final String ENDPOINT_OVERRIDE_ENV_VAR =
   "CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW";
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/772fb3f1/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
index 98d8ab6..6c5a2be 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/MonitoringUtilTest.java
@@ -133,7 +133,7 @@ public class MonitoringUtilTest {
 options.setProject(PROJECT_ID);
 options.setGcpCredential(new TestCredential());
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
-assertEquals("gcloud alpha dataflow jobs --project=someProject cancel 
1234", cancelCommand);
+assertEquals("gcloud beta dataflow jobs --project=someProject cancel 
1234", cancelCommand);
   }
 
   @Test
@@ -147,7 +147,7 @@ public class MonitoringUtilTest {
 String cancelCommand = MonitoringUtil.getGcloudCancelCommand(options, 
JOB_ID);
 assertEquals(
 
"CLOUDSDK_API_ENDPOINT_OVERRIDES_DATAFLOW=https://dataflow.googleapis.com/v0neverExisted/
 "
-+ "gcloud alpha dataflow jobs --project=someProject cancel 1234",
++ "gcloud beta dataflow jobs --project=someProject cancel 1234",
 cancelCommand);
   }
 

[jira] [Created] (BEAM-770) Improve registered runner resolution

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-770:


 Summary: Improve registered runner resolution
 Key: BEAM-770
 URL: https://issues.apache.org/jira/browse/BEAM-770
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


Runner is generally a duplicate word within the Pipeline args specification - 
specifically {{--runner=FooRunner}} duplicates the word "Runner"; the 
PipelineOptionsFactory can check to see if, when passed {{--runner=Foo}}, if 
there's a registered {{FooRunner}} and use that.

Making matching case-insensitive would be nice to have.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] incubator-beam git commit: Closes #1070

[robertwb]

Closes #1070


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/afa5ebc7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/afa5ebc7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/afa5ebc7

Branch: refs/heads/python-sdk
Commit: afa5ebc70aef3a7ad20b4e70a62e23a90bd7547a
Parents: 24b7bcc 5d9506d
Author: Robert Bradshaw 
Authored: Tue Oct 18 10:53:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:53:59 2016 -0700

--
 sdks/python/apache_beam/pipeline_test.py|  1 +
 .../python/apache_beam/transforms/sideinputs.py |  4 ++
 .../apache_beam/transforms/sideinputs_test.py   | 40 
 sdks/python/apache_beam/transforms/util.py  |  7 +++-
 4 files changed, 50 insertions(+), 2 deletions(-)
--

[1/3] incubator-beam git commit: Fix tests unnecessarily using windowed side inputs

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 24b7bcc26 -> afa5ebc70


Fix tests unnecessarily using windowed side inputs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5d9506df
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5d9506df
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5d9506df

Branch: refs/heads/python-sdk
Commit: 5d9506df44e5b2cd703702d4188c25c5bbb16e7f
Parents: 552f6d7
Author: Robert Bradshaw 
Authored: Fri Oct 7 16:17:47 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 10:48:16 2016 -0700

--
 sdks/python/apache_beam/pipeline_test.py   | 1 +
 sdks/python/apache_beam/transforms/util.py | 7 +--
 2 files changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d9506df/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index 8c1b3ba..dedd732 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -239,6 +239,7 @@ class PipelineTest(unittest.TestCase):
 'oom:combine/GroupByKey/group_by_key': 1,
 ('oom:check', None): 1,
 'assert_that/singleton': 1,
+('assert_that/WindowInto', None): 1,
 ('assert_that/Map(match)', None): 1,
 ('oom:combine/GroupByKey/group_by_window', None): 1,
 ('oom:combine/Combine/ParDo(CombineValuesDoFn)', None): 1})

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5d9506df/sdks/python/apache_beam/transforms/util.py
--
diff --git a/sdks/python/apache_beam/transforms/util.py 
b/sdks/python/apache_beam/transforms/util.py
index 4564cf9..aeac0d9 100644
--- a/sdks/python/apache_beam/transforms/util.py
+++ b/sdks/python/apache_beam/transforms/util.py
@@ -21,6 +21,8 @@
 from __future__ import absolute_import
 
 from apache_beam.pvalue import AsIter as AllOf
+from apache_beam.transforms import core
+from apache_beam.transforms import window
 from apache_beam.transforms.core import CombinePerKey, Create, Flatten, 
GroupByKey, Map
 from apache_beam.transforms.ptransform import PTransform
 from apache_beam.transforms.ptransform import ptransform_fn
@@ -220,8 +222,9 @@ def assert_that(actual, matcher, label='assert_that'):
   class AssertThat(PTransform):
 
 def apply(self, pipeline):
-  return pipeline | 'singleton' >> Create([None]) | Map(match,
-AllOf(actual))
+  return pipeline | 'singleton' >> Create([None]) | Map(
+  match,
+  AllOf(actual | core.WindowInto(window.GlobalWindows(
 
 def default_label(self):
   return label

[GitHub] incubator-beam pull request #1128: [BEAM-771] Add a Residual Shard for Empty...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1128

[BEAM-771] Add a Residual Shard for Empty Readers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam read_when_no_inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1128


commit 87f26972c02bb7640c1eee5069ad842a4a3ab1a0
Author: Thomas Groh 
Date:   2016-10-18T17:57:53Z

Add a Residual Shard for Empty Readers

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-771) UnboundedReadEvaluator should reuse the input reader if it contains no elements

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586181#comment-15586181
 ] 

ASF GitHub Bot commented on BEAM-771:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1128

[BEAM-771] Add a Residual Shard for Empty Readers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam read_when_no_inputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1128


commit 87f26972c02bb7640c1eee5069ad842a4a3ab1a0
Author: Thomas Groh 
Date:   2016-10-18T17:57:53Z

Add a Residual Shard for Empty Readers

When the UnboundedReadEvaluator starts, it checks to see if the reader
has any elements available. If not, it immediately terminates (without
taking a new checkpoint). It should also ensure that the result contains
a residual shard that will continue to read from the shard, utilizing
the same reader if possible.




> UnboundedReadEvaluator should reuse the input reader if it contains no 
> elements
> ---
>
> Key: BEAM-771
> URL: https://issues.apache.org/jira/browse/BEAM-771
> Project: Beam
>  Issue Type: Bug
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> The UnboundedReadEvaluator checks, after it obtains the reader, that the 
> reader contains elements, and terminates without taking a new checkpoint if 
> not. It must also (but currently does not) add that reader and shard back as 
> a residual element.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Add support for experiments

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk afa5ebc70 -> fe1f39609


Add support for experiments

Adds a new --experiment flag, to allow runners to optionally enable
experimental features.

This is similar to the same flag in the java sdk.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2c5f3a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2c5f3a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2c5f3a8

Branch: refs/heads/python-sdk
Commit: b2c5f3a8f227bdb2cf0092c489701f4e1d762b7b
Parents: afa5ebc
Author: Ahmet Altay 
Authored: Mon Oct 17 18:17:24 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:04:58 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py  |  6 ++
 sdks/python/apache_beam/utils/options.py   |  9 +
 sdks/python/apache_beam/utils/pipeline_options_test.py | 10 +-
 3 files changed, 24 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/internal/apiclient.py 
b/sdks/python/apache_beam/internal/apiclient.py
index 3f82f29..8c7cc29 100644
--- a/sdks/python/apache_beam/internal/apiclient.py
+++ b/sdks/python/apache_beam/internal/apiclient.py
@@ -37,6 +37,7 @@ from apache_beam.utils import retry
 from apache_beam.utils.dependency import get_required_container_version
 from apache_beam.utils.dependency import get_sdk_name_and_version
 from apache_beam.utils.names import PropertyNames
+from apache_beam.utils.options import DebugOptions
 from apache_beam.utils.options import GoogleCloudOptions
 from apache_beam.utils.options import StandardOptions
 from apache_beam.utils.options import WorkerOptions
@@ -107,6 +108,7 @@ class Environment(object):
 self.standard_options = options.view_as(StandardOptions)
 self.google_cloud_options = options.view_as(GoogleCloudOptions)
 self.worker_options = options.view_as(WorkerOptions)
+self.debug_options = options.view_as(DebugOptions)
 self.proto = dataflow.Environment()
 self.proto.clusterManagerApiService = COMPUTE_API_SERVICE
 self.proto.dataset = '%s/cloud_dataflow' % BIGQUERY_API_SERVICE
@@ -141,6 +143,10 @@ class Environment(object):
 value=to_json_value(job_type)),
 dataflow.Environment.VersionValue.AdditionalProperty(
 key='major', value=to_json_value(environment_version))])
+# Experiments
+if self.debug_options.experiments:
+  for experiment in self.debug_options.experiments:
+self.proto.experiments.append(experiment)
 # Worker pool(s) information.
 package_descriptors = []
 for package in packages:

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/utils/options.py
--
diff --git a/sdks/python/apache_beam/utils/options.py 
b/sdks/python/apache_beam/utils/options.py
index a3cea1e..4a56ee2 100644
--- a/sdks/python/apache_beam/utils/options.py
+++ b/sdks/python/apache_beam/utils/options.py
@@ -364,6 +364,15 @@ class DebugOptions(PipelineOptions):
 parser.add_argument('--dataflow_job_file',
 default=None,
 help='Debug file to write the workflow specification.')
+parser.add_argument(
+'--experiment',
+dest='experiments',
+action='append',
+default=None,
+help=
+('Runners may provide a number of experimental features that can be '
+ 'enabled with this flag. Please sync with the owners of the runner '
+ 'before enabling any experiments.'))
 
 
 class ProfilingOptions(PipelineOptions):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2c5f3a8/sdks/python/apache_beam/utils/pipeline_options_test.py
--
diff --git a/sdks/python/apache_beam/utils/pipeline_options_test.py 
b/sdks/python/apache_beam/utils/pipeline_options_test.py
index 249aa04..ddeecaf 100644
--- a/sdks/python/apache_beam/utils/pipeline_options_test.py
+++ b/sdks/python/apache_beam/utils/pipeline_options_test.py
@@ -80,7 +80,7 @@ class PipelineOptionsTest(unittest.TestCase):
   PipelineOptionsTest.MockOptions).mock_option,
case['expected']['mock_option'])
 
-  def test_option_with_spcae(self):
+  def test_option_with_space(self):
 options = PipelineOptions(flags=['--option with space= value with space'])
 self.assertEqual(
 getattr(options.view_as(PipelineOptionsTest.MockOptions),

[2/2] incubator-beam git commit: Closes #1123

[robertwb]

Closes #1123


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fe1f3960
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fe1f3960
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fe1f3960

Branch: refs/heads/python-sdk
Commit: fe1f39609fe262b48fc647d6af684f30c130b27b
Parents: afa5ebc b2c5f3a
Author: Robert Bradshaw 
Authored: Tue Oct 18 12:04:59 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:04:59 2016 -0700

--
 sdks/python/apache_beam/internal/apiclient.py  |  6 ++
 sdks/python/apache_beam/utils/options.py   |  9 +
 sdks/python/apache_beam/utils/pipeline_options_test.py | 10 +-
 3 files changed, 24 insertions(+), 1 deletion(-)
--

[1/3] incubator-beam git commit: Document that source objects should not be mutated.

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk fe1f39609 -> 6e6d89d48


Document that source objects should not be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentracy tests for text and Avro sources.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2ab8d62a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2ab8d62a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2ab8d62a

Branch: refs/heads/python-sdk
Commit: 2ab8d62ac48481a52fa04c704491f3a5889de27c
Parents: fe1f396
Author: Chamikara Jayalath 
Authored: Wed Oct 12 16:51:20 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:08:41 2016 -0700

--
 sdks/python/apache_beam/io/avroio_test.py   | 28 ++-
 sdks/python/apache_beam/io/iobase.py|  8 ++
 sdks/python/apache_beam/io/source_test_utils.py | 13 +---
 sdks/python/apache_beam/io/textio.py| 80 +++-
 sdks/python/apache_beam/io/textio_test.py   | 37 +
 5 files changed, 121 insertions(+), 45 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2ab8d62a/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index 1c96d72..eb2c81c 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -110,7 +110,7 @@ class TestAvro(unittest.TestCase):
 return file_name_prefix + os.path.sep + 'mytemp*'
 
   def _run_avro_test(self, pattern, desired_bundle_size, perform_splitting,
- expected_result):
+ expected_result, test_reentrancy=False):
 source = AvroSource(pattern)
 
 read_records = []
@@ -128,9 +128,23 @@ class TestAvro(unittest.TestCase):
   (split.source, split.start_position, split.stop_position)
   for split in splits
   ]
+  if test_reentrancy:
+for source_info in sources_info:
+  reader_iter = source_info[0].read(source_info[0].get_range_tracker(
+  source_info[1], source_info[2]))
+  try:
+next(reader_iter)
+  except StopIteration:
+# Ignoring empty bundle
+pass
+
   source_test_utils.assertSourcesEqualReferenceSource((source, None, None),
   sources_info)
 else:
+  if test_reentrancy:
+reader_iter = source.read(source.get_range_tracker(None, None))
+next(reader_iter)
+
   read_records = source_test_utils.readFromSource(source, None, None)
   self.assertItemsEqual(expected_result, read_records)
 
@@ -144,6 +158,18 @@ class TestAvro(unittest.TestCase):
 expected_result = self.RECORDS
 self._run_avro_test(file_name, 100, True, expected_result)
 
+  def test_read_reentrant_without_splitting(self):
+file_name = self._write_data()
+expected_result = self.RECORDS
+self._run_avro_test(file_name, None, False, expected_result,
+test_reentrancy=True)
+
+  def test_read_reantrant_with_splitting(self):
+file_name = self._write_data()
+expected_result = self.RECORDS
+self._run_avro_test(file_name, 100, True, expected_result,
+test_reentrancy=True)
+
   def test_read_without_splitting_multiple_blocks(self):
 file_name = self._write_data(count=12000)
 expected_result = self.RECORDS * 2000

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2ab8d62a/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index 8239e26..edd3524 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -91,6 +91,14 @@ class BoundedSource(object):
   positions passed to the method ``get_range_tracker()`` are ``None``
   (2) Method read() will be invoked with the ``RangeTracker`` obtained in the
   previous step.
+
+  **Mutability**
+
+  A ``BoundedSource`` object should be fully mutated before being submitted
+  for reading. A ``BoundedSource`` object should not be mutated while
+  its methods (for example, ``read()``) are being invoked by a runner. Runner
+  implementations may invoke methods of ``BoundedSource`` objects through
+  multi-threaded and/or re-entrant execution modes.
   """
 
   def 

[2/3] incubator-beam git commit: Adds an assertion to source_test_utils for testing reentrancy.

[robertwb]

Adds an assertion to source_test_utils for testing reentrancy.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bdcb04cb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bdcb04cb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bdcb04cb

Branch: refs/heads/python-sdk
Commit: bdcb04cb9146d035339f02559127a810166721ab
Parents: 2ab8d62
Author: Chamikara Jayalath 
Authored: Sat Oct 15 17:49:13 2016 -0700
Committer: Robert Bradshaw 
Committed: Tue Oct 18 12:09:19 2016 -0700

--
 sdks/python/apache_beam/io/avroio_test.py   | 30 ---
 sdks/python/apache_beam/io/iobase.py|  5 +-
 sdks/python/apache_beam/io/source_test_utils.py | 55 
 sdks/python/apache_beam/io/textio.py| 29 +--
 sdks/python/apache_beam/io/textio_test.py   | 34 +++-
 5 files changed, 99 insertions(+), 54 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/avroio_test.py
--
diff --git a/sdks/python/apache_beam/io/avroio_test.py 
b/sdks/python/apache_beam/io/avroio_test.py
index eb2c81c..f72c3f3 100644
--- a/sdks/python/apache_beam/io/avroio_test.py
+++ b/sdks/python/apache_beam/io/avroio_test.py
@@ -110,7 +110,7 @@ class TestAvro(unittest.TestCase):
 return file_name_prefix + os.path.sep + 'mytemp*'
 
   def _run_avro_test(self, pattern, desired_bundle_size, perform_splitting,
- expected_result, test_reentrancy=False):
+ expected_result):
 source = AvroSource(pattern)
 
 read_records = []
@@ -128,23 +128,9 @@ class TestAvro(unittest.TestCase):
   (split.source, split.start_position, split.stop_position)
   for split in splits
   ]
-  if test_reentrancy:
-for source_info in sources_info:
-  reader_iter = source_info[0].read(source_info[0].get_range_tracker(
-  source_info[1], source_info[2]))
-  try:
-next(reader_iter)
-  except StopIteration:
-# Ignoring empty bundle
-pass
-
   source_test_utils.assertSourcesEqualReferenceSource((source, None, None),
   sources_info)
 else:
-  if test_reentrancy:
-reader_iter = source.read(source.get_range_tracker(None, None))
-next(reader_iter)
-
   read_records = source_test_utils.readFromSource(source, None, None)
   self.assertItemsEqual(expected_result, read_records)
 
@@ -160,15 +146,17 @@ class TestAvro(unittest.TestCase):
 
   def test_read_reentrant_without_splitting(self):
 file_name = self._write_data()
-expected_result = self.RECORDS
-self._run_avro_test(file_name, None, False, expected_result,
-test_reentrancy=True)
+source = AvroSource(file_name)
+source_test_utils.assertReentrantReadsSucceed((source, None, None))
 
   def test_read_reantrant_with_splitting(self):
 file_name = self._write_data()
-expected_result = self.RECORDS
-self._run_avro_test(file_name, 100, True, expected_result,
-test_reentrancy=True)
+source = AvroSource(file_name)
+splits = [
+split for split in source.split(desired_bundle_size=10)]
+assert len(splits) == 1
+source_test_utils.assertReentrantReadsSucceed(
+(splits[0].source, splits[0].start_position, splits[0].stop_position))
 
   def test_read_without_splitting_multiple_blocks(self):
 file_name = self._write_data(count=12000)

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/iobase.py
--
diff --git a/sdks/python/apache_beam/io/iobase.py 
b/sdks/python/apache_beam/io/iobase.py
index edd3524..9701964 100644
--- a/sdks/python/apache_beam/io/iobase.py
+++ b/sdks/python/apache_beam/io/iobase.py
@@ -94,11 +94,10 @@ class BoundedSource(object):
 
   **Mutability**
 
-  A ``BoundedSource`` object should be fully mutated before being submitted
-  for reading. A ``BoundedSource`` object should not be mutated while
+  A ``BoundedSource`` object should not be mutated while
   its methods (for example, ``read()``) are being invoked by a runner. Runner
   implementations may invoke methods of ``BoundedSource`` objects through
-  multi-threaded and/or re-entrant execution modes.
+  multi-threaded and/or reentrant execution modes.
   """
 
   def estimate_size(self):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/bdcb04cb/sdks/python/apache_beam/io/source_test_utils.py