[3/3] beam-site git commit: This closes #150

[altay]

This closes #150


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/4282f6bc
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/4282f6bc
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/4282f6bc

Branch: refs/heads/asf-site
Commit: 4282f6bc54a0ccc6bb7a4bac65547e429d78538e
Parents: a1e2a39 aa9b7fe
Author: Ahmet Altay 
Authored: Mon Feb 13 12:11:36 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 13 12:11:36 2017 -0800

--
 content/documentation/programming-guide/index.html | 9 -
 content/get-started/wordcount-example/index.html   | 4 ++--
 src/documentation/programming-guide.md | 9 -
 3 files changed, 10 insertions(+), 12 deletions(-)
--

[1/3] beam-site git commit: Update ParDo documentation for Python

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site a1e2a39f6 -> 4282f6bc5


Update ParDo documentation for Python


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8bc63920
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8bc63920
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8bc63920

Branch: refs/heads/asf-site
Commit: 8bc639206de6054a15da9601d56a7bf82c4f076f
Parents: a1e2a39
Author: Hadar Hod 
Authored: Thu Feb 9 10:37:07 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 13 12:11:01 2017 -0800

--
 src/documentation/programming-guide.md | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/8bc63920/src/documentation/programming-guide.md
--
diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 587c9f9..641ad2d 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -338,7 +338,7 @@ static class ComputeWordLengthFn extends DoFn { ... }
 Inside your `DoFn` subclass, you'll write a method annotated with 
`@ProcessElement` where you provide the actual processing logic. You don't need 
to manually extract the elements from the input collection; the Beam SDKs 
handle that for you. Your `@ProcessElement` method should accept an object of 
type `ProcessContext`. The `ProcessContext` object gives you access to an input 
element and a method for emitting an output element:
 
 {:.language-py}
-Inside your `DoFn` subclass, you'll write a method `process` where you provide 
the actual processing logic. You don't need to manually extract the elements 
from the input collection; the Beam SDKs handle that for you. Your `process` 
method should accept an object of type `context`. The `context` object gives 
you access to an input element and output is emitted by using `yield` or 
`return` statement inside `process` method.
+Inside your `DoFn` subclass, you'll write a method `process` where you provide 
the actual processing logic. You don't need to manually extract the elements 
from the input collection; the Beam SDKs handle that for you. Your `process` 
method should accept an object of type `element`. This is the input element and 
output is emitted by using `yield` or `return` statement inside `process` 
method.
 
 ```java
 static class ComputeWordLengthFn extends DoFn {
@@ -429,11 +429,11 @@ words = ...
 
  Using GroupByKey
 
-`GroupByKey` is a Beam transform for processing collections of key/value 
pairs. It's a parallel reduction operation, analagous to the Shuffle phase of a 
Map/Shuffle/Reduce-style algorithm. The input to `GroupByKey` is a collection 
of key/value pairs that represents a *multimap*, where the collection contains 
multiple pairs that have the same key, but different values. Given such a 
collection, you use `GroupByKey` to collect all of the values associated with 
each unique key.
+`GroupByKey` is a Beam transform for processing collections of key/value 
pairs. It's a parallel reduction operation, analogous to the Shuffle phase of a 
Map/Shuffle/Reduce-style algorithm. The input to `GroupByKey` is a collection 
of key/value pairs that represents a *multimap*, where the collection contains 
multiple pairs that have the same key, but different values. Given such a 
collection, you use `GroupByKey` to collect all of the values associated with 
each unique key.
 
 `GroupByKey` is a good way to aggregate data that has something in common. For 
example, if you have a collection that stores records of customer orders, you 
might want to group together all the orders from the same postal code (wherein 
the "key" of the key/value pair is the postal code field, and the "value" is 
the remainder of the record).
 
-Let's examine the mechanics of `GroupByKey` with a simple xample case, where 
our data set consists of words from a text file and the line number on which 
they appear. We want to group together all the line numbers (values) that share 
the same word (key), letting us see all the places in the text where a 
particular word appears.
+Let's examine the mechanics of `GroupByKey` with a simple example case, where 
our data set consists of words from a text file and the line number on which 
they appear. We want to group together all the line numbers (values) that share 
the same word (key), letting us see all the places in the text where a 
particular word appears.
 
 Our input is a `PCollection` of key/value pairs where each word is a key, and 
the value is a line number in the file where the word appears. Here's a list of 
the key/value pairs in the 

[GitHub] beam-site pull request #150: Update ParDo documentation for Python

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/150


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/aa9b7fea
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/aa9b7fea
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/aa9b7fea

Branch: refs/heads/asf-site
Commit: aa9b7fea05f5e1177ff40f7b3977cdfb9ec0dd19
Parents: 8bc6392
Author: Ahmet Altay 
Authored: Mon Feb 13 12:11:35 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 13 12:11:35 2017 -0800

--
 content/documentation/programming-guide/index.html | 9 -
 content/get-started/wordcount-example/index.html   | 4 ++--
 2 files changed, 6 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/aa9b7fea/content/documentation/programming-guide/index.html
--
diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index f02fd40..0aa0575 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -515,7 +515,7 @@
 
 Inside your DoFn subclass, you’ll write a method 
annotated with @ProcessElement where you 
provide the actual processing logic. You don’t need to manually extract the 
elements from the input collection; the Beam SDKs handle that for you. Your 
@ProcessElement method should accept an 
object of type ProcessContext. The ProcessContext object gives you access to an 
input element and a method for emitting an output element:
 
-Inside your DoFn 
subclass, you’ll write a method process where you provide the actual 
processing logic. You don’t need to manually extract the elements from the 
input collection; the Beam SDKs handle that for you. Your process method should accept an object of type 
context. The context object gives you access to an input 
element and output is emitted by using yield or return statement inside process method.
+Inside your DoFn 
subclass, you’ll write a method process where you provide the actual 
processing logic. You don’t need to manually extract the elements from the 
input collection; the Beam SDKs handle that for you. Your process method should accept an object of type 
element. This is the input element and 
output is emitted by using yield or 
return statement inside process method.
 
 static class ComputeWordLengthFn extends DoFnString, 
Integer {
   @ProcessElement
@@ -610,11 +610,11 @@
 
 Using GroupByKey
 
-GroupByKey is a Beam transform for 
processing collections of key/value pairs. It’s a parallel reduction 
operation, analagous to the Shuffle phase of a Map/Shuffle/Reduce-style 
algorithm. The input to GroupByKey is a 
collection of key/value pairs that represents a multimap, where the 
collection contains multiple pairs that have the same key, but different 
values. Given such a collection, you use GroupByKey to collect all of the values 
associated with each unique key.
+GroupByKey is a Beam transform for 
processing collections of key/value pairs. It’s a parallel reduction 
operation, analogous to the Shuffle phase of a Map/Shuffle/Reduce-style 
algorithm. The input to GroupByKey is a 
collection of key/value pairs that represents a multimap, where the 
collection contains multiple pairs that have the same key, but different 
values. Given such a collection, you use GroupByKey to collect all of the values 
associated with each unique key.
 
 GroupByKey is a good way to 
aggregate data that has something in common. For example, if you have a 
collection that stores records of customer orders, you might want to group 
together all the orders from the same postal code (wherein the “key” of the 
key/value pair is the postal code field, and the “value” is the remainder 
of the record).
 
-Let’s examine the mechanics of GroupByKey with a simple xample case, where 
our data set consists of words from a text file and the line number on which 
they appear. We want to group together all the line numbers (values) that share 
the same word (key), letting us see all the places in the text where a 
particular word appears.
+Let’s examine the mechanics of GroupByKey with a simple example case, where 
our data set consists of words from a text file and the line number on which 
they appear. We want to group together all the line numbers (values) that share 
the same word (key), letting us see all the places in the text where a 
particular word appears.
 
 Our input is a PCollection of 
key/value pairs where each word is a key, and the value is a line number in the 
file where the word appears. Here’s a list of the key/value pairs in the 
input collection:
 
@@ -1046,7 +1046,7 @@ tree, [2]
 
 
 # We can also pass side inputs to a 

[jira] [Commented] (BEAM-1346) Drop Late Data in ReduceFnRunner

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864437#comment-15864437
 ] 

Kenneth Knowles commented on BEAM-1346:
---

I agree with your paranoia. This is related to BEAM-696. I think 
{{PushbackSideInputRunner}} is technically OK because it pushes all the 
complexity to the runner. The runner can provide a 
{{ReadyCheckingSideInputReader}} that understands the merging, and the runner 
decides when to wake up the processing and feed the pushed-back elements, so it 
can alter its timers, etc, according to merging. And if this is all too 
complex, maybe the runner can [disable any troublesome 
optimizations|https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java#L811]
 for now.

> Drop Late Data in ReduceFnRunner
> 
>
> Key: BEAM-1346
> URL: https://issues.apache.org/jira/browse/BEAM-1346
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>
> I think these two commits recently broke late-data dropping for the Flink 
> Runner (and maybe for other runners as well):
> - https://github.com/apache/beam/commit/2b26ec8
> - https://github.com/apache/beam/commit/8989473
> It boils down to the {{LateDataDroppingDoFnRunner}} not being used anymore  
> because {{DoFnRunners.lateDataDroppingRunner()}} is not called anymore when a 
> {{DoFn}} is a {{ReduceFnExecutor}} (because that interface was removed).
> Maybe we should think about dropping late data in another place, my 
> suggestion is {{ReduceFnRunner}} but that's open for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1280) Remove label-first variants of PTransform constructors (use >> instead)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864436#comment-15864436
 ] 

ASF GitHub Bot commented on BEAM-1280:
--

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1997


> Remove label-first variants of PTransform constructors (use >> instead)
> ---
>
> Key: BEAM-1280
> URL: https://issues.apache.org/jira/browse/BEAM-1280
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>  Labels: sdk-consistency
>
> (Related in Java SDK: https://issues.apache.org/jira/browse/BEAM-370)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1997: [BEAM-1280] Remove passing label from PTransform in...

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1997


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Upgrade google-api-services-dataflow to v1b3-rev186-1.22.0

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 2c0cffaf7 -> a628ce353


Upgrade google-api-services-dataflow to v1b3-rev186-1.22.0


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cad43544
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cad43544
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cad43544

Branch: refs/heads/master
Commit: cad4354401ac8e43b216adc900b01429f659a5ec
Parents: 2c0cffa
Author: Eric Roshan-Eisner 
Authored: Mon Feb 13 13:38:37 2017 -0800
Committer: Eric Roshan-Eisner 
Committed: Mon Feb 13 13:38:37 2017 -0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cad43544/pom.xml
--
diff --git a/pom.xml b/pom.xml
index f4e458e..be75659 100644
--- a/pom.xml
+++ b/pom.xml
@@ -108,7 +108,7 @@
 v1-rev6-1.22.0
 0.1.0
 v2-rev8-1.22.0
-v1b3-rev43-1.22.0
+v1b3-rev186-1.22.0
 0.5.160222
 1.4.0
 1.3.0

[GitHub] beam pull request #1999: Upgrade google-api-services-dataflow

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1999


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1999

[dhalperi]

This closes #1999


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a628ce35
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a628ce35
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a628ce35

Branch: refs/heads/master
Commit: a628ce35388c36dac6f96f8069542d9068d31035
Parents: 2c0cffa cad4354
Author: Dan Halperin 
Authored: Mon Feb 13 14:49:05 2017 -0800
Committer: Dan Halperin 
Committed: Mon Feb 13 14:49:05 2017 -0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #2000: [BEAM-115] Unify Fn API and Runner API FunctionSpec

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2000

[BEAM-115] Unify Fn API and Runner API FunctionSpec

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Do not review yet - utilizing Jenkins

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam fn-api-functionspec

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2000.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2000


commit 90ed46256915d726b537454b1239cb70767b2e2d
Author: Kenneth Knowles 
Date:   2017-02-13T16:38:40Z

Unify Fn API and Runner API FunctionSpec




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864604#comment-15864604
 ] 

Davor Bonaci commented on BEAM-1461:


I think what [~tgroh] did make sense. [~tgroh], can you please expand?

> duplication with StartBundle and prepareForProcessing in DoFn
> -
>
> Key: BEAM-1461
> URL: https://issues.apache.org/jira/browse/BEAM-1461
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> There're one annotation `StartBundle`, and one public function 
> `prepareForProcessing` in DoFn, which are called both before 
> `ProcessElement`. It's confused which one should be implemented in a subclass.
> The call sequence seems as:
> prepareForProcessing -> StartBundle -> processElement



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/5] beam-site git commit: Add klk as a blog author

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 4282f6bc5 -> 00e375003


Add klk as a blog author


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/31387225
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/31387225
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/31387225

Branch: refs/heads/asf-site
Commit: 3138722512cfe60ea4dd231389e7736eb8d420f3
Parents: 4282f6b
Author: Kenneth Knowles 
Authored: Thu Feb 9 19:37:56 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 13 13:29:08 2017 -0800

--
 src/_data/authors.yml | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/31387225/src/_data/authors.yml
--
diff --git a/src/_data/authors.yml b/src/_data/authors.yml
index 147dd7b..e4aa332 100644
--- a/src/_data/authors.yml
+++ b/src/_data/authors.yml
@@ -36,3 +36,7 @@ thw:
 name: Thomas Weise
 email: t...@apache.org
 twitter: thweise
+klk:
+name: Kenneth Knowles
+email: k...@apache.org
+twitter: KennKnowles

[GitHub] beam-site pull request #152: Add blog post "Stateful Processing with Apache ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/152


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/5] beam-site git commit: Regenerate website

[davor]

http://git-wip-us.apache.org/repos/asf/beam-site/blob/2dd05932/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index f94ee48..10bccbd 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -9,6 +9,580 @@
 Jekyll v3.2.0
 
   
+Stateful processing with Apache Beam
+pBeam lets you process unbounded, out-of-order, 
global-scale data with portable
+high-level pipelines. Stateful processing is a new feature of the Beam model
+that expands the capabilities of Beam, unlocking new use cases and new
+efficiencies. In this post, I will guide you through stateful processing in
+Beam: how it works, how it fits in with the other features of the Beam model,
+what you might use it for, and what it looks like in code./p
+
+!--more--
+
+blockquote
+  pstrongWarning: new features ahead!/strong: This is 
a very new aspect of the Beam
+model. Runners are still adding support. You can try it out today on multiple
+runners, but do check the a 
href=/documentation/runners/capability-matrix/runner capability
+matrix/a for
+the current status in each runner./p
+/blockquote
+
+pFirst, a quick recap: In Beam, a big data processing 
empipeline/em is a directed,
+acyclic graph of parallel operations called emcode 
class=highlighter-rougePTransforms/code/em 
processing data
+from emcode 
class=highlighter-rougePCollections/code/em 
I’ll expand on that by walking through this illustration:/p
+
+pimg class=center-block 
src=/images/blog/stateful-processing/pipeline.png alt=A Beam 
Pipeline - PTransforms are boxes - PCollections are arrows 
width=300 //p
+
+pThe boxes are code 
class=highlighter-rougePTransforms/code and the edges 
represent the data in code 
class=highlighter-rougePCollections/code
+flowing from one code 
class=highlighter-rougePTransform/code to the next. A 
code class=highlighter-rougePCollection/code may be 
embounded/em (which
+means it is finite and you know it) or emunbounded/em (which 
means you don’t know if
+it is finite or not - basically, it is like an incoming stream of data that may
+or may not ever terminate). The cylinders are the data sources and sinks at the
+edges of your pipeline, such as bounded collections of log files or unbounded
+data streaming over a Kafka topic. This blog post isn’t about sources or 
sinks,
+but about what happens in between - your data processing./p
+
+pThere are two main building blocks for processing your data in Beam: 
emcode 
class=highlighter-rougeParDo/code/em,
+for performing an operation in parallel across all elements, and 
emcode 
class=highlighter-rougeGroupByKey/code/em
+(and the closely related code 
class=highlighter-rougeCombinePerKey/code that I will 
talk about quite soon)
+for aggregating elements to which you have assigned the same key. In the
+picture below (featured in many of our presentations) the color indicates the
+key of the element. Thus the code 
class=highlighter-rougeGroupByKey/code/code 
class=highlighter-rougeCombinePerKey/code transform 
gathers all the
+green squares to produce a single output element./p
+
+pimg class=center-block 
src=/images/blog/stateful-processing/pardo-and-gbk.png 
alt=ParDo and GroupByKey/CombinePerKey:  Elementwise versus 
aggregating computations width=400 //p
+
+pBut not all use cases are easily expressed as pipelines of simple 
code class=highlighter-rougeParDo/code/code 
class=highlighter-rougeMap/code and
+code 
class=highlighter-rougeGroupByKey/code/code 
class=highlighter-rougeCombinePerKey/code transforms. 
The topic of this blog post is a new
+extension to the Beam programming model: strongper-element operation 
augmented with
+mutable state/strong./p
+
+pimg class=center-block 
src=/images/blog/stateful-processing/stateful-pardo.png 
alt=Stateful ParDo - sequential per-key processing with persistent 
state width=300 //p
+
+pIn the illustration above, ParDo now has a bit of durable, consistent 
state on
+the side, which can be read and written during the processing of each element.
+The state is partitioned by key, so it is drawn as having disjoint sections for
+each color. It is also partitioned per window, but I thought plaid 
+img src=/images/blog/stateful-processing/plaid.png alt=A 
plaid storage cylinder width=20 / 
+would be a bit much  :-). I’ll talk about
+why state is partitioned this way a bit later, via my first example./p
+
+pFor the rest of this post, I will describe this new feature of Beam 
in detail -
+how it works at a high level, how it differs from existing features, how to
+make sure it is still massively scalable. After that introduction at the model
+level, I’ll walk through a simple example of how you use it in the Beam Java
+SDK./p
+
+h2 id=how-does-stateful-processing-in-beam-workHow does 
stateful processing in Beam work?/h2
+
+pThe processing logic of your code 
class=highlighter-rougeParDo/code transform is 
expressed through the code 

[jira] [Commented] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864609#comment-15864609
 ] 

Davor Bonaci commented on BEAM-1457:


I'd keep this issue open for a little bit. I'd probably not make a move on it 
just yet, and see how things progress build-wise from this point on. Now, with 
Python in, build times are off quite a bit, and impact therefore is a bit 
muted. Let's sit on it and reevaluate shortly.

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Assignee: Davor Bonaci
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2283

[Apache Jenkins Server]

See 

[2/5] beam-site git commit: Add blog post "Stateful Processing with Apache Beam"

[davor]

Add blog post "Stateful Processing with Apache Beam"


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/4c439913
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/4c439913
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/4c439913

Branch: refs/heads/asf-site
Commit: 4c4399132b1987884c03edaf2570614c00f651cb
Parents: 3138722
Author: Kenneth Knowles 
Authored: Thu Feb 9 19:38:13 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 13 13:29:11 2017 -0800

--
 src/_posts/2017-02-13-stateful-processing.md| 550 +++
 .../blog/stateful-processing/assign-indices.png | Bin 0 -> 29308 bytes
 .../blog/stateful-processing/combinefn.png  | Bin 0 -> 18138 bytes
 .../stateful-processing/combiner-lifting.png| Bin 0 -> 37974 bytes
 .../blog/stateful-processing/pardo-and-gbk.png  | Bin 0 -> 27227 bytes
 .../blog/stateful-processing/pipeline.png   | Bin 0 -> 14308 bytes
 src/images/blog/stateful-processing/plaid.png   | Bin 0 -> 46216 bytes
 .../blog/stateful-processing/stateful-dofn.png  | Bin 0 -> 9 bytes
 .../blog/stateful-processing/stateful-pardo.png | Bin 0 -> 18035 bytes
 9 files changed, 550 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/4c439913/src/_posts/2017-02-13-stateful-processing.md
--
diff --git a/src/_posts/2017-02-13-stateful-processing.md 
b/src/_posts/2017-02-13-stateful-processing.md
new file mode 100644
index 000..28aee12
--- /dev/null
+++ b/src/_posts/2017-02-13-stateful-processing.md
@@ -0,0 +1,550 @@
+---
+layout: post
+title:  "Stateful processing with Apache Beam"
+date:   2017-02-13 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+  - klk
+---
+
+Beam lets you process unbounded, out-of-order, global-scale data with portable
+high-level pipelines. Stateful processing is a new feature of the Beam model
+that expands the capabilities of Beam, unlocking new use cases and new
+efficiencies. In this post, I will guide you through stateful processing in
+Beam: how it works, how it fits in with the other features of the Beam model,
+what you might use it for, and what it looks like in code.
+
+
+
+> **Warning: new features ahead!**: This is a very new aspect of the Beam
+> model. Runners are still adding support. You can try it out today on multiple
+> runners, but do check the [runner capability
+> matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) for
+> the current status in each runner.
+
+First, a quick recap: In Beam, a big data processing _pipeline_ is a directed,
+acyclic graph of parallel operations called _`PTransforms`_ processing data
+from _`PCollections`_ I'll expand on that by walking through this illustration:
+
+
+
+The boxes are `PTransforms` and the edges represent the data in `PCollections`
+flowing from one `PTransform` to the next. A `PCollection` may be _bounded_ 
(which
+means it is finite and you know it) or _unbounded_ (which means you don't know 
if
+it is finite or not - basically, it is like an incoming stream of data that may
+or may not ever terminate). The cylinders are the data sources and sinks at the
+edges of your pipeline, such as bounded collections of log files or unbounded
+data streaming over a Kafka topic. This blog post isn't about sources or sinks,
+but about what happens in between - your data processing.
+
+There are two main building blocks for processing your data in Beam: _`ParDo`_,
+for performing an operation in parallel across all elements, and _`GroupByKey`_
+(and the closely related `CombinePerKey` that I will talk about quite soon)
+for aggregating elements to which you have assigned the same key. In the
+picture below (featured in many of our presentations) the color indicates the
+key of the element. Thus the `GroupByKey`/`CombinePerKey` transform gathers 
all the
+green squares to produce a single output element.
+
+
+
+But not all use cases are easily expressed as pipelines of simple 
`ParDo`/`Map` and
+`GroupByKey`/`CombinePerKey` transforms. The topic of this blog post is a new
+extension to the Beam programming model: **per-element operation augmented with
+mutable state**.
+
+
+
+In the illustration above, ParDo now has a bit of durable, consistent state on
+the side, which can be read and written during the processing of each element.
+The state is partitioned by key, so it is drawn as having disjoint sections for
+each color. It is also partitioned per window, but I thought plaid 
+ 
+would be a bit much  :-). I'll talk about
+why state is partitioned this way a bit later, via my first example.
+
+For the rest of this post, I will describe this new feature of Beam in detail -
+how it works at a high level, how 

[4/5] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/2dd05932
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/2dd05932
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/2dd05932

Branch: refs/heads/asf-site
Commit: 2dd059328e13788c7e58a6be5e2098ea5af82336
Parents: 4c43991
Author: Davor Bonaci 
Authored: Mon Feb 13 13:31:19 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 13 13:31:19 2017 -0800

--
 .../blog/2017/02/13/stateful-processing.html| 751 +++
 content/blog/index.html |  21 +
 content/feed.xml| 593 ++-
 .../blog/stateful-processing/assign-indices.png | Bin 0 -> 29308 bytes
 .../blog/stateful-processing/combinefn.png  | Bin 0 -> 18138 bytes
 .../stateful-processing/combiner-lifting.png| Bin 0 -> 37974 bytes
 .../blog/stateful-processing/pardo-and-gbk.png  | Bin 0 -> 27227 bytes
 .../blog/stateful-processing/pipeline.png   | Bin 0 -> 14308 bytes
 .../images/blog/stateful-processing/plaid.png   | Bin 0 -> 46216 bytes
 .../blog/stateful-processing/stateful-dofn.png  | Bin 0 -> 9 bytes
 .../blog/stateful-processing/stateful-pardo.png | Bin 0 -> 18035 bytes
 content/index.html  |   4 +-
 12 files changed, 1348 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/2dd05932/content/blog/2017/02/13/stateful-processing.html
--
diff --git a/content/blog/2017/02/13/stateful-processing.html 
b/content/blog/2017/02/13/stateful-processing.html
new file mode 100644
index 000..dcf077a
--- /dev/null
+++ b/content/blog/2017/02/13/stateful-processing.html
@@ -0,0 +1,751 @@
+
+
+
+  
+  
+  
+  
+
+  Stateful processing with Apache Beam
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  https://beam.apache.org/blog/2017/02/13/stateful-processing.html; 
data-proofer-ignore>
+  https://beam.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+ Get Started 
+ 
+ Beam 
Overview
+Quickstart - Java
+Quickstart - Python
+ 
+ Example Walkthroughs
+ WordCount
+ Mobile Gaming
+  
+  Resources
+  Downloads
+  Support
+ 
+   
+
+ Documentation 
+ 
+ Using the 
Documentation
+ 
+ Beam Concepts
+ Programming Guide
+ Additional 
Resources
+ 
+  Pipeline Fundamentals
+  Design Your 
Pipeline
+  Create Your 
Pipeline
+  Test 
Your Pipeline
+  
+ SDKs
+ Java 
SDK
+ Java SDK API Reference 
+
+Python SDK
+ 
+ Runners
+ Capability Matrix
+ Direct 
Runner
+ Apache 
Apex Runner
+ Apache 
Flink Runner
+ Apache 
Spark Runner
+ Cloud 
Dataflow Runner
+ 
+   
+
+ Contribute 
+ 
+ Get Started 
Contributing
+
+Guides
+ Contribution Guide
+Testing Guide
+Release Guide
+PTransform Style 
Guide
+
+Technical References
+Design Principles
+ Ongoing 
Projects
+Source 
Repository  
+
+ Promotion
+Presentation 
Materials
+Logos and Design
+
+Maturity Model
+Team
+ 
+   
+
+Blog
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png; 

[5/5] beam-site git commit: This closes #152

[davor]

This closes #152


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/00e37500
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/00e37500
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/00e37500

Branch: refs/heads/asf-site
Commit: 00e375003c3c33e1e9234ea506a6c746967f5988
Parents: 4282f6b 2dd0593
Author: Davor Bonaci 
Authored: Mon Feb 13 13:31:20 2017 -0800
Committer: Davor Bonaci 
Committed: Mon Feb 13 13:31:20 2017 -0800

--
 .../blog/2017/02/13/stateful-processing.html| 751 +++
 content/blog/index.html |  21 +
 content/feed.xml| 593 ++-
 .../blog/stateful-processing/assign-indices.png | Bin 0 -> 29308 bytes
 .../blog/stateful-processing/combinefn.png  | Bin 0 -> 18138 bytes
 .../stateful-processing/combiner-lifting.png| Bin 0 -> 37974 bytes
 .../blog/stateful-processing/pardo-and-gbk.png  | Bin 0 -> 27227 bytes
 .../blog/stateful-processing/pipeline.png   | Bin 0 -> 14308 bytes
 .../images/blog/stateful-processing/plaid.png   | Bin 0 -> 46216 bytes
 .../blog/stateful-processing/stateful-dofn.png  | Bin 0 -> 9 bytes
 .../blog/stateful-processing/stateful-pardo.png | Bin 0 -> 18035 bytes
 content/index.html  |   4 +-
 src/_data/authors.yml   |   4 +
 src/_posts/2017-02-13-stateful-processing.md| 550 ++
 .../blog/stateful-processing/assign-indices.png | Bin 0 -> 29308 bytes
 .../blog/stateful-processing/combinefn.png  | Bin 0 -> 18138 bytes
 .../stateful-processing/combiner-lifting.png| Bin 0 -> 37974 bytes
 .../blog/stateful-processing/pardo-and-gbk.png  | Bin 0 -> 27227 bytes
 .../blog/stateful-processing/pipeline.png   | Bin 0 -> 14308 bytes
 src/images/blog/stateful-processing/plaid.png   | Bin 0 -> 46216 bytes
 .../blog/stateful-processing/stateful-dofn.png  | Bin 0 -> 9 bytes
 .../blog/stateful-processing/stateful-pardo.png | Bin 0 -> 18035 bytes
 22 files changed, 1902 insertions(+), 21 deletions(-)
--

[GitHub] beam pull request #1999: Upgrade google-api-services-dataflow

[edre]

GitHub user edre opened a pull request:

https://github.com/apache/beam/pull/1999

Upgrade google-api-services-dataflow

to v1b3-rev186-1.22.0

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/edre/beam master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1999.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1999


commit cad4354401ac8e43b216adc900b01429f659a5ec
Author: Eric Roshan-Eisner 
Date:   2017-02-13T21:38:37Z

Upgrade google-api-services-dataflow to v1b3-rev186-1.22.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1464) Make sure all tests use the TestPipeline class

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1464.
-
Resolution: Fixed

> Make sure all tests use the TestPipeline class
> --
>
> Key: BEAM-1464
> URL: https://issues.apache.org/jira/browse/BEAM-1464
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> Some tests still specify using the DirectRunner. They should all use the 
> TestRunner in those cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1473) Remove unused windmill proto files from python sdk

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-1473:
---

 Summary: Remove unused windmill proto files from python sdk
 Key: BEAM-1473
 URL: https://issues.apache.org/jira/browse/BEAM-1473
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Affects Versions: Not applicable
Reporter: Sourabh Bajaj
Assignee: Sourabh Bajaj
Priority: Minor
 Fix For: Not applicable






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1473) Remove unused windmill proto files from python sdk

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj updated BEAM-1473:

Description: 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/internal/windmill_pb2.py
 

There are two unused windmill files in beam that should be cleaned.

> Remove unused windmill proto files from python sdk
> --
>
> Key: BEAM-1473
> URL: https://issues.apache.org/jira/browse/BEAM-1473
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/internal/windmill_pb2.py
>  
> There are two unused windmill files in beam that should be cleaned.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1473) Remove unused windmill proto files from python sdk

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864831#comment-15864831
 ] 

ASF GitHub Bot commented on BEAM-1473:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2001

[BEAM-1473] Remove unused windmill proto from python sdk

R: @charlesccychen PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1473-remove-windmill-proto

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2001






> Remove unused windmill proto files from python sdk
> --
>
> Key: BEAM-1473
> URL: https://issues.apache.org/jira/browse/BEAM-1473
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/internal/windmill_pb2.py
>  
> There are two unused windmill files in beam that should be cleaned.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Add cross-sdk tests for IterableCoder

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master b67bd111e -> 8e0573ba5


Add cross-sdk tests for IterableCoder


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cab5e634
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cab5e634
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cab5e634

Branch: refs/heads/master
Commit: cab5e6347f3fcece2cac4819e268488d7ce66830
Parents: b67bd11
Author: Vikas Kedigehalli 
Authored: Mon Feb 13 10:23:28 2017 -0800
Committer: Vikas Kedigehalli 
Committed: Mon Feb 13 15:35:19 2017 -0800

--
 .../org/apache/beam/fn/v1/standard_coders.yaml  | 20 
 .../apache/beam/sdk/coders/CommonCoderTest.java | 12 
 .../apache_beam/coders/standard_coders_test.py  |  2 ++
 .../apache_beam/tests/data/standard_coders.yaml | 20 
 4 files changed, 54 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/cab5e634/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
--
diff --git 
a/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
 
b/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
index 948ac6b..58a2a90 100644
--- 
a/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
+++ 
b/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
@@ -106,3 +106,23 @@ examples:
   
"\u0080\u\u0001\u0053\u0034\u00ec\u0074\u00e8\u0080\u0090\u00fb\u00d3\u0009"
 : {end: 1456881825000, span: 259200}
   "\u007f\u00df\u003b\u0064\u005a\u001c\u00ad\u0076\u00ed\u0002" : {end: 
-9223372036854410, span: 365}
   "\u0080\u0020\u00c4\u009b\u00a5\u00e3\u0053\u00f7\u" : {end: 
9223372036854775, span: 0}
+
+---
+
+coder:
+  urn: "urn:beam:coders:stream:0.1"
+  components: [{urn: "urn:beam:coders:varint:0.1"}]
+examples:
+  "\0\0\0\u0001\0": [0]
+  "\0\0\0\u0004\u0001\n\u00c8\u0001\u00e8\u0007": [1, 10, 200, 1000]
+  "\0\0\0\0": []
+
+---
+
+coder:
+  urn: "urn:beam:coders:stream:0.1"
+  components: [{urn: "urn:beam:coders:bytes:0.1"}]
+examples:
+  "\0\0\0\u0001\u0003abc": ["abc"]
+  "\0\0\0\u0002\u0004ab\0c\u0004de\0f": ["ab\0c", "de\0f"]
+  "\0\0\0\0": []
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/beam/blob/cab5e634/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CommonCoderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CommonCoderTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CommonCoderTest.java
index ad5d9c3..7eafbe2 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CommonCoderTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/CommonCoderTest.java
@@ -70,6 +70,7 @@ public class CommonCoderTest {
   .put("urn:beam:coders:kv:0.1", KvCoder.class)
   .put("urn:beam:coders:varint:0.1", VarLongCoder.class)
   .put("urn:beam:coders:intervalwindow:0.1", IntervalWindowCoder.class)
+  .put("urn:beam:coders:stream:0.1", IterableCoder.class)
   .build();
 
   @AutoValue
@@ -198,6 +199,15 @@ public class CommonCoderTest {
 Duration span = Duration.millis(((Number) 
kvMap.get("span")).longValue());
 return new IntervalWindow(end.minus(span), span);
   }
+  case "urn:beam:coders:stream:0.1":
+Coder elementCoder = ((IterableCoder) coder).getElemCoder();
+List elements = (List) value;
+List convertedElements = new LinkedList<>();
+for (Object element : elements) {
+  convertedElements.add(
+  convertValue(element, coderSpec.getComponents().get(0), 
elementCoder));
+}
+return convertedElements;
   default:
 throw new IllegalStateException("Unknown coder URN: " + 
coderSpec.getUrn());
 }
@@ -217,6 +227,8 @@ public class CommonCoderTest {
 return VarLongCoder.of();
   case "urn:beam:coders:intervalwindow:0.1":
 return IntervalWindowCoder.of();
+  case "urn:beam:coders:stream:0.1":
+return IterableCoder.of(components.get(0));
   default:
 throw new IllegalStateException("Unknown coder URN: " + 
coder.getUrn());
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/cab5e634/sdks/python/apache_beam/coders/standard_coders_test.py
--
diff --git a/sdks/python/apache_beam/coders/standard_coders_test.py 
b/sdks/python/apache_beam/coders/standard_coders_test.py
index e66ec7b..d4179eb 100644
--- a/sdks/python/apache_beam/coders/standard_coders_test.py
+++ 

[2/2] beam git commit: This closes #1996

[dhalperi]

This closes #1996


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8e0573ba
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8e0573ba
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8e0573ba

Branch: refs/heads/master
Commit: 8e0573ba53918b1c4403a6f48a8e983f12d53270
Parents: b67bd11 cab5e63
Author: Dan Halperin 
Authored: Mon Feb 13 17:39:22 2017 -0800
Committer: Dan Halperin 
Committed: Mon Feb 13 17:39:22 2017 -0800

--
 .../org/apache/beam/fn/v1/standard_coders.yaml  | 20 
 .../apache/beam/sdk/coders/CommonCoderTest.java | 12 
 .../apache_beam/coders/standard_coders_test.py  |  2 ++
 .../apache_beam/tests/data/standard_coders.yaml | 20 
 4 files changed, 54 insertions(+)
--

[GitHub] beam pull request #1996: [BEAM-1471]: Add cross-sdk tests for IterableCoder

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1996


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1310) Add running integration tests for JdbcIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864887#comment-15864887
 ] 

ASF GitHub Bot commented on BEAM-1310:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1841


> Add running integration tests for JdbcIO
> 
>
> Key: BEAM-1310
> URL: https://issues.apache.org/jira/browse/BEAM-1310
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> Jdbc IO could use some integration tests! We'd like to have them run against 
> a real list instance of postgres.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Upgrading to datatore 0.7.0 library

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 4018c835c -> f32cb3e7a


Upgrading to datatore 0.7.0 library


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d8201a93
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d8201a93
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d8201a93

Branch: refs/heads/master
Commit: d8201a93c7776dbd0e77b4f428fb14a9840dd18a
Parents: 4018c83
Author: Ahmet Altay 
Authored: Fri Feb 10 19:02:25 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 13 18:25:38 2017 -0800

--
 .../apache_beam/examples/cookbook/datastore_wordcount.py | 6 +++---
 sdks/python/apache_beam/examples/snippets/snippets.py| 4 ++--
 sdks/python/apache_beam/io/datastore/v1/datastoreio.py   | 2 +-
 sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py  | 4 ++--
 sdks/python/apache_beam/io/datastore/v1/fake_datastore.py| 4 ++--
 sdks/python/apache_beam/io/datastore/v1/helper.py| 6 +++---
 sdks/python/apache_beam/io/datastore/v1/helper_test.py   | 8 
 sdks/python/apache_beam/io/datastore/v1/query_splitter.py| 8 
 .../apache_beam/io/datastore/v1/query_splitter_test.py   | 6 +++---
 sdks/python/setup.py | 5 +++--
 10 files changed, 27 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d8201a93/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py 
b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
index 282afbf..5d3bef6 100644
--- a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
+++ b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
@@ -35,7 +35,7 @@ The following options must be provided to run this pipeline 
in read-only mode:
 --project YOUR_PROJECT_ID
 --kind YOUR_DATASTORE_KIND
 --output [YOUR_LOCAL_FILE *or* gs://YOUR_OUTPUT_PATH]
---read-only
+--read_only
 ``
 
 Read-write Mode: In this mode, this example reads words from an input file,
@@ -66,8 +66,8 @@ import logging
 import re
 import uuid
 
-from google.datastore.v1 import entity_pb2
-from google.datastore.v1 import query_pb2
+from google.cloud.proto.datastore.v1 import entity_pb2
+from google.cloud.proto.datastore.v1 import query_pb2
 from googledatastore import helper as datastore_helper, PropertyFilter
 
 import apache_beam as beam

http://git-wip-us.apache.org/repos/asf/beam/blob/d8201a93/sdks/python/apache_beam/examples/snippets/snippets.py
--
diff --git a/sdks/python/apache_beam/examples/snippets/snippets.py 
b/sdks/python/apache_beam/examples/snippets/snippets.py
index 9ba46cd..6f081df 100644
--- a/sdks/python/apache_beam/examples/snippets/snippets.py
+++ b/sdks/python/apache_beam/examples/snippets/snippets.py
@@ -862,8 +862,8 @@ def model_datastoreio():
   """Using a Read and Write transform to read/write to Cloud Datastore."""
 
   import uuid
-  from google.datastore.v1 import entity_pb2
-  from google.datastore.v1 import query_pb2
+  from google.cloud.proto.datastore.v1 import entity_pb2
+  from google.cloud.proto.datastore.v1 import query_pb2
   import googledatastore
   import apache_beam as beam
   from apache_beam.utils.pipeline_options import PipelineOptions

http://git-wip-us.apache.org/repos/asf/beam/blob/d8201a93/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
--
diff --git a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py 
b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
index 5f6663a..562d88e 100644
--- a/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/datastore/v1/datastoreio.py
@@ -19,7 +19,7 @@
 
 import logging
 
-from google.datastore.v1 import datastore_pb2
+from google.cloud.proto.datastore.v1 import datastore_pb2
 from googledatastore import helper as datastore_helper
 
 from apache_beam.io.datastore.v1 import helper

http://git-wip-us.apache.org/repos/asf/beam/blob/d8201a93/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
--
diff --git a/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py 
b/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
index cdaccb1..3bd4630 100644
--- a/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
+++ b/sdks/python/apache_beam/io/datastore/v1/datastoreio_test.py
@@ -17,8 +17,8 @@
 
 import unittest
 
-from google.datastore.v1 import datastore_pb2
-from google.datastore.v1 import query_pb2
+from 

[GitHub] beam pull request #1989: [BEAM-1468] Upgrading datatore dependency to 0.7.0 ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1989


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1468) Upgrade datastore dependency the 0.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864916#comment-15864916
 ] 

ASF GitHub Bot commented on BEAM-1468:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1989


> Upgrade datastore dependency the 0.7.0
> --
>
> Key: BEAM-1468
> URL: https://issues.apache.org/jira/browse/BEAM-1468
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864666#comment-15864666
 ] 

Thomas Groh commented on BEAM-1461:
---

Sure;

{{prepareForProcessing}} exists purely to support Aggregators - it sets up some 
finalization details within the base DoFn class. However, we plan on removing 
Aggregators, tracked in [BEAM-775]. Once that's done, we should remove the 
parts of the SDK that exist to support Aggregators, which includes 
{{prepareForProcessing}}. We should signal this now, and I believe making the 
method final and deprecated is the most effective way to signal that it's an 
implementation detail of the DoFn internals rather than the user-visible 
processing method.

Additionally, overriding {{prepareForProcessing}} could lead to a lack of 
precondition enforcement within DoFn, so it is not generally safe to override.

On an additional note, we probably would have missed the naming duplication in 
prepareForProcessing as well as the fact that it's actually an Aggregator 
method without this Jira, so thank you for posting it.

> duplication with StartBundle and prepareForProcessing in DoFn
> -
>
> Key: BEAM-1461
> URL: https://issues.apache.org/jira/browse/BEAM-1461
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> There're one annotation `StartBundle`, and one public function 
> `prepareForProcessing` in DoFn, which are called both before 
> `ProcessElement`. It's confused which one should be implemented in a subclass.
> The call sequence seems as:
> prepareForProcessing -> StartBundle -> processElement



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-59) IOChannelFactory rethinking/redesign

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-59?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864859#comment-15864859
 ] 

ASF GitHub Bot commented on BEAM-59:


GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/2002

[BEAM-59] Beam GcsFileSystem: port expand() from GcsUtil for glob matching.



Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam gcs-util-refactor-expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2002


commit 7cd3aed6ea9261f4624c9a387f5379ed99de619a
Author: Pei He 
Date:   2017-02-14T01:17:55Z

[BEAM-59] Beam GcsFileSystem: port expand() from GcsUtil for glob matching.




> IOChannelFactory rethinking/redesign
> 
>
> Key: BEAM-59
> URL: https://issues.apache.org/jira/browse/BEAM-59
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Daniel Halperin
>Assignee: Pei He
>
> Right now, FileBasedSource and FileBasedSink communication is mediated by 
> IOChannelFactory. There are a number of issues:
> * Global configuration -- e.g., all 'gs://' URIs use the same credentials. 
> This should be per-source/per-sink/etc.
> * Supported APIs -- currently IOChannelFactory is in the "non-public API" 
> util package and subject to change. We need users to be able to add new 
> backends ('s3://', 'hdfs://', etc.) directly, without fear that they will be 
> broken.
> * Per-backend features: e.g., creating buckets in GCS/s3, setting expiration 
> time, etc.
> Updates:
> Design docs posted on dev@ list:
> Part 1: IOChannelFactory Redesign: 
> https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit#
> Part 2: Configurable BeamFileSystem:
> https://docs.google.com/document/d/1-7vo9nLRsEEzDGnb562PuL4q9mUiq_ZVpCAiyyJw8p8/edit#heading=h.p3gc3colc2cs



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1984

[dhalperi]

This closes #1984


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b67bd111
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b67bd111
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b67bd111

Branch: refs/heads/master
Commit: b67bd111e86e5e4f57fb8f1eddc790d7399209b2
Parents: a628ce3 ac7c471
Author: Dan Halperin 
Authored: Mon Feb 13 15:17:24 2017 -0800
Committer: Dan Halperin 
Committed: Mon Feb 13 15:17:24 2017 -0800

--
 .../org/apache/beam/fn/v1/standard_coders.yaml  | 10 +++
 .../transforms/windowing/IntervalWindow.java| 15 ++
 .../org/apache/beam/sdk/util/CoderUtils.java|  2 ++
 .../apache/beam/sdk/coders/CommonCoderTest.java | 31 
 sdks/python/apache_beam/coders/coder_impl.py| 31 +++-
 sdks/python/apache_beam/coders/coders.py| 15 ++
 .../apache_beam/coders/coders_test_common.py|  9 +-
 sdks/python/apache_beam/coders/slow_stream.py   |  6 
 .../apache_beam/coders/standard_coders_test.py  | 11 +--
 .../apache_beam/tests/data/standard_coders.yaml | 10 +++
 10 files changed, 130 insertions(+), 10 deletions(-)
--

[1/2] beam git commit: Add JDBC postgres IT, load script and k8 script

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 8e0573ba5 -> 4018c835c


Add JDBC postgres IT, load script and k8 script


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b284fb4d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b284fb4d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b284fb4d

Branch: refs/heads/master
Commit: b284fb4dedde188e302be01dd9426d09d7ef0021
Parents: 8e0573b
Author: Stephen Sisk 
Authored: Tue Jan 24 17:56:35 2017 -0800
Committer: Thomas Groh 
Committed: Mon Feb 13 18:12:01 2017 -0800

--
 sdks/java/io/jdbc/pom.xml   |  11 ++
 .../org/apache/beam/sdk/io/jdbc/JdbcIOIT.java   | 175 +++
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 107 +---
 .../beam/sdk/io/jdbc/JdbcTestDataSet.java   | 127 ++
 .../beam/sdk/io/jdbc/PostgresTestOptions.java   |  60 +++
 .../kubernetes/postgres-pod-no-vol.yml  |  32 
 .../kubernetes/postgres-service-public.yml  |  27 +++
 .../kubernetes/setup-postgres-service.sh|  20 +++
 8 files changed, 494 insertions(+), 65 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b284fb4d/sdks/java/io/jdbc/pom.xml
--
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index 92a3761..23feab6 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -74,6 +74,11 @@
   2.1.1
 
 
+
+  joda-time
+  joda-time
+
+
 
 
   com.google.auto.value
@@ -120,6 +125,12 @@
   slf4j-jdk14
   test
 
+
+  org.postgresql
+  postgresql
+  9.4.1212.jre7
+  test
+
   
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/b284fb4d/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
--
diff --git 
a/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java 
b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
new file mode 100644
index 000..15206c7
--- /dev/null
+++ b/sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.jdbc;
+
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.beam.sdk.coders.BigEndianIntegerCoder;
+import org.apache.beam.sdk.coders.KvCoder;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Count;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.postgresql.ds.PGSimpleDataSource;
+
+
+/**
+ * A test of {@link org.apache.beam.sdk.io.jdbc.JdbcIO} on an independent 
Postgres instance.
+ *
+ * This test requires a running instance of Postgres, and the test dataset 
must exist in the
+ * database. `JdbcTestDataSet` will create the read table.
+ *
+ * You can run just this test by doing the following:
+ * 
+ * mvn test-compile compile failsafe:integration-test -D 
beamTestPipelineOptions='[
+ * "--postgresServerName=1.2.3.4",
+ * "--postgresUsername=postgres",
+ * "--postgresDatabaseName=myfancydb",
+ * "--postgresPassword=yourpassword",
+ * "--postgresSsl=false"
+ * ]' -DskipITs=false -Dit.test=org.apache.beam.sdk.io.jdbc.JdbcIOIT 
-DfailIfNoTests=false
+ * 
+ */
+@RunWith(JUnit4.class)
+public class JdbcIOIT {
+ 

[2/2] beam git commit: This closes #1841

[tgroh]

This closes #1841


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4018c835
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4018c835
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4018c835

Branch: refs/heads/master
Commit: 4018c835c421cd6672c563b10d1686685af01079
Parents: 8e0573b b284fb4
Author: Thomas Groh 
Authored: Mon Feb 13 18:12:07 2017 -0800
Committer: Thomas Groh 
Committed: Mon Feb 13 18:12:07 2017 -0800

--
 sdks/java/io/jdbc/pom.xml   |  11 ++
 .../org/apache/beam/sdk/io/jdbc/JdbcIOIT.java   | 175 +++
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 107 +---
 .../beam/sdk/io/jdbc/JdbcTestDataSet.java   | 127 ++
 .../beam/sdk/io/jdbc/PostgresTestOptions.java   |  60 +++
 .../kubernetes/postgres-pod-no-vol.yml  |  32 
 .../kubernetes/postgres-service-public.yml  |  27 +++
 .../kubernetes/setup-postgres-service.sh|  20 +++
 8 files changed, 494 insertions(+), 65 deletions(-)
--

[GitHub] beam pull request #1841: BEAM-1310 Add integration tests for JdbcIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1841


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add cross-SDK implementations and tests of IntervalWindowCoder

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master a628ce353 -> b67bd111e


Add cross-SDK implementations and tests of IntervalWindowCoder


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ac7c4714
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ac7c4714
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ac7c4714

Branch: refs/heads/master
Commit: ac7c471473510e4f9a9281447a99ceb9552acd17
Parents: a628ce3
Author: Dan Halperin 
Authored: Fri Feb 10 11:56:00 2017 -0800
Committer: Dan Halperin 
Committed: Mon Feb 13 15:17:21 2017 -0800

--
 .../org/apache/beam/fn/v1/standard_coders.yaml  | 10 +++
 .../transforms/windowing/IntervalWindow.java| 15 ++
 .../org/apache/beam/sdk/util/CoderUtils.java|  2 ++
 .../apache/beam/sdk/coders/CommonCoderTest.java | 31 
 sdks/python/apache_beam/coders/coder_impl.py| 31 +++-
 sdks/python/apache_beam/coders/coders.py| 15 ++
 .../apache_beam/coders/coders_test_common.py|  9 +-
 sdks/python/apache_beam/coders/slow_stream.py   |  6 
 .../apache_beam/coders/standard_coders_test.py  | 11 +--
 .../apache_beam/tests/data/standard_coders.yaml | 10 +++
 10 files changed, 130 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ac7c4714/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
--
diff --git 
a/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
 
b/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
index afa92e9..948ac6b 100644
--- 
a/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
+++ 
b/sdks/common/fn-api/src/test/resources/org/apache/beam/fn/v1/standard_coders.yaml
@@ -96,3 +96,13 @@ nested: true
 examples:
   "\u0003abc\u0003def": {key: abc, value: def}
   "\u0004ab\0c\u0004de\0f": {key: "ab\0c", value: "de\0f"}
+
+---
+
+coder:
+  urn: "urn:beam:coders:intervalwindow:0.1"
+examples:
+  "\u0080\u\u0001\u0052\u009a\u00a4\u009b\u0068\u0080\u00dd\u00db\u0001" : 
{end: 1454293425000, span: 360}
+  
"\u0080\u\u0001\u0053\u0034\u00ec\u0074\u00e8\u0080\u0090\u00fb\u00d3\u0009"
 : {end: 1456881825000, span: 259200}
+  "\u007f\u00df\u003b\u0064\u005a\u001c\u00ad\u0076\u00ed\u0002" : {end: 
-9223372036854410, span: 365}
+  "\u0080\u0020\u00c4\u009b\u00a5\u00e3\u0053\u00f7\u" : {end: 
9223372036854775, span: 0}

http://git-wip-us.apache.org/repos/asf/beam/blob/ac7c4714/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/IntervalWindow.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/IntervalWindow.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/IntervalWindow.java
index fb0fc11..c0ad2c0 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/IntervalWindow.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/IntervalWindow.java
@@ -26,6 +26,7 @@ import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.DurationCoder;
 import org.apache.beam.sdk.coders.InstantCoder;
+import org.apache.beam.sdk.util.CloudObject;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.joda.time.ReadableDuration;
@@ -166,10 +167,9 @@ public class IntervalWindow extends BoundedWindow
   /**
* Encodes an {@link IntervalWindow} as a pair of its upper bound and 
duration.
*/
-  private static class IntervalWindowCoder extends AtomicCoder 
{
+  public static class IntervalWindowCoder extends AtomicCoder {
 
-private static final IntervalWindowCoder INSTANCE =
-new IntervalWindowCoder();
+private static final IntervalWindowCoder INSTANCE = new 
IntervalWindowCoder();
 
 private static final Coder instantCoder = InstantCoder.of();
 private static final Coder durationCoder = 
DurationCoder.of();
@@ -180,9 +180,7 @@ public class IntervalWindow extends BoundedWindow
 }
 
 @Override
-public void encode(IntervalWindow window,
-   OutputStream outStream,
-   Context context)
+public void encode(IntervalWindow window, OutputStream outStream, Context 
context)
 throws IOException, CoderException {
   instantCoder.encode(window.end, outStream, context.nested());
   durationCoder.encode(new Duration(window.start, window.end), outStream, 
context);
@@ -195,5 +193,10 @@ public class IntervalWindow extends BoundedWindow
   

[jira] [Created] (BEAM-1472) Use cross-language serialization schema for triggers in Python SDK

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1472:
-

 Summary: Use cross-language serialization schema for triggers in 
Python SDK
 Key: BEAM-1472
 URL: https://issues.apache.org/jira/browse/BEAM-1472
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Kenneth Knowles
Assignee: Vikas Kedigehalli


Basically, we need to do something like 
https://github.com/apache/beam/pull/1988/files for Python.

This is key for non-Python runners being able to understand trigger specs from 
Python.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2001: [BEAM-1473] Remove unused windmill proto from pytho...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2001

[BEAM-1473] Remove unused windmill proto from python sdk

R: @charlesccychen PTAL

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1473-remove-windmill-proto

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2001






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1471) Make IterableCoder binary compatible across SDKs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864848#comment-15864848
 ] 

ASF GitHub Bot commented on BEAM-1471:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1996


> Make IterableCoder binary compatible across SDKs
> 
>
> Key: BEAM-1471
> URL: https://issues.apache.org/jira/browse/BEAM-1471
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model
>Affects Versions: 0.5.0
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>
> Ensure IterableCoder across SDKs binary compatible and add tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #2626

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #2285

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1477) Support MapState in Spark runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1477:
-

 Summary: Support MapState in Spark runner
 Key: BEAM-1477
 URL: https://issues.apache.org/jira/browse/BEAM-1477
 Project: Beam
  Issue Type: New Feature
  Components: runner-spark
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1346) Drop Late Data in ReduceFnRunner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865303#comment-15865303
 ] 

Aljoscha Krettek commented on BEAM-1346:


Ah dammit, you're right, I forgot about BEAM-696. In fact the Flink Runner also 
[disables that 
optimization|https://github.com/apache/beam/blob/master/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java#L850].

> Drop Late Data in ReduceFnRunner
> 
>
> Key: BEAM-1346
> URL: https://issues.apache.org/jira/browse/BEAM-1346
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>
> I think these two commits recently broke late-data dropping for the Flink 
> Runner (and maybe for other runners as well):
> - https://github.com/apache/beam/commit/2b26ec8
> - https://github.com/apache/beam/commit/8989473
> It boils down to the {{LateDataDroppingDoFnRunner}} not being used anymore  
> because {{DoFnRunners.lateDataDroppingRunner()}} is not called anymore when a 
> {{DoFn}} is a {{ReduceFnExecutor}} (because that interface was removed).
> Maybe we should think about dropping late data in another place, my 
> suggestion is {{ReduceFnRunner}} but that's open for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1975: Changed name of ToString.of() to ToString.elements().

[kenn]

This closes #1975: Changed name of ToString.of() to ToString.elements().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/98d8834a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/98d8834a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/98d8834a

Branch: refs/heads/master
Commit: 98d8834afba705470a43ebc0349f76334d49f00a
Parents: f32cb3e a49acda
Author: Kenneth Knowles 
Authored: Mon Feb 13 20:29:32 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 13 20:29:32 2017 -0800

--
 .../src/main/java/org/apache/beam/sdk/transforms/ToString.java   | 4 ++--
 .../core/src/test/java/org/apache/beam/sdk/io/WriteTest.java | 2 +-
 .../test/java/org/apache/beam/sdk/transforms/ToStringTest.java   | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: Changed name of ToString.of() to ToString.elements().

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master f32cb3e7a -> 98d8834af


Changed name of ToString.of() to ToString.elements().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a49acdad
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a49acdad
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a49acdad

Branch: refs/heads/master
Commit: a49acdadac37fd52cc1184e5d29cc30b99742a35
Parents: effca63
Author: Jesse Anderson 
Authored: Fri Feb 10 12:17:21 2017 -0800
Committer: Jesse Anderson 
Committed: Fri Feb 10 12:17:21 2017 -0800

--
 .../src/main/java/org/apache/beam/sdk/transforms/ToString.java   | 4 ++--
 .../core/src/test/java/org/apache/beam/sdk/io/WriteTest.java | 2 +-
 .../test/java/org/apache/beam/sdk/transforms/ToStringTest.java   | 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a49acdad/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
index d5c9784..5069a3c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToString.java
@@ -42,7 +42,7 @@ public final class ToString {
* element of the input {@link PCollection} to a {@link String} using the
* {@link Object#toString} method.
*/
-  public static PTransform of() {
+  public static PTransform elements() {
 return new SimpleToString();
   }
 
@@ -97,7 +97,7 @@ public final class ToString {
* Example of use:
* {@code
* PCollection longs = ...;
-   * PCollection strings = longs.apply(ToString.of());
+   * PCollection strings = longs.apply(ToString.elements());
* }
*
*

http://git-wip-us.apache.org/repos/asf/beam/blob/a49acdad/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
--
diff --git a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
index f81cc0c..846d445 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteTest.java
@@ -297,7 +297,7 @@ public class WriteTest {
   @Test
   public void testWriteUnbounded() {
 PCollection unbounded = p.apply(CountingInput.unbounded())
-.apply(ToString.of());
+.apply(ToString.elements());
 
 TestSink sink = new TestSink();
 thrown.expect(IllegalArgumentException.class);

http://git-wip-us.apache.org/repos/asf/beam/blob/a49acdad/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToStringTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToStringTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToStringTest.java
index ab984f1..d2116da 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToStringTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToStringTest.java
@@ -48,7 +48,7 @@ public class ToStringTest {
 Integer[] ints = {1, 2, 3, 4, 5};
 String[] strings = {"1", "2", "3", "4", "5"};
 PCollection input = p.apply(Create.of(Arrays.asList(ints)));
-PCollection output = input.apply(ToString.of());
+PCollection output = input.apply(ToString.elements());
 PAssert.that(output).containsInAnyOrder(strings);
 p.run();
   }

[jira] [Commented] (BEAM-1460) Change ToString Method Name

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865051#comment-15865051
 ] 

ASF GitHub Bot commented on BEAM-1460:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1975


> Change ToString Method Name
> ---
>
> Key: BEAM-1460
> URL: https://issues.apache.org/jira/browse/BEAM-1460
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.5.0
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
>
> Need to change ToString's of() method to elements() to comply with the naming 
> guidelines.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1482) Support SetState in Spark runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1482:
-

 Summary: Support SetState in Spark runner
 Key: BEAM-1482
 URL: https://issues.apache.org/jira/browse/BEAM-1482
 Project: Beam
  Issue Type: New Feature
  Components: runner-spark
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1483) Support SetState in Flink runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1483:
-

 Summary: Support SetState in Flink runner
 Key: BEAM-1483
 URL: https://issues.apache.org/jira/browse/BEAM-1483
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1009) Upgrade from mockito-all 1 to mockito-core 2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865173#comment-15865173
 ] 

ASF GitHub Bot commented on BEAM-1009:
--

Github user peihe closed the pull request at:

https://github.com/apache/beam/pull/1448


> Upgrade from mockito-all 1 to mockito-core 2
> 
>
> Key: BEAM-1009
> URL: https://issues.apache.org/jira/browse/BEAM-1009
> Project: Beam
>  Issue Type: Test
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>
> Mockito 2 provides useful features, and the mockito-all module is no longer 
> generated.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1448: [BEAM-1009] Update to Mockito 2 with mockito-core m...

[peihe]

Github user peihe closed the pull request at:

https://github.com/apache/beam/pull/1448


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1891: Correct Javadoc on accessing windows in DoFn

[kenn]

This closes #1891: Correct Javadoc on accessing windows in DoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/49809d1d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/49809d1d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/49809d1d

Branch: refs/heads/master
Commit: 49809d1d43c409b23771238af085f9ebcd0d3cb3
Parents: 98d8834 c1c8d83
Author: Kenneth Knowles 
Authored: Mon Feb 13 20:44:21 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 13 20:44:21 2017 -0800

--
 .../src/main/java/org/apache/beam/sdk/transforms/DoFn.java | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/49809d1d/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--

[1/2] beam git commit: Correct Javadoc on accessing windows in DoFn

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 98d8834af -> 49809d1d4


Correct Javadoc on accessing windows in DoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c1c8d838
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c1c8d838
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c1c8d838

Branch: refs/heads/master
Commit: c1c8d8386fc035c65362b5ce36c20b38fe00f9a4
Parents: 0e6b379
Author: Ben Chambers 
Authored: Wed Feb 1 15:08:13 2017 -0800
Committer: Kenneth Knowles 
Committed: Mon Feb 13 20:44:04 2017 -0800

--
 .../src/main/java/org/apache/beam/sdk/transforms/DoFn.java | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c1c8d838/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
index a161919..1ad05bb 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
@@ -532,8 +532,10 @@ public abstract class DoFn implements 
Serializable, HasDisplayD
* href="https://s.apache.org/splittable-do-fn;>splittable {@link 
DoFn} subject to the
* separate requirements described below. Items below are assuming this 
is not a splittable
* {@link DoFn}.
-   * If one of its arguments is {@link BoundedWindow}, this argument 
corresponds to the window
-   * of the current element. If absent, a runner may perform additional 
optimizations.
+   * If one of its arguments is a subtype of {@link BoundedWindow} then it 
will
+   * be passed the window of the current element. When applied by {@link 
ParDo} the subtype
+   * of {@link BoundedWindow} must match the type of windows on the input 
{@link PCollection}.
+   * If the window is not accessed a runner may perform additional 
optimizations.
* It must return {@code void}.
* 
*

[GitHub] beam pull request #1891: Correct Javadoc on accessing windows in DoFn

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1891


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-59] Beam FileSystem: match() and its local implementation.

[pei]

Repository: beam
Updated Branches:
  refs/heads/master 49809d1d4 -> bea101a44


[BEAM-59] Beam FileSystem: match() and its local implementation.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d1648c47
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d1648c47
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d1648c47

Branch: refs/heads/master
Commit: d1648c47dd4fef00273ceb46d42d784325b3b1e8
Parents: 49809d1
Author: Pei He 
Authored: Fri Feb 10 21:53:31 2017 -0800
Committer: Pei He 
Committed: Mon Feb 13 22:08:50 2017 -0800

--
 .../java/org/apache/beam/sdk/io/FileSystem.java |  30 
 .../org/apache/beam/sdk/io/LocalFileSystem.java |  74 ++
 .../org/apache/beam/sdk/io/fs/MatchResult.java  | 125 
 .../apache/beam/sdk/io/LocalFileSystemTest.java | 148 +++
 .../beam/sdk/util/FileIOChannelFactoryTest.java |  13 +-
 .../beam/sdk/io/gcp/storage/GcsFileSystem.java  |   6 +
 .../beam/sdk/io/hdfs/HadoopFileSystem.java  |   6 +
 7 files changed, 391 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d1648c47/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
index ecfa29b..001f596 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileSystem.java
@@ -24,6 +24,7 @@ import java.nio.channels.WritableByteChannel;
 import java.util.Collection;
 import java.util.List;
 import org.apache.beam.sdk.io.fs.CreateOptions;
+import org.apache.beam.sdk.io.fs.MatchResult;
 import org.apache.beam.sdk.io.fs.ResourceId;
 
 /**
@@ -35,6 +36,35 @@ import org.apache.beam.sdk.io.fs.ResourceId;
  * Clients should use {@link FileSystems} utility.
  */
 public abstract class FileSystem {
+  /**
+   * This is the entry point to convert user-provided specs to {@link 
ResourceIdT ResourceIds}.
+   * Callers should use {@link #match} to resolve users specs ambiguities 
before
+   * calling other methods.
+   *
+   * Implementation should handle the following ambiguities of a 
user-provided spec:
+   * 
+   * {@code spec} could be a glob or a uri. {@link #match} should be able 
to tell and
+   * choose efficient implementations.
+   * The user-provided {@code spec} might refer to files or directories. 
It is common that
+   * users that wish to indicate a directory will omit the trailing {@code /}, 
such as in a spec of
+   * {@code "/tmp/dir"}. The {@link FileSystem} should be able to recognize a 
directory with
+   * the trailing {@code /} omitted, but should always return a correct {@link 
ResourceIdT}
+   * (e.g., {@code "/tmp/dir/"} inside the returned {@link MatchResult}.
+   * 
+   *
+   * All {@link FileSystem} implementations should support glob in the 
final hierarchical path
+   * component of {@link ResourceIdT}. This allows SDK libraries to construct 
file system agnostic
+   * spec. {@link FileSystem FileSystems} can support additional patterns for 
user-provided specs.
+   *
+   * @return {@code List} in the same order of the input specs.
+   *
+   * @throws IllegalArgumentException if specs are invalid.
+   * @throws IOException if all specs failed to match due to issues like:
+   * network connection, authorization.
+   * Exception for individual spec need to be deferred until callers retrieve
+   * metadata with {@link MatchResult#metadata()}.
+   */
+  protected abstract List match(List specs) throws 
IOException;
 
   /**
* Returns a write channel for the given {@link ResourceIdT}.

http://git-wip-us.apache.org/repos/asf/beam/blob/d1648c47/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
index 0e79c9c..fe6b643 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/LocalFileSystem.java
@@ -19,6 +19,11 @@ package org.apache.beam.sdk.io;
 
 import static com.google.common.base.Preconditions.checkArgument;
 
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
 import java.io.BufferedOutputStream;
 import java.io.File;
 import java.io.FileInputStream;
@@ -29,10 +34,16 @@ 

[2/2] beam git commit: This closes #1982

[pei]

This closes #1982


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bea101a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bea101a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bea101a4

Branch: refs/heads/master
Commit: bea101a4479645b020198dccefb2c6f51a7cc6f0
Parents: 49809d1 d1648c4
Author: Pei He 
Authored: Mon Feb 13 22:09:59 2017 -0800
Committer: Pei He 
Committed: Mon Feb 13 22:09:59 2017 -0800

--
 .../java/org/apache/beam/sdk/io/FileSystem.java |  30 
 .../org/apache/beam/sdk/io/LocalFileSystem.java |  74 ++
 .../org/apache/beam/sdk/io/fs/MatchResult.java  | 125 
 .../apache/beam/sdk/io/LocalFileSystemTest.java | 148 +++
 .../beam/sdk/util/FileIOChannelFactoryTest.java |  13 +-
 .../beam/sdk/io/gcp/storage/GcsFileSystem.java  |   6 +
 .../beam/sdk/io/hdfs/HadoopFileSystem.java  |   6 +
 7 files changed, 391 insertions(+), 11 deletions(-)
--

[GitHub] beam pull request #1982: [BEAM-59] Beam FileSystem: match() and its local im...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1982


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1045) Windows OS compatibilities

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865206#comment-15865206
 ] 

ASF GitHub Bot commented on BEAM-1045:
--

Github user peihe closed the pull request at:

https://github.com/apache/beam/pull/1412


> Windows OS compatibilities
> --
>
> Key: BEAM-1045
> URL: https://issues.apache.org/jira/browse/BEAM-1045
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Pei He
>
> One known issue is "*" is not allowed in Windows OS.
> For example, Paths.get("tempDir/*") might throw when code runs in Windows OS.
> http://stackoverflow.com/questions/27522581/asterisks-in-java-path
> This affecting IOChannelFactory.resolve(), toPath(), and match().
> For match(), since it only requires support globs in the final component of a 
> path. (local) FileIOChannelFactory could do things similar as 
> GcsIOChannelFactory:
> First, list all files under the directory path (this won't contain glob, such 
> as *).
> Then, check each returned files whether it matches glob.
> In this way, glob (*) stays within Apache Beam's code.
> From match()'s javadoc:
> """
> Glob handling is dependent on the implementation.  Implementations should
>* all support globs in the final component of a path (eg /foo/bar/*.txt),
>* however they are not required to support globs in the directory paths.
> """



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1480) Support SetState in Apex runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1480:
-

 Summary: Support SetState in Apex runner
 Key: BEAM-1480
 URL: https://issues.apache.org/jira/browse/BEAM-1480
 Project: Beam
  Issue Type: New Feature
  Components: runner-apex
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1479) Support SetState in Dataflow runner

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1479:
-

 Summary: Support SetState in Dataflow runner
 Key: BEAM-1479
 URL: https://issues.apache.org/jira/browse/BEAM-1479
 Project: Beam
  Issue Type: New Feature
  Components: runner-dataflow
Reporter: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-27) Add user-ready API for interacting with timers

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-27?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-27.
-
   Resolution: Implemented
Fix Version/s: 0.6.0

This is supported in enough runners that we should track only limitations and 
bugs going forward.

> Add user-ready API for interacting with timers
> --
>
> Key: BEAM-27
> URL: https://issues.apache.org/jira/browse/BEAM-27
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
> Fix For: 0.6.0
>
>
> Pipeline authors will benefit from a different factorization of interaction 
> with underlying timers. The current APIs are targeted at runner implementers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-25) Add user-ready API for interacting with state

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-25.
-
   Resolution: Implemented
Fix Version/s: 0.6.0

This is now supported by enough runners that we should track only follow-up 
tickets for limitations, not the overall ticket.

> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
> Fix For: 0.6.0
>
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[Stas Levin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stas Levin reassigned BEAM-1048:


Assignee: Stas Levin  (was: Kobi Salant)

> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Stas Levin
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1346) Drop Late Data in ReduceFnRunner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863721#comment-15863721
 ] 

Aljoscha Krettek commented on BEAM-1346:


[~kenn] another thing that crossed my mind is elements being pushed back due to 
their side input not being ready. Think {{PushbackSideInputRunner}} and similar 
implementations for other runners, if they have it. It's similar to this issue 
but in the end we probably need a separate issue.

The problem occurs when you have a special implementation for "combine" that 
doesn't simply do {{GroupByKey | ParDo(CombineFn)}} where the first one is 
{{GroupByKey: KV → KV}}. The {{CombineFn}} can access side 
inputs and the side input that it can access is determined by the window that 
the value has after merging (as evident from the proper definition of combine 
given above). {{PushbackSideInputRunner}}, however, only considers the 
(proto-)window that the value has before merging so the pushing back and 
determining when a side input is ready is based on the wrong information.

Do you agree or is that just me getting a little paranoid with the whole 
merging stuff? ;-)

> Drop Late Data in ReduceFnRunner
> 
>
> Key: BEAM-1346
> URL: https://issues.apache.org/jira/browse/BEAM-1346
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>
> I think these two commits recently broke late-data dropping for the Flink 
> Runner (and maybe for other runners as well):
> - https://github.com/apache/beam/commit/2b26ec8
> - https://github.com/apache/beam/commit/8989473
> It boils down to the {{LateDataDroppingDoFnRunner}} not being used anymore  
> because {{DoFnRunners.lateDataDroppingRunner()}} is not called anymore when a 
> {{DoFn}} is a {{ReduceFnExecutor}} (because that interface was removed).
> Maybe we should think about dropping late data in another place, my 
> suggestion is {{ReduceFnRunner}} but that's open for discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1048) Spark Runner streaming batch duration does not include duration of reading from source

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863792#comment-15863792
 ] 

Stas Levin commented on BEAM-1048:
--

[~ksalant], following our discussion and your blessing I'm assigning this to 
myself :)

> Spark Runner streaming batch duration does not include duration of reading 
> from source 
> ---
>
> Key: BEAM-1048
> URL: https://issues.apache.org/jira/browse/BEAM-1048
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 0.4.0
>Reporter: Kobi Salant
>Assignee: Kobi Salant
>
> Spark Runner streaming batch duration does not include duration of reading 
> from source this is because we perform rdd.count in SparkUnboundedSourcewhich 
> that invokes a regular spark job outside the streaming context.
> We do it for reporting the batch size both for UI and back pressure



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863967#comment-15863967
 ] 

Aljoscha Krettek commented on BEAM-1461:


{{DoFn.prepareForProcessing()}} is only invoked once, before executing a 
pipeline using a Runner. The {{@StartBundle}} method is invoked before 
processing a bundle, this can happen several times if you have many bundles.

[~davor] you think we should maybe add better JavaDoc to 
{{DoFn.prepareForProcessing()}}? If not, I think we should close this issue. 
What do you think, [~mingmxu]?

> duplication with StartBundle and prepareForProcessing in DoFn
> -
>
> Key: BEAM-1461
> URL: https://issues.apache.org/jira/browse/BEAM-1461
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> There're one annotation `StartBundle`, and one public function 
> `prepareForProcessing` in DoFn, which are called both before 
> `ProcessElement`. It's confused which one should be implemented in a subclass.
> The call sequence seems as:
> prepareForProcessing -> StartBundle -> processElement



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1433) Remove coder from TextIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863923#comment-15863923
 ] 

ASF GitHub Bot commented on BEAM-1433:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1944


> Remove coder from TextIO
> 
>
> Key: BEAM-1433
> URL: https://issues.apache.org/jira/browse/BEAM-1433
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Remove coder usage in TextIO.
> TextIO should only deal with Strings.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-1433] Remove coder from TextIO

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master d4db4fb2c -> e2ef2d020


[BEAM-1433] Remove coder from TextIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/adba4c66
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/adba4c66
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/adba4c66

Branch: refs/heads/master
Commit: adba4c660ef54b98055d30ee1ad7cbf440420030
Parents: d4db4fb
Author: Aviem Zur 
Authored: Tue Feb 7 22:56:17 2017 +0200
Committer: Dan Halperin 
Committed: Mon Feb 13 08:14:06 2017 -0800

--
 .../examples/complete/TopWikipediaSessions.java |  25 +-
 .../FlinkStreamingTransformTranslators.java |  18 +-
 .../spark/translation/TransformTranslator.java  |  12 +-
 .../java/org/apache/beam/sdk/io/TextIO.java | 383 +++
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 164 +++-
 5 files changed, 240 insertions(+), 362 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/adba4c66/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
index 7eb80b7..4c07ca4 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
@@ -18,9 +18,9 @@
 package org.apache.beam.examples.complete;
 
 import com.google.api.services.bigquery.model.TableRow;
+import java.io.IOException;
 import java.util.List;
 import org.apache.beam.sdk.Pipeline;
-import org.apache.beam.sdk.coders.TableRowJsonCoder;
 import org.apache.beam.sdk.io.TextIO;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
@@ -29,15 +29,18 @@ import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.Validation;
 import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.MapElements;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
 import org.apache.beam.sdk.transforms.SerializableComparator;
+import org.apache.beam.sdk.transforms.SimpleFunction;
 import org.apache.beam.sdk.transforms.Top;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.CalendarWindows;
 import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.transforms.windowing.Sessions;
 import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.util.Transport;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 import org.joda.time.Duration;
@@ -136,6 +139,17 @@ public class TopWikipediaSessions {
 }
   }
 
+  static class ParseTableRowJson extends SimpleFunction {
+@Override
+public TableRow apply(String input) {
+  try {
+return Transport.getJsonFactory().fromString(input, TableRow.class);
+  } catch (IOException e) {
+throw new RuntimeException("Failed parsing table row json", e);
+  }
+}
+  }
+
   static class ComputeTopSessions extends PTransform {
 
 private final double samplingThreshold;
@@ -193,11 +207,10 @@ public class TopWikipediaSessions {
 
 double samplingThreshold = 0.1;
 
-p.apply(TextIO.Read
-.from(options.getInput())
-.withCoder(TableRowJsonCoder.of()))
- .apply(new ComputeTopSessions(samplingThreshold))
- .apply("Write", TextIO.Write.withoutSharding().to(options.getOutput()));
+p.apply(TextIO.Read.from(options.getInput()))
+.apply(MapElements.via(new ParseTableRowJson()))
+.apply(new ComputeTopSessions(samplingThreshold))
+.apply("Write", 
TextIO.Write.withoutSharding().to(options.getOutput()));
 
 p.run().waitUntilFinish();
   }

http://git-wip-us.apache.org/repos/asf/beam/blob/adba4c66/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index 1195c82..b9b5059 100644
--- 

[jira] [Commented] (BEAM-1469) Estimate Pi test is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863982#comment-15863982
 ] 

ASF GitHub Bot commented on BEAM-1469:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1992

[BEAM-1469] Increase the bounds on the test to reduce flakiness

R: @aaltay PTAL

Surprisingly I ran the test 100 times in a loop and didn't see a flake 

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1469-fix-flaky-pi-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1992.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1992


commit 430ba5f56abff562effc000d1b0fc93973b2e3e0
Author: Sourabh Bajaj 
Date:   2017-02-13T16:59:59Z

Increase the bounds on the test to reduce flakiness




> Estimate Pi test is flaky
> -
>
> Key: BEAM-1469
> URL: https://issues.apache.org/jira/browse/BEAM-1469
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> I get this error sometimes:
> DataflowAssertException: Failed assert: 3.150432 not in [3.13, 3.15] 
> [while running 'assert_that/Match']
> root: WARNING: A task failed with exception.
>  Failed assert: 3.150432 not in [3.13, 3.15] [while running 
> 'assert_that/Match']
> This is probably related to the reduced trial size, We can relax the 
> upper/lower bounds a little bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #1944

[dhalperi]

This closes #1944


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e2ef2d02
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e2ef2d02
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e2ef2d02

Branch: refs/heads/master
Commit: e2ef2d020e1f9f40b79e122807fb98a31e1f439e
Parents: d4db4fb adba4c6
Author: Dan Halperin 
Authored: Mon Feb 13 08:14:09 2017 -0800
Committer: Dan Halperin 
Committed: Mon Feb 13 08:14:09 2017 -0800

--
 .../examples/complete/TopWikipediaSessions.java |  25 +-
 .../FlinkStreamingTransformTranslators.java |  18 +-
 .../spark/translation/TransformTranslator.java  |  12 +-
 .../java/org/apache/beam/sdk/io/TextIO.java | 383 +++
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 164 +++-
 5 files changed, 240 insertions(+), 362 deletions(-)
--

[GitHub] beam pull request #1944: [BEAM-1433] Remove coder from TextIO

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1944


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1468) Upgrade datastore dependency the 0.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863196#comment-15863196
 ] 

ASF GitHub Bot commented on BEAM-1468:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1989

[BEAM-1468] Upgrading datatore dependency to 0.7.0 version

R: @vikkyrk 

Vikas, how can I test this beyond unit tests?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam ds

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1989.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1989


commit 6b5786969766415b9c3b62bca4283c9ab7024e71
Author: Ahmet Altay 
Date:   2017-02-11T03:02:25Z

Upgrading to datatore 0.7.0 library




> Upgrade datastore dependency the 0.7.0
> --
>
> Key: BEAM-1468
> URL: https://issues.apache.org/jira/browse/BEAM-1468
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1195) Give triggers a cross-language serialization schema

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1195:
--
Summary: Give triggers a cross-language serialization schema  (was: Give 
triggers URNs / JSON formats)

> Give triggers a cross-language serialization schema
> ---
>
> Key: BEAM-1195
> URL: https://issues.apache.org/jira/browse/BEAM-1195
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We have recently gotten to the point where triggers are just syntax, but it 
> is still shipped via Java serialization. To make it language-independent, we 
> need a concrete syntax.
> Something like the following is fairly concise, tag adjacent to payload. I 
> haven't bothered making up fully verbose/namespaced URNs here.
> {code}
> {
> "$urn": "OrFinally",
> "main": {
>   "$urn": "EndOfWindow",
>   "early": 
> },
> "finally": {
>   "$urn": "AfterCount",
>   "count": 45
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1195) Give triggers URNs / JSON formats

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863188#comment-15863188
 ] 

Kenneth Knowles commented on BEAM-1195:
---

Instead, they will be directly represented in a proto schema.

> Give triggers URNs / JSON formats
> -
>
> Key: BEAM-1195
> URL: https://issues.apache.org/jira/browse/BEAM-1195
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We have recently gotten to the point where triggers are just syntax, but it 
> is still shipped via Java serialization. To make it language-independent, we 
> need a concrete syntax.
> Something like the following is fairly concise, tag adjacent to payload. I 
> haven't bothered making up fully verbose/namespaced URNs here.
> {code}
> {
> "$urn": "OrFinally",
> "main": {
>   "$urn": "EndOfWindow",
>   "early": 
> },
> "finally": {
>   "$urn": "AfterCount",
>   "count": 45
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1195) Give triggers a cross-language serialization schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863190#comment-15863190
 ] 

ASF GitHub Bot commented on BEAM-1195:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1988

[BEAM-1195, BEAM-115] Add full-fidelity conversion to/from Java SDK 
triggers and Runner API protos

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This PR has somewhat independent commits that make sense to review together:

1. Alter the Java SDK's trigger classes to have the necessary functionality 
to reconstruct easily from a serialized form.
2. Simplify some inheritances relationship in the Java SDK that were 
actually incorrect.
3. Flesh out the Runner API proto to include all the needed bits to 
serialize triggers fully.
4. Actually serialize triggers and their inverse; test the round trip.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam trigger-proto

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1988.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1988


commit 46df18fc2a0de22ced7dbeaf47508a5bdd9c1493
Author: Kenneth Knowles 
Date:   2017-02-12T23:03:48Z

Reify delay and alignment in AfterProcessingTime transform

commit 8d62655b75377be8d0a6be2803abf25882d36663
Author: Kenneth Knowles 
Date:   2017-02-12T00:47:15Z

Add runner API config to poms

commit 16d5704ec5f6a0db8c678c69f2acc122aa1e0b4a
Author: Kenneth Knowles 
Date:   2017-02-12T00:48:05Z

Fix typo in runner API generated Java class

commit 412145654af5f126776913fcc99595a729f10967
Author: Kenneth Knowles 
Date:   2017-02-12T01:26:56Z

Add neglected ElementCount trigger to runner API proto

commit 0b222ed0aa653faf1d3cd39873f2ff845016f2f8
Author: Kenneth Knowles 
Date:   2017-02-12T01:32:21Z

Add TimeDomain to runner API protos

commit 1e669eecaadadb1e8cbf5b031aff613efe30c0db
Author: Kenneth Knowles 
Date:   2017-02-12T01:38:01Z

Flesh out AfterProcessingTime in runner API

commit 6954b4c539568b463aa602e012fa5aad2309a741
Author: Kenneth Knowles 
Date:   2017-02-12T23:53:17Z

Upgrade Java triggers to support runner API deserialization

commit 3100790d33dfa741c24fd62002cc62b17b142342
Author: Kenneth Knowles 
Date:   2017-02-12T01:50:27Z

Add conversion to/from Java SDK trigger to runner API proto

commit 69d0957d1109125294be5f057643684ea005dd06
Author: Kenneth Knowles 
Date:   2017-02-13T04:11:59Z

fixup! Reify delay and alignment in AfterProcessingTime transform




> Give triggers a cross-language serialization schema
> ---
>
> Key: BEAM-1195
> URL: https://issues.apache.org/jira/browse/BEAM-1195
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> We have recently gotten to the point where triggers are just syntax, but it 
> is still shipped via Java serialization. To make it language-independent, we 
> need a concrete syntax.
> Something like the following is fairly concise, tag adjacent to payload. I 
> haven't bothered making up fully verbose/namespaced URNs here.
> {code}
> {
> "$urn": "OrFinally",
> "main": {
>   "$urn": "EndOfWindow",
>   "early": 
> },
> "finally": {
>   "$urn": "AfterCount",
>   "count": 45
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1294) Long running UnboundedSource Readers via Broadcasts

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1294:
---

Assignee: Aviem Zur  (was: Amit Sela)

> Long running UnboundedSource Readers via Broadcasts
> ---
>
> Key: BEAM-1294
> URL: https://issues.apache.org/jira/browse/BEAM-1294
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Aviem Zur
>
> When reading from an UnboundedSource, current implementation will cause each 
> split to create a new Reader every micro-batch.
> As long as the overhead of creating a reader is relatively low, it's 
> reasonable (though I'd still be happy to get rid of), but in cases where the 
> creation overhead is large it becomes unreasonable forcing large batches.
> One way to solve this could be to create a pool of lazy-init readers to serve 
> each executor, maybe via Broadcast variables. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1470) A quite logger for testing

[Amit Sela (JIRA)]

Amit Sela created BEAM-1470:
---

 Summary: A quite logger for testing
 Key: BEAM-1470
 URL: https://issues.apache.org/jira/browse/BEAM-1470
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela
Priority: Minor


Make test logger quite and clean to avoid flooding build logs and to allows 
better visibility in case something breaks since current logging is very 
crowded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1457) Enable rat plugin and findbugs plugin in default build

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863223#comment-15863223
 ] 

Aviem Zur commented on BEAM-1457:
-

Not really sure what was decided around this issue.

> Enable rat plugin and findbugs plugin in default build
> --
>
> Key: BEAM-1457
> URL: https://issues.apache.org/jira/browse/BEAM-1457
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system
>Reporter: Aviem Zur
>Assignee: Davor Bonaci
>
> Today, maven rat plugin and findbugs plugin only run when `release` profile 
> is specified.
> Since these plugins do not add a large amount of time compared to the normal 
> build, and their checks are required to pass to approve pull requests - let's 
> enable them by default.
> [Original dev list 
> discussion|https://lists.apache.org/thread.html/e1f80e54b44b4a39630d978abe79fb6a6cecf71d9821ee1881b47afb@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1470) A quite logger for testing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863385#comment-15863385
 ] 

ASF GitHub Bot commented on BEAM-1470:
--

GitHub user amitsela opened a pull request:

https://github.com/apache/beam/pull/1991

[BEAM-1470] A quite logger for testing.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/beam BEAM-1470

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1991


commit a96be8d511efafa3986af3887b9421dca693b16a
Author: Sela 
Date:   2017-02-13T09:27:42Z

[BEAM-1470] A quite logger for testing.




> A quite logger for testing
> --
>
> Key: BEAM-1470
> URL: https://issues.apache.org/jira/browse/BEAM-1470
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>
> Make test logger quite and clean to avoid flooding build logs and to allows 
> better visibility in case something breaks since current logging is very 
> crowded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1347) Basic Java harness capable of understanding process bundle tasks and sending data over the Fn Api

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863253#comment-15863253
 ] 

ASF GitHub Bot commented on BEAM-1347:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/1990

[BEAM-1347,BEAM-1348] Use strings for ids in Fn API

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Currently the Fn API uses 64 bit integers pervasively for identifiers. This 
is acceptable for performance-sensitive locations if it is known that 
performance is improved.

The Runner API uses strings for identifiers, which is consistent with proto 
best practices for extensibility and usability. And in the Runner API these two 
factors trump performance, since it has negligibly low throughput requirements.

Since the Fn API must transition to `FunctionSpec` and `Coder` from the 
Runner API (largely the same, but more fleshed out) and we desire to also share 
`GraphNode`, I have done the brain-dead automated thing to convert the 
identifiers to strings.

In the Runner API many of these identifiers disappear anyhow, but 
conflating the unification of the APIs with this conversion got a bit out of 
hand, so I'd like to do this first.

I would like feedback from those more familiar with the harness about where 
this is known to be a performance bottleneck.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam fn-api-string-ids

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1990


commit 54c1a9e1583e99a01167cbf605043e3139798870
Author: Kenneth Knowles 
Date:   2017-02-13T06:23:32Z

Use strings for ids in Fn API




> Basic Java harness capable of understanding process bundle tasks and sending 
> data over the Fn Api
> -
>
> Key: BEAM-1347
> URL: https://issues.apache.org/jira/browse/BEAM-1347
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-fn-api
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>
> Create a basic Java harness capable of understanding process bundle requests 
> and able to stream data over the Fn Api.
> Overview: https://s.apache.org/beam-fn-api



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1468) Upgrade datastore dependency the 0.7.0

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1468:
-

 Summary: Upgrade datastore dependency the 0.7.0
 Key: BEAM-1468
 URL: https://issues.apache.org/jira/browse/BEAM-1468
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Ahmet Altay
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1992: [BEAM-1469] Increase the bounds on the test to redu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1992


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1469) Estimate Pi test is flaky

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864071#comment-15864071
 ] 

ASF GitHub Bot commented on BEAM-1469:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1992


> Estimate Pi test is flaky
> -
>
> Key: BEAM-1469
> URL: https://issues.apache.org/jira/browse/BEAM-1469
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>Priority: Minor
>
> I get this error sometimes:
> DataflowAssertException: Failed assert: 3.150432 not in [3.13, 3.15] 
> [while running 'assert_that/Match']
> root: WARNING: A task failed with exception.
>  Failed assert: 3.150432 not in [3.13, 3.15] [while running 
> 'assert_that/Match']
> This is probably related to the reduced trial size, We can relax the 
> upper/lower bounds a little bit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Increase the bounds on the test to reduce flakiness

[altay]

Repository: beam
Updated Branches:
  refs/heads/master e2ef2d020 -> 982ea7af7


Increase the bounds on the test to reduce flakiness


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c5e53633
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c5e53633
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c5e53633

Branch: refs/heads/master
Commit: c5e53633fbaee58d09a0c7e60be15a87a9a25699
Parents: e2ef2d0
Author: Sourabh Bajaj 
Authored: Mon Feb 13 09:10:48 2017 -0800
Committer: Sourabh Bajaj 
Committed: Mon Feb 13 09:10:48 2017 -0800

--
 sdks/python/apache_beam/examples/complete/estimate_pi_test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c5e53633/sdks/python/apache_beam/examples/complete/estimate_pi_test.py
--
diff --git a/sdks/python/apache_beam/examples/complete/estimate_pi_test.py 
b/sdks/python/apache_beam/examples/complete/estimate_pi_test.py
index ddd5f76..235cee9 100644
--- a/sdks/python/apache_beam/examples/complete/estimate_pi_test.py
+++ b/sdks/python/apache_beam/examples/complete/estimate_pi_test.py
@@ -42,8 +42,8 @@ class EstimatePiTest(unittest.TestCase):
 result = p | 'Estimate' >> estimate_pi.EstimatePiTransform(5000)
 
 # Note: Probabilistically speaking this test can fail with a probability
-# that is very small (VERY) given that we run at least 10 million trials.
-assert_that(result, in_between(3.13, 3.15))
+# that is very small (VERY) given that we run at least 500 thousand trials.
+assert_that(result, in_between(3.125, 3.155))
 p.run()
 
 

[GitHub] beam pull request #1993: [BEAM-1461] Make PrepareForProcessing Final, Deprec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1993


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: This closes #1993

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 982ea7af7 -> 30cb93ced


This closes #1993


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/30cb93ce
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/30cb93ce
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/30cb93ce

Branch: refs/heads/master
Commit: 30cb93ced35494db79c81c49e07a298f560c3905
Parents: 982ea7a 8269027
Author: Thomas Groh 
Authored: Mon Feb 13 09:50:12 2017 -0800
Committer: Thomas Groh 
Committed: Mon Feb 13 09:50:12 2017 -0800

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864094#comment-15864094
 ] 

ASF GitHub Bot commented on BEAM-646:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1995

[BEAM-646] Add ReplaceOutputs to PTransformOverrideFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This maps the outputs produced by applying a Replacement PTransform to
the outputs produced by the original PTransform.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam replacement_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1995.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1995


commit f46459d3179c7d3aaa675bff2f88e15fca1ae883
Author: Thomas Groh 
Date:   2017-02-09T19:11:23Z

Add ReplaceOutputs to PTransformOverrideFactory

This maps the outputs produced by applying a Replacement PTransform to
the outputs produced by the original PTransform.




> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1995: [BEAM-646] Add ReplaceOutputs to PTransformOverride...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1995

[BEAM-646] Add ReplaceOutputs to PTransformOverrideFactory

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This maps the outputs produced by applying a Replacement PTransform to
the outputs produced by the original PTransform.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam replacement_output

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1995.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1995


commit f46459d3179c7d3aaa675bff2f88e15fca1ae883
Author: Thomas Groh 
Date:   2017-02-09T19:11:23Z

Add ReplaceOutputs to PTransformOverrideFactory

This maps the outputs produced by applying a Replacement PTransform to
the outputs produced by the original PTransform.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1433) Remove coder from TextIO

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur resolved BEAM-1433.
-
   Resolution: Done
Fix Version/s: 0.6.0

> Remove coder from TextIO
> 
>
> Key: BEAM-1433
> URL: https://issues.apache.org/jira/browse/BEAM-1433
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
> Fix For: 0.6.0
>
>
> Remove coder usage in TextIO.
> TextIO should only deal with Strings.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1461) duplication with StartBundle and prepareForProcessing in DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863990#comment-15863990
 ] 

ASF GitHub Bot commented on BEAM-1461:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1993

[BEAM-1461] Make PrepareForProcessing Final, Deprecated

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This is only used by aggregators; Users should use @Setup or
@StartBundle instead.

Aggregators are set to be removed in 
[BEAM-775](https://issues.apache.org/jira/browse/BEAM-775)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam prepare_for_processing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1993


commit fda5f1a39a9214d7dc9cc6566c01715f367493b1
Author: Thomas Groh 
Date:   2017-02-13T17:03:32Z

Make PrepareForProcessing Final, Deprecated

This is only used by aggregators; Users should use @Setup or
@StartBundle instead.




> duplication with StartBundle and prepareForProcessing in DoFn
> -
>
> Key: BEAM-1461
> URL: https://issues.apache.org/jira/browse/BEAM-1461
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Xu Mingmin
>Assignee: Davor Bonaci
>
> There're one annotation `StartBundle`, and one public function 
> `prepareForProcessing` in DoFn, which are called both before 
> `ProcessElement`. It's confused which one should be implemented in a subclass.
> The call sequence seems as:
> prepareForProcessing -> StartBundle -> processElement



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1993: [BEAM-1461] Make PrepareForProcessing Final, Deprec...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/1993

[BEAM-1461] Make PrepareForProcessing Final, Deprecated

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
This is only used by aggregators; Users should use @Setup or
@StartBundle instead.

Aggregators are set to be removed in 
[BEAM-775](https://issues.apache.org/jira/browse/BEAM-775)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam prepare_for_processing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1993


commit fda5f1a39a9214d7dc9cc6566c01715f367493b1
Author: Thomas Groh 
Date:   2017-02-13T17:03:32Z

Make PrepareForProcessing Final, Deprecated

This is only used by aggregators; Users should use @Setup or
@StartBundle instead.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1040) Hadoop InputFormat - IO Transform for reads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864056#comment-15864056
 ] 

ASF GitHub Bot commented on BEAM-1040:
--

GitHub user diptikul opened a pull request:

https://github.com/apache/beam/pull/1994

 BEAM-1040 Hadoop InputFormat - IO Transform for reads

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/diptikul/incubator-beam HadoopInputFormatIO

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1994.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1994


commit 45cf382e14fceb32b2738b548c851472659f84d9
Author: Radhika S Kulkarni 
Date:   2016-12-22T13:49:38Z

First Hadoop InputFormat IO project code check in

commit 9c1a54d8d686ac396a1f500c9b69abe4d44e18a5
Author: Radhika S Kulkarni 
Date:   2016-12-23T08:15:07Z

Merge rtest from original

Take latest from original. remote-tracking branch 'upstream/master'

commit 5f927add640a9888ca26662450ceb1ecfa92c08c
Author: Sheetal Tarodekar 
Date:   2016-12-23T08:55:12Z

 erg end() method

 remote-tracking branch 'upstream/master'

commit 6c7ec6a37286ae55e16f267f126342694eb02b0e
Author: Dipti Kulkarni 
Date:   2016-12-23T09:43:48Z

Merge remote-tracking branch 'upstream/master'
taking latest-23/12

commit e49f22fed8fe0a8798fa1fe729c33df105e54cb5
Author: Radhika S Kulkarni 
Date:   2016-12-23T10:20:45Z

Updated Pom files

commit 9e4fdba32171b86b2f67f0c73780a06f6ca1b187
Author: Dipti Kulkarni 
Date:   2016-12-23T10:24:04Z

Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 6357bdf3c151c8c7575abad0f0e2f4bc4a424d7a
Author: Sheetal Tarodekar 
Date:   2016-12-23T14:41:23Z

Added code and tests to read from DummyInputFormat

commit a2163e823d7de3a024913acbb6a4035b8d8b6dd3
Author: Sheetal Tarodekar 
Date:   2016-12-23T14:48:38Z

 Resolved conflictsMerge branch 'master' of 
https://github.com/diptikul/incubator-beam

commit fb1f2de7e479920ac53de25ba18dd673046c69da
Author: Dipti Kulkarni 
Date:   2016-12-23T14:50:06Z

Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 571fcb40838a61148d0d49669ec18df35779b784
Author: Radhika S Kulkarni 
Date:   2016-12-26T05:01:37Z

Added Hadoop Input Format module in io pom file and minor changes in pom 
file.

commit a785180fc3b95f66a9b30a03c184fb7d63ad
Author: Dipti Kulkarni 
Date:   2016-12-26T05:30:25Z

Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit b29276ae953e9fff7ed6eb0c241d21e34122c5d2
Author: Radhika S Kulkarni 
Date:   2016-12-26T09:49:39Z

Moved MyCassandraRow to test folder

commit 70f05266328d62ba3a9127d54bc9f3769effbde4
Author: Radhika S Kulkarni 
Date:   2016-12-26T10:51:29Z

Removed CassandraRowCoder and Custom coder unit test, added MyCassandraRow 
in test, used Avro coder for MyCassandraRow

commit f129ead54b1b7047f4c5ff592bc4f2e7ecc7bc43
Author: Dipti Kulkarni 
Date:   2016-12-26T11:56:58Z

Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 276c3c217df6c062b8dd4b5deda0e3425c775513
Author: Sheetal Tarodekar 
Date:   2016-12-27T06:02:14Z

Added test cases for validate method, getFractionsConsumed method

commit 020576283b10c7630bb38df8836387404a57b76e
Author: Dipti Kulkarni 
Date:   2016-12-27T07:03:24Z

Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit c01544ffb61d6c129a932eef6d957e57c698b858
Author: Sheetal Tarodekar 
Date:   2016-12-28T10:46:54Z

[2/2] beam git commit: This closes #1992

[altay]

This closes #1992


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/982ea7af
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/982ea7af
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/982ea7af

Branch: refs/heads/master
Commit: 982ea7af7bd631fd74a97482b7de4f65f660cbdd
Parents: e2ef2d0 c5e5363
Author: Ahmet Altay 
Authored: Mon Feb 13 09:47:47 2017 -0800
Committer: Ahmet Altay 
Committed: Mon Feb 13 09:47:47 2017 -0800

--
 sdks/python/apache_beam/examples/complete/estimate_pi_test.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[GitHub] beam pull request #1986: Fixed javadoc comment in ResumeFromCheckpointStream...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1986


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fixed javadoc comment in ResumeFromCheckpointStreamingTest.

[staslevin]

Repository: beam
Updated Branches:
  refs/heads/master 93ae666be -> 4beed50f2


Fixed javadoc comment in ResumeFromCheckpointStreamingTest.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2f63b1f6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2f63b1f6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2f63b1f6

Branch: refs/heads/master
Commit: 2f63b1f63fefaff996c4ae267af8a01a5718e0c0
Parents: 93ae666
Author: Aviem Zur 
Authored: Sun Feb 12 19:56:48 2017 +0200
Committer: Stas Levin 
Committed: Mon Feb 13 12:40:23 2017 +0200

--
 .../streaming/ResumeFromCheckpointStreamingTest.java | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2f63b1f6/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
--
diff --git 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
index 7094c86..5a27b29 100644
--- 
a/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
+++ 
b/runners/spark/src/test/java/org/apache/beam/runners/spark/translation/streaming/ResumeFromCheckpointStreamingTest.java
@@ -64,12 +64,7 @@ import org.junit.rules.TemporaryFolder;
 
 
 /**
- * Tests DStream recovery from checkpoint - recreate the job and continue 
(from checkpoint).
- *
- * Tests Aggregators, which rely on Accumulators - Aggregators should be 
available, though
- * state is not preserved (Spark issue), so they start from initial value.
- * //TODO: after the runner supports recovering the state of Aggregators, 
update this test's
- * expected values for the recovered (second) run.
+ * Test pipelines which are resumed from checkpoint.
  */
 public class ResumeFromCheckpointStreamingTest {
   private static final EmbeddedKafkaCluster.EmbeddedZookeeper 
EMBEDDED_ZOOKEEPER =
@@ -118,6 +113,10 @@ public class ResumeFromCheckpointStreamingTest {
 }
   }
 
+  /**
+   * Tests DStream recovery from checkpoint - recreate the job and continue 
(from checkpoint).
+   * Also tests Aggregator values, which should be restored upon recovery 
from checkpoint.
+   */
   @Test
   public void testRun() throws Exception {
 Duration batchIntervalDuration = Duration.standardSeconds(5);

[2/2] beam git commit: This closes #1986

[staslevin]

This closes #1986


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4beed50f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4beed50f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4beed50f

Branch: refs/heads/master
Commit: 4beed50f27a745ec38c14ea4a4ed88c0a346f485
Parents: 93ae666 2f63b1f
Author: Stas Levin 
Authored: Mon Feb 13 12:45:54 2017 +0200
Committer: Stas Levin 
Committed: Mon Feb 13 12:45:54 2017 +0200

--
 .../streaming/ResumeFromCheckpointStreamingTest.java | 11 +--
 1 file changed, 5 insertions(+), 6 deletions(-)
--

[GitHub] beam pull request #1996: Add cross-sdk tests for IterableCoder

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/1996

Add cross-sdk tests for IterableCoder

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam common_iterable_coder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1996.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1996


commit 965160c0c0bada4500d51ca44c6c8e5e1c0476bf
Author: Vikas Kedigehalli 
Date:   2017-02-13T18:23:28Z

Add cross-sdk tests for IterableCoder




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1995

[tgroh]

This closes #1995


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2c0cffaf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2c0cffaf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2c0cffaf

Branch: refs/heads/master
Commit: 2c0cffaf7e55e2ca8b49368b360823b5735c5b10
Parents: 30cb93c 86f00db
Author: Thomas Groh 
Authored: Mon Feb 13 10:58:59 2017 -0800
Committer: Thomas Groh 
Committed: Mon Feb 13 10:58:59 2017 -0800

--
 .../direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java  | 9 +
 .../runners/direct/DirectGroupByKeyOverrideFactory.java | 9 +
 .../beam/runners/direct/ParDoMultiOverrideFactory.java  | 9 +
 .../runners/direct/ParDoSingleViaMultiOverrideFactory.java  | 9 +
 .../beam/runners/direct/TestStreamEvaluatorFactory.java | 9 +
 .../apache/beam/runners/direct/ViewEvaluatorFactory.java| 9 +
 .../beam/runners/direct/WriteWithShardingFactory.java   | 9 +
 .../apache/beam/sdk/runners/PTransformOverrideFactory.java  | 8 
 8 files changed, 71 insertions(+)
--