[1/2] incubator-beam-site git commit: re-generating to add incubation disclaimer

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site c74bcd0ed -> 5d7ef1d64


re-generating to add incubation disclaimer


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b3bca60e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b3bca60e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b3bca60e

Branch: refs/heads/asf-site
Commit: b3bca60efc8346d69bd98377c7c3ad658e76fabd
Parents: c74bcd0
Author: James Malone 
Authored: Tue Jul 12 08:22:08 2016 -0700
Committer: James Malone 
Committed: Tue Jul 12 08:22:08 2016 -0700

--
 content/capability-matrix/index.html |  2 +-
 content/feed.xml |  4 ++--
 content/index.html   | 12 +---
 3 files changed, 12 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b3bca60e/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index eee24a0..e1fae69 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -99,7 +99,7 @@
 
   
 Apache Beam Capability 
Matrix
-Last updated: 2016-06-20 14:17 
PDT
+Last updated: 2016-07-12 08:21 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b3bca60e/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 8dd4a36..63dc097 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -6,8 +6,8 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Mon, 20 Jun 2016 14:17:33 -0700
-Mon, 20 Jun 2016 14:17:33 -0700
+Tue, 12 Jul 2016 08:21:30 -0700
+Tue, 12 Jul 2016 08:21:30 -0700
 Jekyll v3.1.6
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b3bca60e/content/index.html
--
diff --git a/content/index.html b/content/index.html
index 08b8923..f8f0bcf 100644
--- a/content/index.html
+++ b/content/index.html
@@ -204,9 +204,15 @@ The Apache Beam project is in the process of 
bootstrapping. This includes the cr
   
 
 
-Apache Project
-Apache Beam is an http://www.apache.org";>Apache Software 
Foundation project,
-available under the Apache v2 license.
+Apache Incubation Disclaimer
+
+Apache Beam is an effort undergoing incubation at http://www.apache.org";>The Apache Software Foundation (ASF) 
+sponsored by the Apache Incubator PMC. Incubation is required of all newly 
accepted projects until a further review
+indicates that the infrastructure, communications, and decision making process 
have stabilized in a manner consistent 
+with other successful ASF projects. While incubation status is not necessarily 
a reflection of the completeness or
+stability of the code, it does indicate that the project has yet to be fully 
endorsed by the ASF.
+
+Apache Beam (incubating) is available under http://www.apache.org/licenses/LICENSE-2.0";>Apache License, version 
2.0.
 
   
 

[2/2] incubator-beam-site git commit: Regenerate to add ASF Incubator disclaimer

[jamesmalone]

Regenerate to add ASF Incubator disclaimer

This closes #27


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5d7ef1d6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5d7ef1d6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5d7ef1d6

Branch: refs/heads/asf-site
Commit: 5d7ef1d648daabad7a37e886586cee51f95501a1
Parents: c74bcd0 b3bca60
Author: James Malone 
Authored: Tue Jul 12 11:09:25 2016 -0700
Committer: James Malone 
Committed: Tue Jul 12 11:09:25 2016 -0700

--
 content/capability-matrix/index.html |  2 +-
 content/feed.xml |  4 ++--
 content/index.html   | 12 +---
 3 files changed, 12 insertions(+), 6 deletions(-)
--

[3/3] incubator-beam-site git commit: Branding fixes

[jamesmalone]

Branding fixes

This closes #26


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c74bcd0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c74bcd0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c74bcd0e

Branch: refs/heads/asf-site
Commit: c74bcd0ed3e524b2e3048f7ba2dae77262cfbbbc
Parents: 0d0ba1b ddeca42
Author: James Malone 
Authored: Thu Jun 30 09:40:35 2016 -0700
Committer: James Malone 
Committed: Thu Jun 30 09:40:35 2016 -0700

--
 index.md | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)
--

[2/3] incubator-beam-site git commit: fixed typo

[jamesmalone]

fixed typo

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ddeca427
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ddeca427
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ddeca427

Branch: refs/heads/asf-site
Commit: ddeca427e180f02df74da79346646bb4f17bd63c
Parents: 6277665
Author: Sergio Fernández 
Authored: Thu Jun 30 12:18:26 2016 +0200
Committer: GitHub 
Committed: Thu Jun 30 12:18:26 2016 +0200

--
 index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ddeca427/index.md
--
diff --git a/index.md b/index.md
index 68d640d..147c858 100644
--- a/index.md
+++ b/index.md
@@ -98,4 +98,4 @@ indicates that the infrastructure, communications, and 
decision making process h
 with other successful ASF projects. While incubation status is not necessarily 
a reflection of the completeness or
 stability of the code, it does indicate that the project has yet to be fully 
endorsed by the ASF.
 
-Apache Bean is available under [Apache License, version 
2.0](http://www.apache.org/licenses/LICENSE-2.0).
+Apache Beam (incubating) is available under [Apache License, version 
2.0](http://www.apache.org/licenses/LICENSE-2.0).

[1/3] incubator-beam-site git commit: added incubation disclaimer

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 0d0ba1b90 -> c74bcd0ed


added incubation disclaimer

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/6277665c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/6277665c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/6277665c

Branch: refs/heads/asf-site
Commit: 6277665c236fdbb347a32a3dce4d8b08a6bcaf62
Parents: 0d0ba1b
Author: Sergio Fernández 
Authored: Wed Jun 29 14:19:45 2016 +0200
Committer: GitHub 
Committed: Wed Jun 29 14:19:45 2016 +0200

--
 index.md | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/6277665c/index.md
--
diff --git a/index.md b/index.md
index db5d6f0..68d640d 100644
--- a/index.md
+++ b/index.md
@@ -90,6 +90,12 @@ Interested in working with Apache Beam? Great! Here's how to 
get started:
   
 
 
-## Apache Project
-Apache Beam is an [Apache Software Foundation project](http://www.apache.org),
-available under the Apache v2 license.
+## Apache Incubation Disclaimer
+
+Apache Beam is an effort undergoing incubation at [The Apache Software 
Foundation (ASF)](http://www.apache.org) 
+sponsored by the Apache Incubator PMC. Incubation is required of all newly 
accepted projects until a further review
+indicates that the infrastructure, communications, and decision making process 
have stabilized in a manner consistent 
+with other successful ASF projects. While incubation status is not necessarily 
a reflection of the completeness or
+stability of the code, it does indicate that the project has yet to be fully 
endorsed by the ASF.
+
+Apache Bean is available under [Apache License, version 
2.0](http://www.apache.org/licenses/LICENSE-2.0).

[3/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index 06992fd..eee24a0 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -99,7 +99,7 @@
 
   
 Apache Beam Capability 
Matrix
-Last updated: 2016-06-14 18:36 
PDT
+Last updated: 2016-06-20 14:17 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 
@@ -157,19 +157,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -178,19 +182,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+~
+
 
   
   
@@ -199,19 +207,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -220,19 +232,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -241,19 +257,23 @@
 
 
 
-
+✓
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
   
   
@@ -262,19 +282,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+~ (https://issues.apache.org/jira/browse/BEAM-102";>BEAM-102)
+
 
 
 
-
+~
+
 
   
   
@@ -283,19 +307,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -304,19 +332,23 @@
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
 
 
-
+~
+
 
   
   
@@ -325,19 +357,23 @@
 
 
 
-
+✕ (https://issues.apache.org/jira/browse/BEAM-25";>BEAM-25)
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
   
   
@@ -366,19 +402,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -387,19 +427,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+~
+
 
   
   
@@ -408,19 +452,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+~
+
 
   
   
@@ -429,19 +477,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -450,19 +502,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -471,19 +527,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -492,19 +552,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -533,19 +597,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -554,19 +622,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -575,19 +647,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
   
   
@@ -596,19 +672,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -617,19 +697,23 @@
 
 
 
-
+✕ (https://issues.apache.org/jira/browse/BEAM-101";>BEAM-101)
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
   
   
@@ -638,19 +722,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -659,19 +747,23 @@
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✓
+
 
 
 
-
+✕
+
 
   
   
@@ -680,19 +772,23 @@
 
 
 
-
+✕ (https://issues.apache.org/jira/browse/BEAM-27";>BEAM-27)
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
 
 
-
+✕
+
 
   
   
@@ -721,19 +817,23 @@
 
 
 
-
+✓
+
 
 
 
-

[2/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index fcfa7b2..8dd4a36 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -6,9 +6,9 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Tue, 14 Jun 2016 18:36:16 -0700
-Tue, 14 Jun 2016 18:36:16 -0700
-Jekyll v2.5.0
+Mon, 20 Jun 2016 14:17:33 -0700
+Mon, 20 Jun 2016 14:17:33 -0700
+Jekyll v3.1.6
 
   
 The first release of Apache Beam!
@@ -59,20 +59,20 @@ to us via user’s mailing list
 
-
Before we start, though, let’s quickly talk about the execution of 
Beam programs and how this is relevant to today’s post. A Beam pipeline can 
contain bounded and unbounded sources. If the pipeline only contains bounded 
sources it can be executed in a batch fashion, if it contains some unbounded 
sources it must be executed in a streaming fashion. When executing a Beam 
pipeline on Flink, you don’t have to choose the execution mode. Internally, 
the Flink runner either translates the pipeline to a Flink 
DataSet program or a 
DataStream program, depending on whether unbounded 
sources are used in the pipeline. In the following, when we say “Batch 
runner” what we are really talking about is the Flink runner being in batch 
execution mode.
+
Before we start, though, let’s quickly talk about the execution of 
Beam programs and how this is relevant to today’s post. A Beam pipeline can 
contain bounded and unbounded sources. If the pipeline only contains bounded 
sources it can be executed in a batch fashion, if it contains some unbounded 
sources it must be executed in a streaming fashion. When executing a Beam 
pipeline on Flink, you don’t have to choose the execution mode. Internally, 
the Flink runner either translates the pipeline to a Flink DataSet program or a 
DataStream 
program, depending on whether unbounded sources are used in the pipeline. In 
the following, when we say “Batch runner” what we are really talking about 
is the Flink runner being in batch execution mode.
 
 
What does this mean for 
users?
 
 
Support for windowing was the last missing puzzle piece for making 
the Flink Batch runner compatible with the Beam model. With the latest change 
to the Batch runner users can now run any pipeline that only contains bounded 
sources and be certain that the results match those of the original 
reference-implementation runners that were provided by Google as part of the 
initial code drop coming from the Google Dataflow SDK.
 
-
The most obvious part of the change is that windows can now be 
assigned to elements and that the runner respects these windows for the 
GroupByKey and Combine 
operations. A not-so-obvious change concerns side-inputs. In the Beam model, 
side inputs respect windows; when a value of the main input is being processed 
only the side input that corresponds to the correct window is available to the 
processing function, the DoFn.
+
The most obvious part of the change is that windows can now be 
assigned to elements and that the runner respects these windows for the 
GroupByKey and 
Combine 
operations. A not-so-obvious change concerns side-inputs. In the Beam model, 
side inputs respect windows; when a value of the main input is being processed 
only the side input that corresponds to the correct window is available to the 
processing function, the DoFn.
 
 
Getting side-input semantics right is an important milestone in 
it’s own because it allows to use a big suite of unit tests for verifying the 
correctness of a runner implementation. These tests exercise every obscure 
detail of the Beam programming model and verify that the results produced by a 
runner match what you would expect from a correct implementation. In the suite, 
side inputs are used to compare the expected result to the actual result. With 
these tests being executed regularly we can now be more confident that the 
implementation produces correct results for user-specified pipelines.
 
 
Under the Hood
-
The basis for the changes is the introduction of 
WindowedValue in the generated Flink transformations. 
Before, a Beam PCollection<T> would be 
transformed to a DataSet<T>. Now, we 
instead create a 
DataSet<WindowedValue<T>>. 
The WindowedValue<T> stores meta

[5/5] incubator-beam-site git commit: Cleanup and tests

[jamesmalone]

Cleanup and tests

This closes #23


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0d0ba1b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0d0ba1b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0d0ba1b9

Branch: refs/heads/asf-site
Commit: 0d0ba1b90bb870f7939864a4cbf3cdcdd18b670a
Parents: 2c0a36d 2a61d38
Author: James Malone 
Authored: Mon Jun 20 15:13:33 2016 -0700
Committer: James Malone 
Committed: Mon Jun 20 15:13:33 2016 -0700

--
 .gitignore  |   1 +
 Gemfile |   3 +
 Rakefile|  10 +
 _config.yml |   2 +-
 _includes/capability-matrix-row-blog.md |   2 +-
 _includes/capability-matrix-row-full.md |   2 +-
 _includes/capability-matrix-row-summary.md  |   2 +-
 _pages/blog.md  |   2 +-
 _pages/material.md  |   4 +-
 _pages/public-meetings.md   |   2 +-
 _posts/2016-02-22-beam-has-a-logo.markdown  |   4 +-
 coming-soon.md  |  13 +-
 .../2016/03/17/capability-matrix.html   | 324 ++---
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   4 +-
 .../blog/2016/05/18/splitAtFraction-method.html |   2 +-
 .../05/27/where-is-my-pcollection-dot-map.html  |  12 +-
 .../06/13/flink-batch-runner-milestone.html |   8 +-
 content/blog/index.html |  19 +-
 content/capability-matrix/index.html| 650 ---
 content/coming-soon.html|  13 +-
 content/contribution-guide/index.html   |  45 +-
 content/docs/index.html |  12 +-
 content/feed.xml| 360 ++
 content/index.html  |  13 +-
 content/material/index.html |  36 +-
 content/presentation-materials/index.html   |  10 +-
 content/public-meetings/index.html  |   4 +-
 content/releases/index.html |   7 +-
 content/source_repository/index.html|   9 +-
 docs/index.md   |  12 +-
 index.md|  13 +-
 32 files changed, 1024 insertions(+), 578 deletions(-)
--

[1/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 2c0a36d70 -> 0d0ba1b90


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/releases/index.html
--
diff --git a/content/releases/index.html b/content/releases/index.html
index f438d41..0a936cf 100644
--- a/content/releases/index.html
+++ b/content/releases/index.html
@@ -104,10 +104,10 @@
 http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22";>Maven
 Central Repository.
 
 For example, if you are developing using Maven and want to use the SDK for
-Java with the DirectRunner, add the following dependencies to your
-pom.xml file:
+Java with the DirectRunner, add the 
following dependencies to your
+pom.xml file:
 
-
+
   org.apache.beam
   beam-sdks-java-core
   0.1.0-incubating
@@ -119,6 +119,7 @@ Java with the DirectRunner, add the following 
dependencies to your
   runtime
 
 
+
 
 Additionally, you may want to depend on additional SDK modules, such as IO
 connectors or other extensions, and additional runners to execute your pipeline

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/content/source_repository/index.html
--
diff --git a/content/source_repository/index.html 
b/content/source_repository/index.html
index 7bcf6e6..18603e9 100644
--- a/content/source_repository/index.html
+++ b/content/source_repository/index.html
@@ -110,20 +110,23 @@
 Web Browser Access
 The following is a link to a browsable version of the source repository:
 
-https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Anonymous Access
 The source can be checked out anonymously from Git with this command (See 
http://git-scm.com/docs/git-clone):
 
-$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Developer Access
 Only project developers can access the Git tree via this method (See 
http://git-scm.com/docs/git-clone).
 
-$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
+$ git clone 
https://git-wip-us.apache.org/repos/asf/incubator-beam.git
 
+
 
 Access from Behind a Firewall
 Refer to the documentation of the SCM used for more information about 
access behind a firewall.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/docs/index.md
--
diff --git a/docs/index.md b/docs/index.md
index 8ca1550..e825ed3 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,13 +1,11 @@
 ---
 layout: default
 ---
-
-  
-  
-  ×
-  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
-  
-
+
+
+×
+The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
+
 
 # Apache Beam Documentation
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/index.md
--
diff --git a/index.md b/index.md
index cee6606..db5d6f0 100644
--- a/index.md
+++ b/index.md
@@ -1,13 +1,11 @@
 ---
 layout: default
 ---
-
-  
-  
-  ×
-  The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. For more information about Beam see the getting started page.
-  
-
+
+
+×
+The Apache Beam project is in the process of bootstrapping. This includes the 
creation of project resources, the refactoring of the initial code submission, 
and the formulation of project documentation, planning, and design documents. 
For more information about Beam see the getting 
started page.
+
 
 # Apache Beam (incubating)
 
@@ -95,4 +93,3 @@ Interested in working with Apache Beam? Great! Here's how to 
get started:
 ## Apache Project
 Apache Beam is an [Apache Software Foundation project](http://www.apache.org),
 available under the Apache v2 license.
-

[4/5] incubator-beam-site git commit: Fixed HTML errors; added link and HTML tests via rake

[jamesmalone]

-{% if val.l1 == 'Yes' %}✓{% elsif val.l1 == 'Partially' 
%}~{% else %}✕{% endif %}
+{% if val.l1 == 'Yes' %}✓{% elsif val.l1 == 'Partially' 
%}~{% else %}✕{% endif %}

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_includes/capability-matrix-row-full.md
--
diff --git a/_includes/capability-matrix-row-full.md 
b/_includes/capability-matrix-row-full.md
index 3734a98..9fa2d64 100644
--- a/_includes/capability-matrix-row-full.md
+++ b/_includes/capability-matrix-row-full.md
@@ -1 +1 @@
-{{ val.l1 }}{% if val.l2 != '' %}: {{ val.l2 }}{% endif %}{% if 
val.jira %}({{ val.jira }}){% endif %}{{ val.l3 }}
+{{ val.l1 }}{% if val.l2 != '' %}: {{ val.l2 }}{% endif %}{% if 
val.jira %}({{ val.jira }}){% endif %}{{ val.l3 }}

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_includes/capability-matrix-row-summary.md
--
diff --git a/_includes/capability-matrix-row-summary.md 
b/_includes/capability-matrix-row-summary.md
index 922802a..e1cc162 100644
--- a/_includes/capability-matrix-row-summary.md
+++ b/_includes/capability-matrix-row-summary.md
@@ -1 +1 @@
-{% if val.l1 == 'Yes' %}✓{% elsif val.l1 == 'Partially' 
%}~{% else %}✕{% endif %}{% if val.jira %} ({{ val.jira 
}}){% endif %}
+{% if val.l1 == 'Yes' %}✓{% elsif val.l1 == 'Partially' 
%}~{% else %}✕{% endif %}{% if val.jira %} ({{ val.jira 
}}){% endif %}

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_pages/blog.md
--
diff --git a/_pages/blog.md b/_pages/blog.md
index 6a1f98b..36ada4a 100644
--- a/_pages/blog.md
+++ b/_pages/blog.md
@@ -24,7 +24,7 @@ for the project.
 {% endcapture %}
 {% if excerpt_words != content_words %}
 
-
+
 Read more 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_pages/material.md
--
diff --git a/_pages/material.md b/_pages/material.md
index 23bc808..a372ce5 100644
--- a/_pages/material.md
+++ b/_pages/material.md
@@ -23,7 +23,7 @@ download it.
 {% for type in site.data.logos.types %}
 
 
-
+
 
 
 {% endfor %}
@@ -43,7 +43,7 @@ available in a number of fixed sizes and are optimized for 
web use.
 {% for type in site.data.logos.types %}
 
 
-
+
 
 
 {% for size in site.data.logos.sizes %}

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_pages/public-meetings.md
--
diff --git a/_pages/public-meetings.md b/_pages/public-meetings.md
index ead8c07..2b2a1dd 100644
--- a/_pages/public-meetings.md
+++ b/_pages/public-meetings.md
@@ -52,4 +52,4 @@ We are investigating methods, such as video conferencing and 
IRC chat, to allow
 Public meetings include scheduled Apache Beam Dev/PPMC meetings, Meetup 
events, conference talks, and other events where the public meets to discuss 
Beam.
 
 ### How do I learn about new meetings?
-The Apache Beam community announces upcoming public meetings on the  
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org) mailing 
list. If you want to learn about new events, we recommend you 
[subscribe](dev-subscr...@beam.incubator.apache.org) to that list.  If you are 
holding a public event, please send an email to the dev@ list.
+The Apache Beam community announces upcoming public meetings on the  
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org) mailing 
list. If you want to learn about new events, we recommend you 
[subscribe](mailto:dev-subscr...@beam.incubator.apache.org) to that list.  If 
you are holding a public event, please send an email to the dev@ list.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/_posts/2016-02-22-beam-has-a-logo.markdown
--
diff --git a/_posts/2016-02-22-beam-has-a-logo.markdown 
b/_posts/2016-02-22-beam-has-a-logo.markdown
index f643a36..4ee569f 100644
--- a/_posts/2016-02-22-beam-has-a-logo.markdown
+++ b/_posts/2016-02-22-beam-has-a-logo.markdown
@@ -4,7 +4,7 @@ title:  "Apache Beam has a logo!"
 date:   2016-02-22 10:21:48 -0800
 excerpt_separator: 
 categories: beam update website
-authors: 
+authors:
 - jamesmalone
 ---
 
@@ -18,7 +18,7 @@ now has a logo.
 
 *drum roll* - **Presenting, the Apache Beam Logo!**
 
-
+
 
 We are excited about this logo because it is **simple**, **bright**, and shows 
the
 unification of bath and streaming, as beams of light, within the 'B'. We will 
base

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/2a61d388/coming-soon.md
--
diff --git a/coming

[1/4] incubator-beam-site git commit: Update date for blog post

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 657701378 -> 87d9a07fe


Update date for blog post


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0a1d6c8c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0a1d6c8c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0a1d6c8c

Branch: refs/heads/asf-site
Commit: 0a1d6c8c7f97e45dd9866341e99f11f30c30ad21
Parents: 49ae0d9
Author: James Malone 
Authored: Fri May 27 09:51:26 2016 -0700
Committer: James Malone 
Committed: Fri May 27 10:03:00 2016 -0700

--
 ...016-05-20-where-is-my-pcollection-dot-map.md |   4 +-
 .../05/20/where-is-my-pcollection-dot-map.html  | 212 --
 .../05/27/where-is-my-pcollection-dot-map.html  | 214 +++
 content/blog/index.html |   6 +-
 content/capability-matrix/index.html|   4 +
 content/feed.xml|  17 +-
 content/index.html  |   2 +-
 7 files changed, 237 insertions(+), 222 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a1d6c8c/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
--
diff --git a/_posts/2016-05-20-where-is-my-pcollection-dot-map.md 
b/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
index 3114164..f7ea468 100644
--- a/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
+++ b/_posts/2016-05-20-where-is-my-pcollection-dot-map.md
@@ -1,14 +1,16 @@
 ---
 layout: post
 title:  "Where's my PCollection.map()?"
-date:   2016-05-20 11:00:00 -0700
+date:   2016-05-27 09:00:00 -0700
 excerpt_separator: 
 categories: blog
 authors:
   - robertwb
 ---
 Have you ever wondered why Beam has PTransforms for everything instead of 
having methods on PCollection? Take a look at the history that led to this (and 
other) design decisions.
+
 
+
 Though Beam is relatively new, its design draws heavily on many years of 
experience with real-world pipelines. One of the primary inspirations is 
[FlumeJava](http://research.google.com/pubs/pub35650.html), which is Google's 
internal successor to MapReduce first introduced in 2009.
 
 The original FlumeJava API has methods like `count` and `parallelDo` on the 
PCollections. Though slightly more succinct, this approach has many 
disadvantages to extensibility. Every new user to FlumeJava wanted to add 
transforms, and adding them as methods to PCollection simply doesn't scale 
well. In contrast, a PCollection in Beam has a single `apply` method which 
takes any PTransform as an argument.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a1d6c8c/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
--
diff --git a/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html 
b/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
deleted file mode 100644
index a45cb9c..000
--- a/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
+++ /dev/null
@@ -1,212 +0,0 @@
-
-
-
-  
-  
-  
-  
-
-  Where's my PCollection.map()?
-  
-
-  
-  
-  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
-  
-  http://beam.incubator.apache.org/blog/2016/05/20/where-is-my-pcollection-dot-map.html";>
-  http://beam.incubator.apache.org/feed.xml";>
-  
-
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
-(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
-
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
-
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
-ga('create', 'UA-73650088-1', 'auto');
-ga('send', 'pageview');
-
-  
-  
-
-
-
-  
-
-
-  
-
-  
-
-  
-  
-Toggle navigation
-
-
-
-  
-
-
-  
-
-  Documentation 
-  
-Guides
-Getting Started
-
-Technical Documentation
-Capability Matrix
-https://goo.gl/ps8twC";>Technical Docs
-https://goo.gl/nk5OM0";>Technical Vision
-  
-
-
-  Community 
-  
-Community
-Mailing Lists
-Apache Beam Team
-Public Meetings
-
-Contribute
-Contribution Guide
-Source Repository
-Issue Tracking
-  
-
-Blog
-
-  Project 
-  
-Presentation 
Materials
-Logos and design
-http://apache.org/licenses

[2/4] incubator-beam-site git commit: Resolve merge,2

[jamesmalone]

Resolve merge,2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/49ae0d94
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/49ae0d94
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/49ae0d94

Branch: refs/heads/asf-site
Commit: 49ae0d9493fe2f58a2db35c97e781073cc1e15f5
Parents: b61cca9
Author: James Malone 
Authored: Fri May 27 09:58:43 2016 -0700
Committer: James Malone 
Committed: Fri May 27 10:03:00 2016 -0700

--
 content/capability-matrix/index.html | 6 +-
 content/feed.xml | 4 ++--
 2 files changed, 3 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/49ae0d94/content/capability-matrix/index.html
--
diff --git a/content/capability-matrix/index.html 
b/content/capability-matrix/index.html
index 752e6b8..2d5f7d3 100644
--- a/content/capability-matrix/index.html
+++ b/content/capability-matrix/index.html
@@ -95,11 +95,7 @@
 
   
 Apache Beam Capability 
Matrix
-<<< 657701378fc1935a67e5b5a5fe1427997b6c9029
-Last updated: 2016-05-27 18:38 
CEST
-===
-Last updated: 2016-05-20 14:42 
PDT
->>> Regenerate to post PCollection blog post
+Last updated: 2016-05-27 09:58 
PDT
 
 Apache Beam (incubating) provides a portable API layer for building 
sophisticated data-parallel processing engines that may be executed across a 
diversity of exeuction engines, or runners. The core concepts of this 
layer are based upon the Beam Model (formerly referred to as the http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf";>Dataflow Model), and 
implemented to varying degrees in each Beam runner. To help clarify the 
capabilities of individual runners, we’ve created the capability matrix 
below.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/49ae0d94/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 0208117..b85a3e2 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -6,8 +6,8 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Fri, 20 May 2016 14:42:30 -0700
-Fri, 20 May 2016 14:42:30 -0700
+Fri, 27 May 2016 09:58:31 -0700
+Fri, 27 May 2016 09:58:31 -0700
 Jekyll v3.1.3
 
   

[4/4] incubator-beam-site git commit: New Beam blog post

[jamesmalone]

New Beam blog post

This closes #18


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/87d9a07f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/87d9a07f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/87d9a07f

Branch: refs/heads/asf-site
Commit: 87d9a07fe9bd3e8c23f69d50256d8da1fab486f0
Parents: 6577013 0a1d6c8
Author: James Malone 
Authored: Fri May 27 10:05:36 2016 -0700
Committer: James Malone 
Committed: Fri May 27 10:05:36 2016 -0700

--
 ...016-05-20-where-is-my-pcollection-dot-map.md |   4 +-
 .../05/27/where-is-my-pcollection-dot-map.html  | 214 +++
 content/blog/index.html |  16 ++
 content/capability-matrix/index.html|   6 +-
 content/feed.xml| 103 -
 content/index.html  |   2 +
 6 files changed, 341 insertions(+), 4 deletions(-)
--

[3/4] incubator-beam-site git commit: Resolve merge

[jamesmalone]

Resolve merge


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/b61cca94
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/b61cca94
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/b61cca94

Branch: refs/heads/asf-site
Commit: b61cca94a607e86ba0eec8c0dfa5ead35f9318ea
Parents: 15dd578
Author: James Malone 
Authored: Fri May 27 09:54:23 2016 -0700
Committer: James Malone 
Committed: Fri May 27 10:03:00 2016 -0700

--
 .../05/20/where-is-my-pcollection-dot-map.html  | 212 +++
 content/blog/index.html |  16 ++
 content/capability-matrix/index.html|   4 +
 content/feed.xml|  96 -
 content/index.html  |   2 +
 5 files changed, 328 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/b61cca94/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
--
diff --git a/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html 
b/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
new file mode 100644
index 000..a45cb9c
--- /dev/null
+++ b/content/blog/2016/05/20/where-is-my-pcollection-dot-map.html
@@ -0,0 +1,212 @@
+
+
+
+  
+  
+  
+  
+
+  Where's my PCollection.map()?
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/blog/2016/05/20/where-is-my-pcollection-dot-map.html";>
+  http://beam.incubator.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+  
+Toggle navigation
+
+
+
+  
+
+
+  
+
+  Documentation 
+  
+Guides
+Getting Started
+
+Technical Documentation
+Capability Matrix
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+  
+
+
+  Community 
+  
+Community
+Mailing Lists
+Apache Beam Team
+Public Meetings
+
+Contribute
+Contribution Guide
+Source Repository
+Issue Tracking
+  
+
+Blog
+
+  Project 
+  
+Presentation 
Materials
+Logos and design
+http://apache.org/licenses/LICENSE-2.0.html";>License
+  
+
+  
+
+  
+
+
+
+
+
+
+
+
+  
+
+
+http://schema.org/BlogPosting";>
+
+  
+Where's my 
PCollection.map()?
+May 20, 2016 •  Robert Bradshaw 
+
+  
+
+  
+Have you ever wondered why Beam has PTransforms for everything instead 
of having methods on PCollection? Take a look at the history that led to this 
(and other) design decisions.
+
+Though Beam is relatively new, its design draws heavily on many years of 
experience with real-world pipelines. One of the primary inspirations is http://research.google.com/pubs/pub35650.html";>FlumeJava, which is 
Google’s internal successor to MapReduce first introduced in 2009.
+
+The original FlumeJava API has methods like count and parallelDo on the PCollections. Though 
slightly more succinct, this approach has many disadvantages to extensibility. 
Every new user to FlumeJava wanted to add transforms, and adding them as 
methods to PCollection simply doesn’t scale well. In contrast, a PCollection 
in Beam has a single apply method which 
takes any PTransform as an argument.
+
+Have you ever wondered why Beam has PTransforms for everything instead of 
having methods on PCollection? Take a look at the history that led to this (and 
other) design decisions.
+
+
+  
+FlumeJava
+Beam
+  
+  
+
+PCollection input = …
+PCollection output = input.count()
+ .parallelDo(...);
+
+
+PCollection input = …
+PCollection output = input.apply(Count.perElement())
+ .apply(ParDo.of(...));
+
+  
+
+
+This is a more scalable approach for several reasons.
+
+Where to draw the line?
+Adding methods to PCollection forces a line to be drawn between operations 
that are “useful” en

[1/3] incubator-beam-site git commit: Addition of PCollection blog post; blog fix

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site d6f8ca78e -> 92fc27503


Addition of PCollection blog post; blog fix

Adding links to 5/4 meeting docs


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/697f4388
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/697f4388
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/697f4388

Branch: refs/heads/asf-site
Commit: 697f4388c82bd63e2e6f32b1fb4c6f910e1146e6
Parents: d6f8ca7
Author: James Malone 
Authored: Fri May 13 16:39:10 2016 -0700
Committer: James Malone 
Committed: Wed May 18 21:01:16 2016 -0700

--
 _data/authors.yml   |   4 +
 _data/meetings.yml  |   9 +-
 _includes/authors-list.md   |   2 +-
 ...016-05-13-where-is-my-pcollection-dot-map.md |  89 
 .../2016/03/17/capability-matrix.html   |   2 +-
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../05/13/where-is-my-pcollection-dot-map.html  | 212 +++
 .../sdk/2016/02/25/python-sdk-now-public.html   |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   2 +-
 content/blog/index.html |  16 ++
 content/capability-matrix/index.html|   2 +-
 content/contribution-guide/index.html   |   4 +-
 content/feed.xml| 110 +-
 .../images/blog/simple-wordcount-pipeline.png   | Bin 0 -> 322878 bytes
 content/index.html  |   2 +
 content/public-meetings/index.html  |   8 +-
 images/blog/simple-wordcount-pipeline.png   | Bin 0 -> 322878 bytes
 17 files changed, 447 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_data/authors.yml
--
diff --git a/_data/authors.yml b/_data/authors.yml
index b71ad8a..ce732c7 100644
--- a/_data/authors.yml
+++ b/_data/authors.yml
@@ -10,3 +10,7 @@ takidau:
 name: Tyler Akidau
 email: taki...@apache.org
 twitter: takidau
+robertwb:
+name: Robert Bradshaw
+email: rober...@apache.org
+twitter:

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_data/meetings.yml
--
diff --git a/_data/meetings.yml b/_data/meetings.yml
index c12111c..f2efe9f 100644
--- a/_data/meetings.yml
+++ b/_data/meetings.yml
@@ -16,6 +16,11 @@ events:
   location: Virtual
   type: Technical Deep Dive
   materials:
-  notes: Virtual attendance options willbe announced prior to the meeting.
+- title: Presentation - Beam Community Meeting
+  link: 
"https://drive.google.com/open?id=17i7SHViboWtLEZw27iabdMisPl987WWxvapJaXg_dEE";
 
-last_updated: 2016/04/14
+- title: Notes - Beam Community Meeting
+  link: 
"https://drive.google.com/open?id=1szhEE_pfhEtrQye61jXAidUcMW7oebZCRc2InUe3ou0";
+  notes:
+
+last_updated: 2016/05/16

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_includes/authors-list.md
--
diff --git a/_includes/authors-list.md b/_includes/authors-list.md
index 1207445..fb2808e 100644
--- a/_includes/authors-list.md
+++ b/_includes/authors-list.md
@@ -1 +1 @@
-{% assign count = authors | size %}{% for name in authors %}{% if 
forloop.first == false and count > 2 %},{% endif %}{% if forloop.last and count 
> 1 %} &{% endif %}{% assign author = site.data.authors[name] %} {{ 
author.name }} [https://twitter.com/{{ author.twitter }}">@{{ 
author.twitter }}]{% endfor %}
+{% assign count = authors | size %}{% for name in authors %}{% if 
forloop.first == false and count > 2 %},{% endif %}{% if forloop.last and count 
> 1 %} &{% endif %}{% assign author = site.data.authors[name] %} {{ 
author.name }} {% if author.twitter %}[https://twitter.com/{{ 
author.twitter }}">@{{ author.twitter }}]{% endif %}{% endfor %}

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/697f4388/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
--
diff --git a/_posts/2016-05-13-where-is-my-pcollection-dot-map.md 
b/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
new file mode 100644
index 000..3193ea1
--- /dev/null
+++ b/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
@@ -0,0 +1,89 @@
+---
+layout: post
+title:  "Where's my PCollection.map()?"
+date:   2016-05-13 11:00:00 -0700
+excerpt_separator: 
+categories: beam model pcollection
+authors:
+  - robertwb
+---
+Have you ever wondered why Beam has PTransforms for everything instead of 
having methods on PCollection

[3/3] incubator-beam-site git commit: New Beam blog posts

[jamesmalone]

New Beam blog posts

This closes #16


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/92fc2750
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/92fc2750
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/92fc2750

Branch: refs/heads/asf-site
Commit: 92fc27503daca58b6484649323cfa773ce9f1c74
Parents: d6f8ca7 568f051
Author: James Malone 
Authored: Wed May 18 21:06:49 2016 -0700
Committer: James Malone 
Committed: Wed May 18 21:06:49 2016 -0700

--
 _data/authors.yml   |   8 ++
 _data/meetings.yml  |   9 +-
 _includes/authors-list.md   |   2 +-
 _posts/2016-05-18-splitAtFraction-method.md |  17 +++
 ...016-05-20-where-is-my-pcollection-dot-map.md |  89 
 .../2016/03/17/capability-matrix.html   |   2 +-
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../sdk/2016/02/25/python-sdk-now-public.html   |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   2 +-
 .../blog/2016/05/18/splitAtFraction-method.html | 139 +++
 content/blog/index.html |  16 +++
 content/capability-matrix/index.html|   2 +-
 content/feed.xml|  31 -
 .../images/blog/simple-wordcount-pipeline.png   | Bin 0 -> 322878 bytes
 content/index.html  |   2 +
 content/public-meetings/index.html  |   8 +-
 images/blog/simple-wordcount-pipeline.png   | Bin 0 -> 322878 bytes
 17 files changed, 315 insertions(+), 16 deletions(-)
--

[2/3] incubator-beam-site git commit: Addition of sAF blog; delay of PCollection post.

[jamesmalone]

Addition of sAF blog; delay of PCollection post.

sAF blog fixes

Title fix


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/568f051a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/568f051a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/568f051a

Branch: refs/heads/asf-site
Commit: 568f051a6f59ffc7e4dc5d300c75b26ec03a78aa
Parents: 697f438
Author: James Malone 
Authored: Wed May 18 20:36:51 2016 -0700
Committer: James Malone 
Committed: Wed May 18 21:01:24 2016 -0700

--
 _data/authors.yml   |   4 +
 ...016-05-13-where-is-my-pcollection-dot-map.md |  89 
 _posts/2016-05-18-splitAtFraction-method.md |  17 ++
 ...016-05-20-where-is-my-pcollection-dot-map.md |  89 
 .../05/13/where-is-my-pcollection-dot-map.html  | 212 ---
 .../blog/2016/05/18/splitAtFraction-method.html | 139 
 content/blog/index.html |   8 +-
 content/capability-matrix/index.html|   2 +-
 content/contribution-guide/index.html   |   4 +-
 content/feed.xml|  99 +
 content/index.html  |   2 +-
 11 files changed, 268 insertions(+), 397 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/568f051a/_data/authors.yml
--
diff --git a/_data/authors.yml b/_data/authors.yml
index ce732c7..930a514 100644
--- a/_data/authors.yml
+++ b/_data/authors.yml
@@ -14,3 +14,7 @@ robertwb:
 name: Robert Bradshaw
 email: rober...@apache.org
 twitter:
+dhalperi:
+name: Dan Halperin
+email: dhalp...@apache.org
+twitter:

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/568f051a/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
--
diff --git a/_posts/2016-05-13-where-is-my-pcollection-dot-map.md 
b/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
deleted file mode 100644
index 3193ea1..000
--- a/_posts/2016-05-13-where-is-my-pcollection-dot-map.md
+++ /dev/null
@@ -1,89 +0,0 @@

-layout: post
-title:  "Where's my PCollection.map()?"
-date:   2016-05-13 11:00:00 -0700
-excerpt_separator: 
-categories: beam model pcollection
-authors:
-  - robertwb

-Have you ever wondered why Beam has PTransforms for everything instead of 
having methods on PCollection? Take a look at the history that led to this (and 
other) design decisions.
-
-
-
-Though Beam is relatively new, its design draws heavily on many years of 
experience with real-world pipelines. One of the primary inspirations is 
FlumeJava, which is Google's internal successor to MapReduce first introduced 
in 2009.
-
-The original FlumeJava API has methods like `count` and `parallelDo` on the 
`PCollection`s. Though slightly more succinct, this approach has many 
disadvantages to extensibility. Every new user to FlumeJava wanted to add 
transforms, and adding them as methods to `PCollection` simply doesn't scale 
well. In contrast, a PCollection in Beam has a single `apply` method which 
takes any PTransform as an argument.
-
-
-  
-FlumeJava
-Beam
-  
-  
-
-PCollection input = …
-PCollection output = input.count()
- .parallelDo(...);
-
-
-PCollection input = …
-PCollection output = input.apply(Count.perElement())
- .apply(ParDo.of(...));
-
-  
-
-
-This is a more scalable approach for several reasons.
-
-## Where to draw the line?
-Adding methods to `PCollection` forces a line to be drawn between operations 
that are "useful" enough to merit this special treatment and those that are 
not. It is easy to make the case for flat map, group by key, and combine per 
key. But what about filter? Count? Approximate count? Approximate quantiles? 
Most frequent? WriteToMyFavoriteSource? Going too far down this path leads to a 
single enormous class that contains nearly everything one could want to do. 
(FlumeJava's PCollection class is over 5000 lines long with around 70 distinct 
operations, and it could have been *much* larger had we accepted every 
proposal.) Furthermore, since Java doesn’t allow adding methods to a class, 
there is a sharp divide syntactically between those operations that are added 
to `PCollection` and those that aren’t. A traditional way to share code is 
with a library of functions, but functions (in traditional languages like Java 
at least) are written prefix-style, which doesn't mix well with the flue
 nt builder style (e.g. `input.operation1().operation2().operation3()` vs. 
`operation3(operation1(input).operation2())`).

[2/2] incubator-beam-site git commit: Nav fix

[jamesmalone]

Nav fix

This closes #15


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d6f8ca78
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d6f8ca78
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d6f8ca78

Branch: refs/heads/asf-site
Commit: d6f8ca78e80772eb0dc9920699692da13885ba75
Parents: c90dbf1 0a5d7e5
Author: James Malone 
Authored: Fri May 6 10:44:14 2016 -0700
Committer: James Malone 
Committed: Fri May 6 10:44:14 2016 -0700

--
 _includes/header.html   |  6 ++
 .../beam/capability/2016/03/17/capability-matrix.html   |  8 +++-
 .../capability/2016/04/03/presentation-materials.html   |  8 +++-
 .../python/sdk/2016/02/25/python-sdk-now-public.html|  8 +++-
 .../beam/update/website/2016/02/22/beam-has-a-logo.html |  8 +++-
 content/blog/index.html |  6 ++
 content/capability-matrix/index.html|  8 +++-
 content/coming-soon.html|  6 ++
 content/contribution-guide/index.html   |  6 ++
 content/docs/index.html |  6 ++
 content/feed.xml| 12 ++--
 content/getting_started/index.html  |  6 ++
 content/index.html  |  6 ++
 content/issue_tracking/index.html   |  6 ++
 content/mailing_lists/index.html|  6 ++
 content/material/index.html |  6 ++
 content/presentation-materials/index.html   |  6 ++
 content/privacy_policy/index.html   |  6 ++
 content/public-meetings/index.html  |  6 ++
 content/source_repository/index.html|  6 ++
 content/team/index.html |  6 ++
 21 files changed, 131 insertions(+), 11 deletions(-)
--

[1/2] incubator-beam-site git commit: Add button to make the navigation bar accesible in mobile browsing

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site c90dbf184 -> d6f8ca78e


Add button to make the navigation bar accesible in mobile browsing


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0a5d7e55
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0a5d7e55
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0a5d7e55

Branch: refs/heads/asf-site
Commit: 0a5d7e55d8c8e363c0348605915d68e2426138a9
Parents: c90dbf1
Author: Ismaël Mejía 
Authored: Thu May 5 23:15:44 2016 +0200
Committer: Ismaël Mejía 
Committed: Thu May 5 23:15:44 2016 +0200

--
 _includes/header.html   |  6 ++
 .../beam/capability/2016/03/17/capability-matrix.html   |  8 +++-
 .../capability/2016/04/03/presentation-materials.html   |  8 +++-
 .../python/sdk/2016/02/25/python-sdk-now-public.html|  8 +++-
 .../beam/update/website/2016/02/22/beam-has-a-logo.html |  8 +++-
 content/blog/index.html |  6 ++
 content/capability-matrix/index.html|  8 +++-
 content/coming-soon.html|  6 ++
 content/contribution-guide/index.html   |  6 ++
 content/docs/index.html |  6 ++
 content/feed.xml| 12 ++--
 content/getting_started/index.html  |  6 ++
 content/index.html  |  6 ++
 content/issue_tracking/index.html   |  6 ++
 content/mailing_lists/index.html|  6 ++
 content/material/index.html |  6 ++
 content/presentation-materials/index.html   |  6 ++
 content/privacy_policy/index.html   |  6 ++
 content/public-meetings/index.html  |  6 ++
 content/source_repository/index.html|  6 ++
 content/team/index.html |  6 ++
 21 files changed, 131 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a5d7e55/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index dbff844..2177687 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -4,6 +4,12 @@
   
 
   
+  
+Toggle navigation
+
+
+
+  
 
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a5d7e55/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index e643fe7..d256c17 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -37,6 +37,12 @@
   
 
   
+  
+Toggle navigation
+
+
+
+  
 
 
   
@@ -93,7 +99,7 @@
 
   
 Clarifying & Formalizing 
Runner Capabilities
-Mar 17, 2016 •  Frances Perry [https://twitter.com/francesjperry";>@francesjperry] & Tyler Akidau 
[https://twitter.com/takidau";>@takidau]
+Mar 17, 2016 •  Frances Perry [https://twitter.com/francesjperry";>@francesjperry] & Tyler Akidau 
[https://twitter.com/takidau";>@takidau]
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a5d7e55/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index adf5ff7..cc06cb3 100644
--- a/content/beam/capability/2016/04/03/presentation-materials.html
+++ b/content/beam/capability/2016/04/03/presentation-materials.html
@@ -37,6 +37,12 @@
   
 
   
+  
+Toggle navigation
+
+
+
+  
 
 
   
@@ -93,7 +99,7 @@
 
   
 Apache Beam Presentation 
Materials
-Apr 3, 2016 •  Frances Perry [https://twitter.com/francesjperry";>@francesjperry] & Tyler Akidau 
[https://twitter.com/takidau";>@takidau]
+Apr 3, 2016 •  Frances Perry [https://twitter.com/francesjperry";>@francesjperry] & Tyler Akidau 
[https://twitter.com/takidau";>@takidau]
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/0a5d7e55/content/beam/python/sdk/2016/02/25/python-sdk-now-public.html
---

[2/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/bw/nameless/beam-logo-bw-nameless.svg
--
diff --git a/images/logos/bw/nameless/beam-logo-bw-nameless.svg 
b/images/logos/bw/nameless/beam-logo-bw-nameless.svg
new file mode 100644
index 000..db48cd9
--- /dev/null
+++ b/images/logos/bw/nameless/beam-logo-bw-nameless.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 103.75402 
144.60986">.cls-1{fill:url(#linear-gradient);}.cls-2{fill:#bdbdbd;}.cls-3,.cls-4{fill:#fff;}.cls-3{opacity:0.6;}.cls-5{fill:#fee900;}.cls-6{fill:#757575;stroke:#757575;stroke-miterlimit:10;}beam-logo_bw
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png
--
diff --git 
a/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png
new file mode 100644
index 000..d5e6b15
Binary files /dev/null and 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-100.png 
differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-1000.png
--
diff --git 
a/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-1000.png 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-1000.png
new file mode 100644
index 000..96ac75e
Binary files /dev/null and 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-1000.png 
differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-200.png
--
diff --git 
a/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-200.png 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-200.png
new file mode 100644
index 000..da8e1be
Binary files /dev/null and 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-200.png 
differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-500.png
--
diff --git 
a/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-500.png 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-500.png
new file mode 100644
index 000..b85df22
Binary files /dev/null and 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom-500.png 
differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom.svg
--
diff --git 
a/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom.svg 
b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom.svg
new file mode 100644
index 000..cb4664a
--- /dev/null
+++ b/images/logos/full-color/name-bottom/beam-logo-full-color-name-bottom.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 46.00439 
52.96988">.cls-1{isolation:isolate;}.cls-2{opacity:0.15;}.cls-12,.cls-19,.cls-2{mix-blend-mode:multiply;}.cls-3{fill:url(#linear-gradient);}.cls-16,.cls-17,.cls-4,.cls-9{fill:#fff;}.cls-5{fill:#fee900;}.cls-6{fill:url(#linear-gradient-2);}.cls-7{fill:url(#linear-gradient-3);}.cls-8{opacity:0.65;fill:url(#linear-gradient-4);}.cls-9{opacity:0.9;}.cls-10,.cls-11{opacity:0.75;}.cls-10{fill:url(#linear-gradient-5);}.cls-12{opacity:0.1;}.cls-13{fill:url(#linear-gradient-6);}.cls-14{fill:url(#linear-gradient-7);}.cls-15{opacity:0.7;fill:url(#linear-gradient-8);}.cls-16{opacity:0.6;}.cls-17{stroke:#fff;}.cls-17,.cls-21{stroke-miterlimit:10;stroke-width:0.07702px;}.cls-18{opacity:0.85;}.cls-19{opacity:0.2;}.cls-20{fill:#ff6d00;}.cls-21{fill:none;stroke:url(#linear-gradient-9);}.cls-22,.cls-23,.cls-24,.cls-25{fill:#e64a19;}.cls-22{opacity:0.25;}.cls-23{opacity:0.35;}.cls-24{op
 acity:0.5;}.cls-25{opacity:0.8;}beam-logo_full-color-name_bottom
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/name-right/beam-logo-full-color-name-right-100.png
--
diff --git 
a/images/logos/full-color/name-right/beam-logo-full-color-name-right-100.png 
b/images/logos/full-color/name-right/beam-logo-full-color-name-right-100.png
new file mode 100644
ind

[5/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-bottom/beam-logo-bw-name-bottom.svg
--
diff --git a/content/images/logos/bw/name-bottom/beam-logo-bw-name-bottom.svg 
b/content/images/logos/bw/name-bottom/beam-logo-bw-name-bottom.svg
new file mode 100644
index 000..10d3a9c
--- /dev/null
+++ b/content/images/logos/bw/name-bottom/beam-logo-bw-name-bottom.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 46.19585 
52.50491">.cls-1,.cls-7{fill:#757575;stroke:#757575;stroke-miterlimit:10;}.cls-1{stroke-width:0.19146px;}.cls-2{fill:url(#linear-gradient);}.cls-3{fill:#bdbdbd;}.cls-4,.cls-5{fill:#fff;}.cls-4{opacity:0.6;}.cls-6{fill:#fee900;}.cls-7{stroke-width:0.19146px;}beam-logo_bw_name-bottom
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-right/beam-logo-bw-name-right-100.png
--
diff --git a/content/images/logos/bw/name-right/beam-logo-bw-name-right-100.png 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-100.png
new file mode 100644
index 000..ce64f9b
Binary files /dev/null and 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-100.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-right/beam-logo-bw-name-right-1000.png
--
diff --git 
a/content/images/logos/bw/name-right/beam-logo-bw-name-right-1000.png 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-1000.png
new file mode 100644
index 000..0c86577
Binary files /dev/null and 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-1000.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-right/beam-logo-bw-name-right-200.png
--
diff --git a/content/images/logos/bw/name-right/beam-logo-bw-name-right-200.png 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-200.png
new file mode 100644
index 000..9ef9e2e
Binary files /dev/null and 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-200.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-right/beam-logo-bw-name-right-500.png
--
diff --git a/content/images/logos/bw/name-right/beam-logo-bw-name-right-500.png 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-500.png
new file mode 100644
index 000..307df53
Binary files /dev/null and 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right-500.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/name-right/beam-logo-bw-name-right.svg
--
diff --git a/content/images/logos/bw/name-right/beam-logo-bw-name-right.svg 
b/content/images/logos/bw/name-right/beam-logo-bw-name-right.svg
new file mode 100644
index 000..355077f
--- /dev/null
+++ b/content/images/logos/bw/name-right/beam-logo-bw-name-right.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 78.03474 
27.68698">.cls-1{fill:url(#linear-gradient);}.cls-2{fill:#bdbdbd;}.cls-3,.cls-4{fill:#fff;}.cls-3{opacity:0.6;}.cls-5{fill:#fee900;}.cls-6,.cls-7{fill:#757575;stroke:#757575;stroke-miterlimit:10;}.cls-6{stroke-width:0.19146px;}.cls-7{stroke-width:0.19146px;}beam-logo_bw_name-right
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/nameless/beam-logo-bw-nameless-100.png
--
diff --git a/content/images/logos/bw/nameless/beam-logo-bw-nameless-100.png 
b/content/images/logos/bw/nameless/beam-logo-bw-nameless-100.png
new file mode 100644
index 000..9258315
Binary files /dev/null and 
b/content/images/logos/bw/nameless/beam-logo-bw-nameless-100.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/nameless/beam-logo-bw-nameless-1000.png
--
diff --git a/content/images/logos/bw/nameless/beam-logo-bw-nameless-1000.png 
b/content/images/logos/bw/nameless/beam-logo-bw-nameless-1000.png
new file mode 100644
index 000..2b83944
Binary files /dev/null and 
b/content/images/logos/bw/nameless/beam-logo-bw-nameless-1000.png differ

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/bw/nameless/beam-logo-bw-nameless-200.png
---

[7/7] incubator-beam-site git commit: Logo files and project materials

[jamesmalone]

Logo files and project materials

This closes #13


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/3b54e40c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/3b54e40c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/3b54e40c

Branch: refs/heads/asf-site
Commit: 3b54e40ce114e80f0184ca890312dcc49c71c95f
Parents: 4af810e 713f080
Author: James Malone 
Authored: Mon May 2 10:24:18 2016 -0700
Committer: James Malone 
Committed: Mon May 2 10:24:18 2016 -0700

--
 _config.yml |   3 +
 _data/logos.yml |   8 +
 _includes/header.html   |  14 +-
 _pages/material.md  |  60 +++
 _posts/2016-02-22-beam-has-a-logo0.markdown |  22 -
 .../2016-02-25-python-sdk-now-public.markdown   |  22 +
 .../2016/03/17/capability-matrix.html   |  14 +-
 .../2016/04/03/presentation-materials.html  |  14 +-
 .../python/sdk/2016/02/25/beam-has-a-logo0.html | 132 --
 .../sdk/2016/02/25/python-sdk-now-public.html   | 138 ++
 .../website/2016/02/22/beam-has-a-logo.html |  14 +-
 content/blog/index.html |  18 +-
 content/capability-matrix/index.html|  16 +-
 content/contribution-guide/index.html   |  14 +-
 content/downloads/logos.zip | Bin 0 -> 664457 bytes
 content/downloads/palette.pdf   | Bin 0 -> 58247 bytes
 content/feed.xml|   8 +-
 content/getting_started/index.html  |  14 +-
 content/images/beam_logo_navbar.png | Bin 2898 -> 7298 bytes
 .../beam-logo-3-color-name-bottom-100.png   | Bin 0 -> 2069 bytes
 .../beam-logo-3-color-name-bottom-1000.png  | Bin 0 -> 23684 bytes
 .../beam-logo-3-color-name-bottom-200.png   | Bin 0 -> 4085 bytes
 .../beam-logo-3-color-name-bottom-500.png   | Bin 0 -> 10905 bytes
 .../beam-logo-3-color-name-bottom.svg   |   1 +
 .../beam-logo-3-color-name-right-100.png| Bin 0 -> 1506 bytes
 .../beam-logo-3-color-name-right-1000.png   | Bin 0 -> 17464 bytes
 .../beam-logo-3-color-name-right-200.png| Bin 0 -> 2963 bytes
 .../beam-logo-3-color-name-right-500.png| Bin 0 -> 7783 bytes
 .../name-right/beam-logo-3-color-name-right.svg |   1 +
 .../nameless/beam-logo-3-color-nameless-100.png | Bin 0 -> 1829 bytes
 .../beam-logo-3-color-nameless-1000.png | Bin 0 -> 21040 bytes
 .../nameless/beam-logo-3-color-nameless-200.png | Bin 0 -> 3522 bytes
 .../nameless/beam-logo-3-color-nameless-500.png | Bin 0 -> 8950 bytes
 .../nameless/beam-logo-3-color-nameless.svg |   1 +
 .../beam-logo-bw-name-bottom-100.png| Bin 0 -> 2134 bytes
 .../beam-logo-bw-name-bottom-1000.png   | Bin 0 -> 24062 bytes
 .../beam-logo-bw-name-bottom-200.png| Bin 0 -> 4172 bytes
 .../beam-logo-bw-name-bottom-500.png| Bin 0 -> 11046 bytes
 .../bw/name-bottom/beam-logo-bw-name-bottom.svg |   1 +
 .../name-right/beam-logo-bw-name-right-100.png  | Bin 0 -> 1510 bytes
 .../name-right/beam-logo-bw-name-right-1000.png | Bin 0 -> 17836 bytes
 .../name-right/beam-logo-bw-name-right-200.png  | Bin 0 -> 3040 bytes
 .../name-right/beam-logo-bw-name-right-500.png  | Bin 0 -> 8059 bytes
 .../bw/name-right/beam-logo-bw-name-right.svg   |   1 +
 .../bw/nameless/beam-logo-bw-nameless-100.png   | Bin 0 -> 1778 bytes
 .../bw/nameless/beam-logo-bw-nameless-1000.png  | Bin 0 -> 20517 bytes
 .../bw/nameless/beam-logo-bw-nameless-200.png   | Bin 0 -> 3416 bytes
 .../bw/nameless/beam-logo-bw-nameless-500.png   | Bin 0 -> 9090 bytes
 .../logos/bw/nameless/beam-logo-bw-nameless.svg |   1 +
 .../beam-logo-full-color-name-bottom-100.png| Bin 0 -> 3395 bytes
 .../beam-logo-full-color-name-bottom-1000.png   | Bin 0 -> 50922 bytes
 .../beam-logo-full-color-name-bottom-200.png| Bin 0 -> 7234 bytes
 .../beam-logo-full-color-name-bottom-500.png| Bin 0 -> 20891 bytes
 .../beam-logo-full-color-name-bottom.svg|   1 +
 .../beam-logo-full-color-name-right-100.png | Bin 0 -> 2462 bytes
 .../beam-logo-full-color-name-right-1000.png| Bin 0 -> 38278 bytes
 .../beam-logo-full-color-name-right-200.png | Bin 0 -> 5222 bytes
 .../beam-logo-full-color-name-right-500.png | Bin 0 -> 15369 bytes
 .../beam-logo-full-color-name-right.svg |   1 +
 .../beam-logo-full-color-nameless-100.png   | Bin 0 -> 5346 bytes
 .../beam-logo-full-color-nameless-1000.png  | Bin 0 -> 133265 bytes
 .../beam-logo-full-color-nameless-200.png   | Bin 0 -> 12642 bytes
 .../beam-logo-full-color-nameless-500.png   | Bin 0 -> 46790 bytes
 .../nameless/beam-logo-full-color-nameless.svg  |   1 +
 content/index.html  |  16 +-
 content/issue_tracking/index.html

[6/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

uot;Full color"
+  3-color: "Three color"
+  bw: "Black and white"
+types: ['nameless', 'name-bottom', 'name-right']
+logo-location: '/images/logos'
+archive-file: logos.zip

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 23a9ebe..dbff844 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -2,7 +2,7 @@
   
 
   
-
+
   
 
 
@@ -10,8 +10,8 @@
 
   Documentation 
   
+Guides
 Getting 
Started
-Presentation Materials
 
 Technical Documentation
 Capability 
Matrix
@@ -24,8 +24,6 @@
   
 Community
 Mailing 
Lists
-https://goo.gl/ps8twC";>Technical Docs
-https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
 Public 
Meetings
 
@@ -36,6 +34,14 @@
   
 
 Blog
+
+  Project 
+  
+Presentation Materials
+Logos and 
design
+http://apache.org/licenses/LICENSE-2.0.html";>License
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/_pages/material.md
--
diff --git a/_pages/material.md b/_pages/material.md
new file mode 100644
index 000..23bc808
--- /dev/null
+++ b/_pages/material.md
@@ -0,0 +1,60 @@
+---
+layout: page
+title: "Apache Beam Material"
+permalink: /material/
+---
+
+This page contains project material for the Apache Beam project.
+
+## Project logos
+You can download [this archive]({{ site.baseurl }}/{{ site.downloads }}/{{ 
site.data.logos.archive-file }})
+containing all of the logos or download the logos individually.
+
+### Scalable Vector Graphics (SVG)
+These [SVG files](https://en.wikipedia.org/wiki/Scalable_Vector_Graphics) can
+be resized easily and are suitable for print or web use. Click on the logo to
+download it.
+
+{% for color in site.data.logos.colors %}
+ {{ color[1] }}
+
+
+
+{% for type in site.data.logos.types %}
+
+
+
+
+
+{% endfor %}
+
+{% endfor %}
+
+
+### Portable Network Graphics (PNG)
+These [PNG files](https://en.wikipedia.org/wiki/Portable_Network_Graphics) are
+available in a number of fixed sizes and are optimized for web use.
+
+{% for color in site.data.logos.colors %}
+ {{ color[1] }}
+
+
+
+{% for type in site.data.logos.types %}
+
+
+
+
+
+{% for size in site.data.logos.sizes %}
+{{ size }}x{{ size }}
+{% unless forloop.last %},{% endunless %}
+{% endfor %}
+
+
+{% endfor %}
+
+{% endfor %}
+
+## Colors and fonts
+The Apache Beam project uses predefined colors and fonts. [This document]({{ 
site.baseurl }}/{{ site.downloads }}/palette.pdf) has more information.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/_posts/2016-02-22-beam-has-a-logo0.markdown
--
diff --git a/_posts/2016-02-22-beam-has-a-logo0.markdown 
b/_posts/2016-02-22-beam-has-a-logo0.markdown
deleted file mode 100644
index 726433c..000
--- a/_posts/2016-02-22-beam-has-a-logo0.markdown
+++ /dev/null
@@ -1,22 +0,0 @@

-layout: post
-title:  "Dataflow Python SDK is now public!"
-date:   2016-02-25 13:00:00 -0800
-excerpt_separator: 
-categories: beam python sdk
-authors:
-- jamesmalone

-
-When the Apache Beam project proposed entry into the [Apache 
Incubator](http://wiki.apache.org/incubator/BeamProposal) the proposal
-included the [Dataflow Java 
SDK](https://github.com/GoogleCloudPlatform/DataflowJavaSDK). In the long term, 
however, Apache Beam aims to support SDKs implemented in multiple languages, 
such as Python.
-
-
-
-Today, Google submitted the [Dataflow Python (2.x) 
SDK](http://github.com/GoogleCloudPlatform/DataflowPythonSDK) on GitHub. Google 
is committed to including the in progress python SDK in Apache Beam and, in 
that spirit, we've moved development of the Python SDK to a public repository. 
While this SDK will not be included with the initial (incubating) releases of 
Apache Beam, our we plan on incorporating the Python SDK into beam during 
incubation. We want to take the time to implement changes from the [technical 
vision](https://goo.gl/nk5OM0) into the Java SDK before we introduce a Python 
SDK for Apache Beam. We believe this will allow us to work on the model and 
SDKs in an ordered fashion.
-
-You can look for the Apache Beam Python SDK in the coming months once we 
finish forking and refactoring the Java SDK.
-
-Best,
-
-Apache Beam Team

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/_posts/2016-02-25-python-sdk-

[3/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/index.html
--
diff --git a/content/index.html b/content/index.html
index dbcef20..31bdacc 100644
--- a/content/index.html
+++ b/content/index.html
@@ -36,7 +36,7 @@
   
 
   
-
+
   
 
 
@@ -44,8 +44,8 @@
 
   Documentation 
   
+Guides
 Getting Started
-Presentation 
Materials
 
 Technical Documentation
 Capability Matrix
@@ -58,8 +58,6 @@
   
 Community
 Mailing Lists
-https://goo.gl/ps8twC";>Technical Docs
-https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
 Public Meetings
 
@@ -70,6 +68,14 @@
   
 
 Blog
+
+  Project 
+  
+Presentation 
Materials
+Logos and design
+http://apache.org/licenses/LICENSE-2.0.html";>License
+  
+
   
 
   
@@ -126,7 +132,7 @@
 
 Mar 17, 2016 - 
Clarifying & Formalizing Runner Capabilities
 
-Feb 25, 2016 - 
Dataflow Python SDK is now public!
+Feb 25, 2016 - 
Dataflow Python SDK is now public!
 
 Feb 22, 2016 - 
Apache Beam has a logo!
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/issue_tracking/index.html
--
diff --git a/content/issue_tracking/index.html 
b/content/issue_tracking/index.html
index c41205e..d5665a2 100644
--- a/content/issue_tracking/index.html
+++ b/content/issue_tracking/index.html
@@ -36,7 +36,7 @@
   
 
   
-
+
   
 
 
@@ -44,8 +44,8 @@
 
   Documentation 
   
+Guides
 Getting Started
-Presentation 
Materials
 
 Technical Documentation
 Capability Matrix
@@ -58,8 +58,6 @@
   
 Community
 Mailing Lists
-https://goo.gl/ps8twC";>Technical Docs
-https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
 Public Meetings
 
@@ -70,6 +68,14 @@
   
 
 Blog
+
+  Project 
+  
+Presentation 
Materials
+Logos and design
+http://apache.org/licenses/LICENSE-2.0.html";>License
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/mailing_lists/index.html
--
diff --git a/content/mailing_lists/index.html b/content/mailing_lists/index.html
index aaa483e..b7710a1 100644
--- a/content/mailing_lists/index.html
+++ b/content/mailing_lists/index.html
@@ -36,7 +36,7 @@
   
 
   
-
+
   
 
 
@@ -44,8 +44,8 @@
 
   Documentation 
   
+Guides
 Getting Started
-Presentation 
Materials
 
 Technical Documentation
 Capability Matrix
@@ -58,8 +58,6 @@
   
 Community
 Mailing Lists
-https://goo.gl/ps8twC";>Technical Docs
-https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
 Public Meetings
 
@@ -70,6 +68,14 @@
   
 
 Blog
+
+  Project 
+  
+Presentation 
Materials
+Logos and design
+http://apache.org/licenses/LICENSE-2.0.html";>License
+  
+
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/material/index.html
--
diff --git a/content/material/index.html b/content/material/index.html
new file mode 100644
index 000..4689690
--- /dev/null
+++ b/content/material/index.html
@@ -0,0 +1,425 @@
+
+
+
+  
+  
+  
+  
+
+  Apache Beam Material
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/material/";>
+  http://beam.incubator.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+
+
+  
+
+  Documentation 
+  
+Guides
+Get

[4/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/content/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
--
diff --git 
a/content/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg 
b/content/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
new file mode 100644
index 000..c0cb27b
--- /dev/null
+++ b/content/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 114.07453 
153.42">.cls-1{isolation:isolate;}.cls-2{opacity:0.15;}.cls-12,.cls-19,.cls-2{mix-blend-mode:multiply;}.cls-3{fill:url(#linear-gradient);}.cls-16,.cls-17,.cls-4,.cls-9{fill:#fff;}.cls-5{fill:#fee900;}.cls-6{fill:url(#linear-gradient-2);}.cls-7{fill:url(#linear-gradient-3);}.cls-8{opacity:0.65;fill:url(#linear-gradient-4);}.cls-9{opacity:0.9;}.cls-10,.cls-11{opacity:0.75;}.cls-10{fill:url(#linear-gradient-5);}.cls-12{opacity:0.1;}.cls-13{fill:url(#linear-gradient-6);}.cls-14{fill:url(#linear-gradient-7);}.cls-15{opacity:0.7;fill:url(#linear-gradient-8);}.cls-16{opacity:0.6;}.cls-17{stroke:#fff;}.cls-17,.cls-21{stroke-miterlimit:10;stroke-width:0.4px;}.cls-18{opacity:0.85;}.cls-19{opacity:0.2;}.cls-20{fill:#ff6d00;}.cls-21{fill:none;stroke:url(#linear-gradient-9);}.cls-22,.cls-23,.cls-24,.cls-25{fill:#e64a19;}.cls-22{opacity:0.25;}.cls-23{opacity:0.35;}.cls-24{opacity
 :0.5;}.cls-25{opacity:0.8;}beam-logo_full-color

[1/7] incubator-beam-site git commit: Addition of material page; nav fixes

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 4af810ead -> 3b54e40ce


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/713f0808/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
--
diff --git a/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg 
b/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
new file mode 100644
index 000..c0cb27b
--- /dev/null
+++ b/images/logos/full-color/nameless/beam-logo-full-color-nameless.svg
@@ -0,0 +1 @@
+http://www.w3.org/2000/svg"; 
xmlns:xlink="http://www.w3.org/1999/xlink"; viewBox="0 0 114.07453 
153.42">.cls-1{isolation:isolate;}.cls-2{opacity:0.15;}.cls-12,.cls-19,.cls-2{mix-blend-mode:multiply;}.cls-3{fill:url(#linear-gradient);}.cls-16,.cls-17,.cls-4,.cls-9{fill:#fff;}.cls-5{fill:#fee900;}.cls-6{fill:url(#linear-gradient-2);}.cls-7{fill:url(#linear-gradient-3);}.cls-8{opacity:0.65;fill:url(#linear-gradient-4);}.cls-9{opacity:0.9;}.cls-10,.cls-11{opacity:0.75;}.cls-10{fill:url(#linear-gradient-5);}.cls-12{opacity:0.1;}.cls-13{fill:url(#linear-gradient-6);}.cls-14{fill:url(#linear-gradient-7);}.cls-15{opacity:0.7;fill:url(#linear-gradient-8);}.cls-16{opacity:0.6;}.cls-17{stroke:#fff;}.cls-17,.cls-21{stroke-miterlimit:10;stroke-width:0.4px;}.cls-18{opacity:0.85;}.cls-19{opacity:0.2;}.cls-20{fill:#ff6d00;}.cls-21{fill:none;stroke:url(#linear-gradient-9);}.cls-22,.cls-23,.cls-24,.cls-25{fill:#e64a19;}.cls-22{opacity:0.25;}.cls-23{opacity:0.35;}.cls-24{opacity
 :0.5;}.cls-25{opacity:0.8;}beam-logo_full-color

[1/2] incubator-beam-site git commit: Update to meeting page

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 0ae64c654 -> 4af810ead


Update to meeting page


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d3c2d89e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d3c2d89e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d3c2d89e

Branch: refs/heads/asf-site
Commit: d3c2d89eadff4be83edec49ffa1ff5fbcd9210e1
Parents: 38d8256
Author: James Malone 
Authored: Thu Apr 14 09:08:48 2016 -0700
Committer: James Malone 
Committed: Thu Apr 14 09:08:48 2016 -0700

--
 _data/meetings.yml | 6 +++---
 content/public-meetings/index.html | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/d3c2d89e/_data/meetings.yml
--
diff --git a/_data/meetings.yml b/_data/meetings.yml
index 9df943b..c12111c 100644
--- a/_data/meetings.yml
+++ b/_data/meetings.yml
@@ -11,11 +11,11 @@ events:
   link: 
"https://docs.google.com/document/d/1SXSLj7FMIgKqj43nTcczFpJzqASeUMUCpbyklk2fBkg/edit?usp=sharing";
   notes:
 
-- date: 2016/05/11
+- date: 2016/05/04
   time: "8:00 - 11:00 Pacific"
   location: Virtual
   type: Technical Deep Dive
   materials:
-  notes: This is a proposed meeting. Virtual attendanceoptions will be 
announced soon.
+  notes: Virtual attendance options willbe announced prior to the meeting.
 
-last_updated: 2016/04/07
+last_updated: 2016/04/14

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/d3c2d89e/content/public-meetings/index.html
--
diff --git a/content/public-meetings/index.html 
b/content/public-meetings/index.html
index 1a7b0aa..edc62cd 100644
--- a/content/public-meetings/index.html
+++ b/content/public-meetings/index.html
@@ -122,17 +122,17 @@
 
   
 
-  2016/05/118:00 - 11:00 Pacific
+  2016/05/048:00 - 11:00 Pacific
   Virtual
   Technical Deep Dive
   
 
   
-  This is a proposed meeting. Virtual attendanceoptions will be 
announced soon.
+  Virtual attendance options willbe announced prior to the 
meeting.
 
   
 
-This list was last updated on 2016/04/07.
+This list was last updated on 2016/04/14.
 
 All Apache Beam meetings are open to the public and we encourage anyone to 
attend. From time to time space in our event location may be limited, so 
preference will be given to PPMC members and others on a first-come, 
first-serve basis.
 

[2/2] incubator-beam-site git commit: Meeting time update

[jamesmalone]

Meeting time update

This closes #12


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4af810ea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4af810ea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4af810ea

Branch: refs/heads/asf-site
Commit: 4af810ead3ef409ea8a8b01e32b336c34e9b5982
Parents: 0ae64c6 d3c2d89
Author: James Malone 
Authored: Thu Apr 14 09:39:32 2016 -0700
Committer: James Malone 
Committed: Thu Apr 14 09:39:32 2016 -0700

--
 _data/meetings.yml | 6 +++---
 content/public-meetings/index.html | 6 +++---
 2 files changed, 6 insertions(+), 6 deletions(-)
--

[2/2] incubator-beam-site git commit: [BEAM-172] Link fixes

[jamesmalone]

[BEAM-172] Link fixes

This closes #11


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/0ae64c65
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/0ae64c65
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/0ae64c65

Branch: refs/heads/asf-site
Commit: 0ae64c654c9e0d119123353ee01ea870d856fea8
Parents: 38d8256 4d374e8
Author: James Malone 
Authored: Fri Apr 8 11:56:47 2016 -0700
Committer: James Malone 
Committed: Fri Apr 8 11:56:47 2016 -0700

--
 content/beam/capability/2016/03/17/capability-matrix.html| 6 +-
 .../beam/capability/2016/04/03/presentation-materials.html   | 1 +
 content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html | 6 +-
 content/beam/update/website/2016/02/22/beam-has-a-logo.html  | 6 +-
 content/blog/index.html  | 1 +
 content/capability-matrix/index.html | 8 ++--
 content/contribution-guide/index.html| 6 +-
 content/feed.xml | 4 ++--
 content/getting_started/index.html   | 6 +-
 content/index.html   | 1 +
 content/issue_tracking/index.html| 6 +-
 content/mailing_lists/index.html | 6 +-
 content/presentation-materials/index.html| 3 ++-
 content/privacy_policy/index.html| 6 +-
 content/source_repository/index.html | 6 +-
 content/team/index.html  | 6 +-
 16 files changed, 63 insertions(+), 15 deletions(-)
--

[1/2] incubator-beam-site git commit: Fix for missing links

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 38d8256d8 -> 0ae64c654


Fix for missing links


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4d374e87
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4d374e87
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4d374e87

Branch: refs/heads/asf-site
Commit: 4d374e875f2f203b276aba2a36b35301d32fb96f
Parents: 38d8256
Author: James Malone 
Authored: Fri Apr 8 10:43:31 2016 -0700
Committer: James Malone 
Committed: Fri Apr 8 10:43:31 2016 -0700

--
 content/beam/capability/2016/03/17/capability-matrix.html| 6 +-
 .../beam/capability/2016/04/03/presentation-materials.html   | 1 +
 content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html | 6 +-
 content/beam/update/website/2016/02/22/beam-has-a-logo.html  | 6 +-
 content/blog/index.html  | 1 +
 content/capability-matrix/index.html | 8 ++--
 content/contribution-guide/index.html| 6 +-
 content/feed.xml | 4 ++--
 content/getting_started/index.html   | 6 +-
 content/index.html   | 1 +
 content/issue_tracking/index.html| 6 +-
 content/mailing_lists/index.html | 6 +-
 content/presentation-materials/index.html| 3 ++-
 content/privacy_policy/index.html| 6 +-
 content/source_repository/index.html | 6 +-
 content/team/index.html  | 6 +-
 16 files changed, 63 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4d374e87/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index be15c84..6fb26e2 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -44,7 +44,10 @@
   Documentation 
   
 Getting Started
-   Capability Matrix
+Presentation 
Materials
+
+Technical Documentation
+Capability Matrix
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
   
@@ -57,6 +60,7 @@
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
+Public Meetings
 
 Contribute
 Contribution Guide

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4d374e87/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index 96f6dac..493cf55 100644
--- a/content/beam/capability/2016/04/03/presentation-materials.html
+++ b/content/beam/capability/2016/04/03/presentation-materials.html
@@ -60,6 +60,7 @@
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
+Public Meetings
 
 Contribute
 Contribution Guide

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4d374e87/content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
--
diff --git a/content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html 
b/content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
index 2f833d7..e4066e6 100644
--- a/content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
+++ b/content/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
@@ -44,7 +44,10 @@
   Documentation 
   
 Getting Started
-   Capability Matrix
+Presentation 
Materials
+
+Technical Documentation
+Capability Matrix
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
   
@@ -57,6 +60,7 @@
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
+Public Meetings
 
 Contribute
 Contribution Guide

http://git-wip-us.apache.org

[3/4] incubator-beam-site git commit: Update for email list

[jamesmalone]

Update for email list


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4001a86f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4001a86f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4001a86f

Branch: refs/heads/asf-site
Commit: 4001a86ff714260c800db7889bc03be7356298e3
Parents: bf4a8fc
Author: James Malone 
Authored: Thu Apr 7 13:52:22 2016 -0700
Committer: James Malone 
Committed: Thu Apr 7 13:52:22 2016 -0700

--
 _pages/public-meetings.md  | 2 +-
 content/public-meetings/index.html | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4001a86f/_pages/public-meetings.md
--
diff --git a/_pages/public-meetings.md b/_pages/public-meetings.md
index 2fc6447..ead8c07 100644
--- a/_pages/public-meetings.md
+++ b/_pages/public-meetings.md
@@ -39,7 +39,7 @@ All Apache Beam meetings are open to the public and we 
encourage anyone to atten
 ## I want to give a public talk about Apache Beam
 To get started, we recommend you review the Apache Beam [presentation 
materials]({{ site.baseurl }}/presentation-materials/) page to review the 
content the Apache Beam community has already created. These materials will 
possibly save you time and energy as you create content for your event.
 
-Once you have scheduled your event, we welcome you to announce it on the 
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org) mailing 
list. Additionally, please open a [JIRA 
item](https://issues.apache.org/jira/browse/BEAM) using the `website` component 
with details so we can update this page.
+Once you have scheduled your event, we welcome you to announce it on the 
[u...@beam.incubator.apache.org](mailto:u...@beam.incubator.apache.org) mailing 
list. Additionally, please open a [JIRA 
item](https://issues.apache.org/jira/browse/BEAM) using the `website` component 
with details so we can update this page.
 
 If you have any questions as you prepare for your event, we recommend you 
reach out to the Apache Beam community through the 
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org) mailing 
list. The Beam community can help provide feedback on your materials and 
promote your event.
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4001a86f/content/public-meetings/index.html
--
diff --git a/content/public-meetings/index.html 
b/content/public-meetings/index.html
index a64d6a0..1a7b0aa 100644
--- a/content/public-meetings/index.html
+++ b/content/public-meetings/index.html
@@ -139,7 +139,7 @@
 I want to give a 
public talk about Apache Beam
 To get started, we recommend you review the Apache Beam presentation materials page to review the 
content the Apache Beam community has already created. These materials will 
possibly save you time and energy as you create content for your event.
 
-Once you have scheduled your event, we welcome you to announce it on the dev@beam.incubator.apache.org
 mailing list. Additionally, please open a https://issues.apache.org/jira/browse/BEAM";>JIRA item using the website component with details so we can 
update this page.
+Once you have scheduled your event, we welcome you to announce it on the user@beam.incubator.apache.org
 mailing list. Additionally, please open a https://issues.apache.org/jira/browse/BEAM";>JIRA item using the website component with details so we can 
update this page.
 
 If you have any questions as you prepare for your event, we recommend you 
reach out to the Apache Beam community through the dev@beam.incubator.apache.org
 mailing list. The Beam community can help provide feedback on your materials 
and promote your event.
 

[4/4] incubator-beam-site git commit: [BEAM-172] Public meetings page

[jamesmalone]

[BEAM-172] Public meetings page

This closes #10


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/38d8256d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/38d8256d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/38d8256d

Branch: refs/heads/asf-site
Commit: 38d8256d8cf8886ecf80937a26e4dd8c5b29606b
Parents: ece3515 4001a86
Author: James Malone 
Authored: Fri Apr 8 10:07:05 2016 -0700
Committer: James Malone 
Committed: Fri Apr 8 10:07:05 2016 -0700

--
 _data/meetings.yml |  21 
 _includes/header.html  |   1 +
 _pages/public-meetings.md  |  55 ++
 content/public-meetings/index.html | 181 
 4 files changed, 258 insertions(+)
--

[1/4] incubator-beam-site git commit: Addition of public meetings page

[jamesmalone]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site ece351589 -> 38d8256d8


Addition of public meetings page


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/970a4fd3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/970a4fd3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/970a4fd3

Branch: refs/heads/asf-site
Commit: 970a4fd3a82bc1ced2c800c3c9c3f569f62f19c9
Parents: ece3515
Author: James Malone 
Authored: Tue Apr 5 18:22:24 2016 -0700
Committer: James Malone 
Committed: Tue Apr 5 18:22:24 2016 -0700

--
 _data/meetings.yml |  21 
 _includes/header.html  |   1 +
 _pages/public-meetings.md  |  55 ++
 content/public-meetings/index.html | 181 
 4 files changed, 258 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/970a4fd3/_data/meetings.yml
--
diff --git a/_data/meetings.yml b/_data/meetings.yml
new file mode 100644
index 000..f72e4c8
--- /dev/null
+++ b/_data/meetings.yml
@@ -0,0 +1,21 @@
+events:
+- date: 2016/04/01
+  time: "9:30 - 16:00 Pacific"
+  location: PayPalSan Jose, CA, USA
+  type: Dev/PPMC Meeting
+  materials:
+- title: Presentation - PPMC Deep Dive
+  link: 
"https://docs.google.com/presentation/d/1uTb7dx4-Y2OM_B0_3XF_whwAL2FlDTTuq2QzP9sJ4Mg/edit?usp=sharing";
+
+- title: Notes - PPMC Deep Dive
+  link: 
"https://docs.google.com/document/d/1SXSLj7FMIgKqj43nTcczFpJzqASeUMUCpbyklk2fBkg/edit?usp=sharing";
+  notes:
+
+- date: 2016/04/25
+  time: "8:30 - 16:00 Pacific"
+  location: Hilton Financial Districthttps://goo.gl/maps/L4cAuK8xdm52";>750 Kearny StSan Francisco, CA 
94108
+  type: Technical Deep Dive
+  materials:
+  notes: This is a proposed meeting. Space may be be limited.Attendance 
preference will be given to PPMC members.
+
+last_updated: 2016/04/05

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/970a4fd3/_includes/header.html
--
diff --git a/_includes/header.html b/_includes/header.html
index 6311f39..23a9ebe 100644
--- a/_includes/header.html
+++ b/_includes/header.html
@@ -27,6 +27,7 @@
 https://goo.gl/ps8twC";>Technical Docs
 https://goo.gl/nk5OM0";>Technical Vision
 Apache Beam Team
+Public 
Meetings
 
 Contribute
 Contribution 
Guide

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/970a4fd3/_pages/public-meetings.md
--
diff --git a/_pages/public-meetings.md b/_pages/public-meetings.md
new file mode 100644
index 000..2fc6447
--- /dev/null
+++ b/_pages/public-meetings.md
@@ -0,0 +1,55 @@
+---
+layout: page
+title: "Apache Beam Public Meetings"
+permalink: /public-meetings/
+---
+Apache Beam is a shared effort within the open source community. To grow and 
develop that effort, it helps to schedule and hold public meetings, including:
+
+* Formal meetings for the Apache Beam community for project planning and 
discussion
+* Informal "Meetup" events to discuss, share the vision and usefulness of 
Apache Beam
+
+The goal of these meetings will vary, though they will typically focus on 
technical discussions, community matters, and decision-making. These meetings 
have been held or are scheduled by the Apache Beam community.
+
+
+  
+  Date & Time
+  Location
+  Type
+  Meeting materials
+  Notes
+  
+  {% for meeting in site.data.meetings.events %}
+
+  {{ meeting.date }}{{ meeting.time }}
+  {{ meeting.location }}
+  {{ meeting.type }}
+  
+{% for material in meeting.materials %}
+{{ material.title }}
+{% endfor %}
+  
+  {{ meeting.notes }}
+
+  {% endfor %}
+
+*This list was last updated on {{ site.data.meetings.last_updated }}.*
+
+All Apache Beam meetings are open to the public and we encourage anyone to 
attend. From time to time space in our event location may be limited, so 
preference will be given to PPMC members and others on a first-come, 
first-serve basis.
+
+## I want to give a public talk about Apache Beam
+To get started, we recommend you review the Apache Beam [presentation 
materials]({{ site.baseurl }}/presentation-materials/) page to review the 
content the Apache Beam community has already created. These materials will 
possibly save you time and energy as you create content for your event.
+
+Once you have scheduled your event, we welcome you to announce it on the 
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org) mailing 

[2/4] incubator-beam-site git commit: Update for second meeting

[jamesmalone]

Update for second meeting


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/bf4a8fcd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/bf4a8fcd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/bf4a8fcd

Branch: refs/heads/asf-site
Commit: bf4a8fcd6d2006714388cb2c94fd03c30a64c0b1
Parents: 970a4fd
Author: James Malone 
Authored: Thu Apr 7 12:50:05 2016 -0700
Committer: James Malone 
Committed: Thu Apr 7 12:50:05 2016 -0700

--
 _data/meetings.yml | 10 +-
 content/public-meetings/index.html |  8 
 2 files changed, 9 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/bf4a8fcd/_data/meetings.yml
--
diff --git a/_data/meetings.yml b/_data/meetings.yml
index f72e4c8..9df943b 100644
--- a/_data/meetings.yml
+++ b/_data/meetings.yml
@@ -11,11 +11,11 @@ events:
   link: 
"https://docs.google.com/document/d/1SXSLj7FMIgKqj43nTcczFpJzqASeUMUCpbyklk2fBkg/edit?usp=sharing";
   notes:
 
-- date: 2016/04/25
-  time: "8:30 - 16:00 Pacific"
-  location: Hilton Financial Districthttps://goo.gl/maps/L4cAuK8xdm52";>750 Kearny StSan Francisco, CA 
94108
+- date: 2016/05/11
+  time: "8:00 - 11:00 Pacific"
+  location: Virtual
   type: Technical Deep Dive
   materials:
-  notes: This is a proposed meeting. Space may be be limited.Attendance 
preference will be given to PPMC members.
+  notes: This is a proposed meeting. Virtual attendanceoptions will be 
announced soon.
 
-last_updated: 2016/04/05
+last_updated: 2016/04/07

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/bf4a8fcd/content/public-meetings/index.html
--
diff --git a/content/public-meetings/index.html 
b/content/public-meetings/index.html
index e7c79fe..a64d6a0 100644
--- a/content/public-meetings/index.html
+++ b/content/public-meetings/index.html
@@ -122,17 +122,17 @@
 
   
 
-  2016/04/258:30 - 16:00 Pacific
-  Hilton Financial Districthttps://goo.gl/maps/L4cAuK8xdm52";>750 Kearny StSan Francisco, CA 
94108
+  2016/05/118:00 - 11:00 Pacific
+  Virtual
   Technical Deep Dive
   
 
   
-  This is a proposed meeting. Space may be be limited.Attendance 
preference will be given to PPMC members.
+  This is a proposed meeting. Virtual attendanceoptions will be 
announced soon.
 
   
 
-This list was last updated on 2016/04/05.
+This list was last updated on 2016/04/07.
 
 All Apache Beam meetings are open to the public and we encourage anyone to 
attend. From time to time space in our event location may be limited, so 
preference will be given to PPMC members and others on a first-come, 
first-serve basis.
 

svn commit: r1733208 - /incubator/beam/website/team/index.html

[jamesmalone]

Author: jamesmalone
Date: Wed Mar  2 04:08:08 2016
New Revision: 1733208

URL: http://svn.apache.org/viewvc?rev=1733208&view=rev
Log:
Update to team site

Modified:
incubator/beam/website/team/index.html

Modified: incubator/beam/website/team/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/team/index.html?rev=1733208&r1=1733207&r2=1733208&view=diff
==
--- incubator/beam/website/team/index.html (original)
+++ incubator/beam/website/team/index.html Wed Mar  2 04:08:08 2016
@@ -105,7 +105,7 @@
 
   Aljoscha Krettek
   aljoscha
-  mailto:aljos...@apache.org";>aljos...@apache.org
+  aljoscha [at] apache [dot] org
   data Artisans
   committer, PPMC
   +1
@@ -114,7 +114,7 @@
 
   Amit Sela
   amitsela
-  mailto:amits...@apache.org";>amits...@apache.org
+  amitsela [at] apache [dot] org
   PayPal
   committer, PPMC
   +2
@@ -123,7 +123,7 @@
 
   Ben Chambers
   
-  mailto:bchamb...@google.com";>bchamb...@google.com
+  bchambers [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -132,7 +132,7 @@
 
   Craig Chambers
   
-  mailto:chamb...@google.com";>chamb...@google.com
+  chambers [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -141,7 +141,7 @@
 
   Dan Halperin
   
-  mailto:dhalp...@google.com";>dhalp...@google.com
+  dhalperi [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -149,8 +149,8 @@
   
 
   Davor Bonaci
-  
-  mailto:da...@google.com";>da...@google.com
+  davor
+  davor [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -158,8 +158,8 @@
   
 
   Frances Perry
-  
-  mailto:f...@google.com";>f...@google.com
+  frances
+  fjp [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -168,7 +168,7 @@
 
   James Malone
   jamesmalone
-  mailto:jamesmal...@google.com";>jamesmal...@google.com
+  jamesmalone [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -177,7 +177,7 @@
 
   Jean-Baptiste Onofré
   jbonofre
-  mailto:da...@google.com";>da...@google.com
+  jbonofre [at] apache [dot] org
   Talend
   champion, committer, PPMC
   +1
@@ -186,7 +186,7 @@
 
   Josh Wills
   jwills
-  mailto:jwi...@apache.org";>jwi...@apache.org
+  jwills [at] apache [dot] org
   
   committer, PPMC
   -8
@@ -195,7 +195,7 @@
 
   Kostas Tzoumas
   
-  mailto:kos...@data-artisans.com";>kos...@data-artisans.com
+  kostas [at] data-artisans [dot] com
   data Artisans
   committer, PPMC
   +1
@@ -204,7 +204,7 @@
 
   Kenneth Knowles
   kenn
-  mailto:k...@google.com";>k...@google.com
+  klk [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -213,7 +213,7 @@
 
   Luke Cwik
   
-  mailto:lc...@google.com";>lc...@google.com
+  lcwik [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -222,7 +222,7 @@
 
   Maximilian Michels
   mxm
-  mailto:m...@apache.org";>m...@apache.org
+  mxm [at] apache [dot] org
   data Artisans
   committer, PPMC
   +1
@@ -231,7 +231,7 @@
 
   Stephan Ewen
   
-  mailto:step...@data-artisans.com";>step...@data-artisans.com
+  stephan [at] data-artisans [dot] com
   data Artisans
   committer, PPMC
   +1
@@ -240,7 +240,7 @@
 
   Tom White
   
-  mailto:t...@cloudera.com";>t...@cloudera.com
+  tom [at] cloudera [dot] com
   Cloudera
   committer, PPMC
   -8
@@ -249,7 +249,7 @@
 
   Tyler Akidau
   
-  mailto:taki...@google.com";>taki...@google.com
+  takidau [at] google [dot] com
   Google
   committer, PPMC
   -8
@@ -258,7 +258,7 @@
 
   Robert Bradshaw
   
-  mailto:rober...@google.com";>rober...@google.com
+  robertwb [at] google [dot] com
   Google
   committer, PPMC
   -8

[19/50] [abbrv] incubator-beam git commit: Handle Undeclared Side Outputs in ParDoInProcessEvaluator

[jamesmalone]

Handle Undeclared Side Outputs in ParDoInProcessEvaluator

The value of an Undeclared Side Output is ignored by the
InProcessPipelineRunner.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115489641


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/045e3436
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/045e3436
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/045e3436

Branch: refs/heads/master
Commit: 045e3436e6b2ce9593c1f8ebeaa57ea7d229134e
Parents: 1cc0211
Author: tgroh 
Authored: Wed Feb 24 13:56:45 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../inprocess/ParDoInProcessEvaluator.java  | 17 -
 .../ParDoMultiEvaluatorFactoryTest.java | 74 
 .../ParDoSingleEvaluatorFactoryTest.java| 42 +++
 3 files changed, 131 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/045e3436/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoInProcessEvaluator.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoInProcessEvaluator.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoInProcessEvaluator.java
index a2b083b..f0b2ca2 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoInProcessEvaluator.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoInProcessEvaluator.java
@@ -27,7 +27,10 @@ import com.google.cloud.dataflow.sdk.values.TupleTag;
 
 import org.joda.time.Instant;
 
+import java.util.ArrayList;
 import java.util.Collection;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
 
 class ParDoInProcessEvaluator {
@@ -61,6 +64,7 @@ class ParDoInProcessEvaluator {
 
   static class BundleOutputManager implements OutputManager {
 private final Map, UncommittedBundle> bundles;
+private final Map, List> undeclaredOutputs;
 
 public static BundleOutputManager create(Map, 
UncommittedBundle> outputBundles) {
   return new BundleOutputManager(outputBundles);
@@ -68,6 +72,7 @@ class ParDoInProcessEvaluator {
 
 private BundleOutputManager(Map, UncommittedBundle> 
bundles) {
   this.bundles = bundles;
+  undeclaredOutputs = new HashMap<>();
 }
 
 @SuppressWarnings("unchecked")
@@ -75,8 +80,16 @@ class ParDoInProcessEvaluator {
 public  void output(TupleTag tag, WindowedValue output) {
   @SuppressWarnings("rawtypes")
   UncommittedBundle bundle = bundles.get(tag);
-  bundle.add(output);
+  if (bundle == null) {
+List undeclaredContents = undeclaredOutputs.get(tag);
+if (undeclaredContents == null) {
+  undeclaredContents = new ArrayList();
+  undeclaredOutputs.put(tag, undeclaredContents);
+}
+undeclaredContents.add(output);
+  } else {
+bundle.add(output);
+  }
 }
   }
 }
-

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/045e3436/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoMultiEvaluatorFactoryTest.java
--
diff --git 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoMultiEvaluatorFactoryTest.java
 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoMultiEvaluatorFactoryTest.java
index 5251a76..c55a9d5 100644
--- 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoMultiEvaluatorFactoryTest.java
+++ 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/ParDoMultiEvaluatorFactoryTest.java
@@ -137,5 +137,79 @@ public class ParDoMultiEvaluatorFactoryTest implements 
Serializable {
 WindowedValue.timestampedValueInGlobalWindow(4, new Instant(1000)),
 WindowedValue.valueInGlobalWindow(5, 
PaneInfo.ON_TIME_AND_ONLY_FIRING)));
   }
+
+  @Test
+  public void testParDoMultiUndeclaredSideOutput() throws Exception {
+TestPipeline p = TestPipeline.create();
+
+PCollection input = p.apply(Create.of("foo", "bara", "bazam"));
+
+TupleTag> mainOutputTag = new TupleTag>() {};
+final TupleTag elementTag = new TupleTag<>();
+final TupleTag lengthTag = new TupleTag<>();
+
+BoundMulti> pardo =
+ParDo.of(new DoFn>() {
+  @Override
+  public void processElement(ProcessContext c) {
+c.output(KV.of(c.element(), 
c.element().length()));
+c.sideOutput(elementTag, c.element());
+c.sideOutput(lengthTag, c.element().length());
+  }
+}).withOutputTag

[32/50] [abbrv] incubator-beam git commit: Adjust dependencies to avoid pulling in unneeded stax-api

[jamesmalone]

Adjust dependencies to avoid pulling in unneeded stax-api

This is the javax.xml.stream API that is provided with Java 1.6 and
up.

 - The dependency on stax-api from stax2-api switched from "compile"
   scope to "provided" scope in current versions, so we can remove
   the transitive dependency simply by upgrade.
 - The woodstox-core-asl package is upgraded to a version explicitly
   compatible with the new version of stax2-api.
 - The dependency on stax-api from woodstox-core-asl remains in "compile"
   scope even after upgrade, so this change explicitly suppresses it.

Selected bits of `mvn dependency:tree -Dverbose`:

 [INFO] com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:1...)
-[INFO] +- org.codehaus.woodstox:stax2-api:jar:3.1.1:compile
-[INFO] |  \- javax.xml.stream:stax-api:jar:1.0-2:compile
-[INFO] +- org.codehaus.woodstox:woodstox-core-asl:jar:4.1.2:compile
-[INFO] |  +- (javax.xml.stream:stax-api:jar:1.0-2:compile - omitted for d...)
-[INFO] |  \- (org.codehaus.woodstox:stax2-api:jar:3.1.1:compile - omitted...)
+[INFO] +- org.codehaus.woodstox:stax2-api:jar:3.1.4:compile
+[INFO] +- org.codehaus.woodstox:woodstox-core-asl:jar:4.4.1:compile
+[INFO] |  \- (org.codehaus.woodstox:stax2-api:jar:3.1.4:compile - omitted...)

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115574469


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6c710405
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6c710405
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6c710405

Branch: refs/heads/master
Commit: 6c710405af5fef7e8a6b334a7e393dc5d8ab006e
Parents: 510a55d
Author: klk 
Authored: Thu Feb 25 10:07:00 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 pom.xml |  2 ++
 sdk/pom.xml | 11 +--
 2 files changed, 11 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6c710405/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 760a10d..bfade1a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -82,7 +82,9 @@
 3.0.0-beta-1
 v1-rev7-1.21.0
 1.7.14
+3.1.4
 v1-rev53-1.21.0
+4.4.1
   
 
   pom

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6c710405/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index aeead72..1f15b02 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -667,15 +667,22 @@
 
   org.codehaus.woodstox
   stax2-api
-  3.1.1
+  ${stax2.version}
   true
 
 
 
   org.codehaus.woodstox
   woodstox-core-asl
-  4.1.2
+  ${woodstox.version}
   true
+  
+
+
+  javax.xml.stream
+  stax-api
+
+  
 
 
 

[26/50] [abbrv] incubator-beam git commit: ProtoCoder: a Coder for Protocol Buffers Messages

[jamesmalone]

ProtoCoder: a Coder for Protocol Buffers Messages

Release Notes
Adds ProtoCoder, which is a Coder for Protocol Buffers messages. ProtoCoder has
the following advantages over Proto2Coder:
 * ProtoCoder is designed to support Protocol Buffers syntax versions 2 and 3,
   not just proto2.
 * ProtoCoder dynamically inspects the Message type it encodes and will throw a
   Coder.NonDeterministicException when the configured message cannot be
   encoded deterministically.
 * Because early versions of the Protocol Buffers 2 syntax did not allow
   non-deterministic types, Proto2Coder does not do this type inspection and
   instead always claims to be able to coder deterministically. For backwards
   compatibility, we are not changing this behavior.

Proto2Coder is now deprecated; we recommend that all users switch to
ProtoCoder.
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115411869


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/01fd8595
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/01fd8595
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/01fd8595

Branch: refs/heads/master
Commit: 01fd8595e7d54009d313e98889fdf322d98b5c63
Parents: db708bb
Author: dhalperi 
Authored: Tue Feb 23 20:07:32 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 sdk/pom.xml |   2 +-
 .../dataflow/sdk/coders/CoderRegistry.java  |   5 +-
 .../cloud/dataflow/sdk/coders/Proto2Coder.java  |   3 +
 .../sdk/coders/StringDelegateCoder.java |   4 +-
 .../sdk/coders/protobuf/ProtoCoder.java | 411 +++
 .../sdk/coders/protobuf/ProtobufUtil.java   | 171 
 .../dataflow/sdk/coders/CoderRegistryTest.java  |  14 +-
 .../dataflow/sdk/coders/Proto2CoderTest.java|   1 +
 .../sdk/coders/protobuf/ProtoCoderTest.java | 182 
 .../sdk/coders/protobuf/ProtobufUtilTest.java   | 195 +
 10 files changed, 980 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/01fd8595/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index bdbdd5f..bc6d519 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -442,7 +442,7 @@
 
   com.google.cloud.dataflow
   google-cloud-dataflow-java-proto-library-all
-  0.5.160127
+  0.5.160222
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/01fd8595/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/CoderRegistry.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/CoderRegistry.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/CoderRegistry.java
index 1ef0971..00982e6 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/CoderRegistry.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/CoderRegistry.java
@@ -18,6 +18,7 @@ package com.google.cloud.dataflow.sdk.coders;
 
 import com.google.api.services.bigquery.model.TableRow;
 import 
com.google.cloud.dataflow.sdk.coders.CannotProvideCoderException.ReasonCode;
+import com.google.cloud.dataflow.sdk.coders.protobuf.ProtoCoder;
 import com.google.cloud.dataflow.sdk.transforms.SerializableFunction;
 import com.google.cloud.dataflow.sdk.util.CoderUtils;
 import com.google.cloud.dataflow.sdk.values.KV;
@@ -81,8 +82,8 @@ public class CoderRegistry implements CoderProvider {
   private static final Logger LOG = 
LoggerFactory.getLogger(CoderRegistry.class);
 
   public CoderRegistry() {
-
setFallbackCoderProvider(CoderProviders.firstOf(Proto2Coder.coderProvider(),
-SerializableCoder.PROVIDER));
+setFallbackCoderProvider(
+CoderProviders.firstOf(ProtoCoder.coderProvider(), 
SerializableCoder.PROVIDER));
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/01fd8595/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
index 56ec0c7..ef91ba9 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
@@ -17,6 +17,7 @@ package com.google.cloud.dataflow.sdk.coders;
 
 import static com.google.common.base.Preconditions.checkArgument;
 
+import com.google.cloud.dataflow.sdk.coders.protobuf.ProtoCoder;
 import com.google.cloud.dataflow.sdk.util.CloudObject;
 import com.google.cloud.dataflow.sdk.util.

[47/50] [abbrv] incubator-beam git commit: Dataflow code drop!

[jamesmalone]

Dataflow code drop!


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/394390fd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/394390fd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/394390fd

Branch: refs/heads/master
Commit: 394390fd98b02e9685f106c624676fd1533e4201
Parents: 41e5cc9 d4dcaaa
Author: Frances Perry 
Authored: Fri Feb 26 12:10:43 2016 -0800
Committer: Frances Perry 
Committed: Fri Feb 26 12:10:43 2016 -0800

--
 .gitattributes  |   26 +
 .gitignore  |   16 +
 .travis.yml |   35 +
 CONTRIBUTING.md |   31 +
 LICENSE |  202 ++
 README.md   |  125 +
 checkstyle.xml  |  413 +++
 contrib/README.md   |   53 +
 contrib/hadoop/AUTHORS.md   |7 +
 contrib/hadoop/README.md|   24 +
 contrib/hadoop/pom.xml  |  169 +
 .../contrib/hadoop/HadoopFileSource.java|  485 +++
 .../dataflow/contrib/hadoop/WritableCoder.java  |  110 +
 .../contrib/hadoop/HadoopFileSourceTest.java|  189 ++
 .../contrib/hadoop/WritableCoderTest.java   |   36 +
 contrib/join-library/AUTHORS.md |6 +
 contrib/join-library/README.md  |   33 +
 contrib/join-library/pom.xml|  185 ++
 .../dataflow/contrib/joinlibrary/Join.java  |  185 ++
 .../contrib/joinlibrary/InnerJoinTest.java  |  142 +
 .../contrib/joinlibrary/OuterLeftJoinTest.java  |  152 +
 .../contrib/joinlibrary/OuterRightJoinTest.java |  152 +
 examples/README.md  |   95 +
 examples/pom.xml|  521 
 .../dataflow/examples/DebuggingWordCount.java   |  182 ++
 .../dataflow/examples/MinimalWordCount.java |  117 +
 .../dataflow/examples/WindowedWordCount.java|  269 ++
 .../cloud/dataflow/examples/WordCount.java  |  206 ++
 .../examples/common/DataflowExampleOptions.java |   34 +
 .../examples/common/DataflowExampleUtils.java   |  485 +++
 .../common/ExampleBigQueryTableOptions.java |   53 +
 ...xamplePubsubTopicAndSubscriptionOptions.java |   44 +
 .../common/ExamplePubsubTopicOptions.java   |   44 +
 .../examples/common/PubsubFileInjector.java |  153 +
 .../examples/complete/AutoComplete.java |  510 +++
 .../cloud/dataflow/examples/complete/README.md  |   44 +
 .../examples/complete/StreamingWordExtract.java |  163 +
 .../cloud/dataflow/examples/complete/TfIdf.java |  431 +++
 .../examples/complete/TopWikipediaSessions.java |  223 ++
 .../examples/complete/TrafficMaxLaneFlow.java   |  425 +++
 .../examples/complete/TrafficRoutes.java|  459 +++
 .../examples/cookbook/BigQueryTornadoes.java|  179 ++
 .../cookbook/CombinePerKeyExamples.java |  223 ++
 .../examples/cookbook/DatastoreWordCount.java   |  269 ++
 .../examples/cookbook/DeDupExample.java |  100 +
 .../examples/cookbook/FilterExamples.java   |  266 ++
 .../examples/cookbook/JoinExamples.java |  185 ++
 .../examples/cookbook/MaxPerKeyExamples.java|  173 +
 .../cloud/dataflow/examples/cookbook/README.md  |   55 +
 .../examples/cookbook/TriggerExample.java   |  564 
 .../examples/MinimalWordCountJava8.java |   68 +
 .../examples/complete/game/GameStats.java   |  347 +++
 .../examples/complete/game/HourlyTeamScore.java |  193 ++
 .../examples/complete/game/LeaderBoard.java |  237 ++
 .../dataflow/examples/complete/game/README.md   |  119 +
 .../examples/complete/game/UserScore.java   |  239 ++
 .../complete/game/injector/Injector.java|  417 +++
 .../complete/game/injector/InjectorUtils.java   |  101 +
 .../injector/RetryHttpInitializerWrapper.java   |  127 +
 .../complete/game/utils/WriteToBigQuery.java|  134 +
 .../game/utils/WriteWindowedToBigQuery.java |   76 +
 .../examples/DebuggingWordCountTest.java|   45 +
 .../cloud/dataflow/examples/WordCountTest.java  |   85 +
 .../examples/complete/AutoCompleteTest.java |  181 ++
 .../dataflow/examples/complete/TfIdfTest.java   |   67 +
 .../complete/TopWikipediaSessionsTest.java  |   62 +
 .../cookbook/BigQueryTornadoesTest.java |   80 +
 .../cookbook/CombinePerKeyExamplesTest.java |   90 +
 .../examples/cookbook/DeDupExampleTest.java |   83 +
 .../examples/cookbook/FilterExamplesTest.java   |   85 +
 .../examples/cookbook/JoinExamplesTest.java |  114 +
 .../cookbook/MaxPerKeyExamplesTest.java |   85 +
 .../examples/cookbook/TriggerExampleTest.java   |  139 +
 .../examples/MinimalWordCountJava8Test.java |  103 +
 .../examples/complete/game/GameStatsTest.jav

[40/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

[jamesmalone]

Finish Flattenning InProcess package

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115601346


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/06c8911e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/06c8911e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/06c8911e

Branch: refs/heads/master
Commit: 06c8911ed421276a240c7e1aa8bf9e8f8d961c05
Parents: 3eb3092
Author: tgroh 
Authored: Thu Feb 25 13:59:48 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../dataflow/sdk/runners/inprocess/Clock.java   |   29 +
 .../inprocess/InMemoryWatermarkManager.java | 1316 ++
 .../sdk/runners/inprocess/InProcessBundle.java  |  117 ++
 .../inprocess/InProcessBundleOutputManager.java |   50 +
 .../inprocess/InProcessExecutionContext.java|6 +-
 .../inprocess/InProcessTimerInternals.java  |   84 ++
 .../inprocess/InProcessTransformResult.java |2 +-
 .../sdk/runners/inprocess/NanosOffsetClock.java |   58 +
 .../runners/inprocess/StepTransformResult.java  |2 +-
 .../sdk/runners/inprocess/util/Clock.java   |   29 -
 .../util/InMemoryWatermarkManager.java  | 1316 --
 .../runners/inprocess/util/InProcessBundle.java |  117 --
 .../util/InProcessBundleOutputManager.java  |   51 -
 .../inprocess/util/InProcessTimerInternals.java |   84 --
 .../inprocess/util/NanosOffsetClock.java|   58 -
 .../BoundedReadEvaluatorFactoryTest.java|1 -
 .../inprocess/FlattenEvaluatorFactoryTest.java  |1 -
 .../GroupByKeyEvaluatorFactoryTest.java |1 -
 .../inprocess/InMemoryWatermarkManagerTest.java | 1099 +++
 .../runners/inprocess/InProcessBundleTest.java  |  143 ++
 .../inprocess/InProcessTimerInternalsTest.java  |  131 ++
 .../sdk/runners/inprocess/MockClock.java|   60 +
 .../ParDoMultiEvaluatorFactoryTest.java |3 +-
 .../ParDoSingleEvaluatorFactoryTest.java|3 +-
 .../UnboundedReadEvaluatorFactoryTest.java  |1 -
 .../inprocess/ViewEvaluatorFactoryTest.java |1 -
 .../util/InMemoryWatermarkManagerTest.java  | 1099 ---
 .../inprocess/util/InProcessBundleTest.java |  143 --
 .../util/InProcessTimerInternalsTest.java   |  131 --
 .../sdk/runners/inprocess/util/MockClock.java   |   60 -
 30 files changed, 3093 insertions(+), 3103 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/Clock.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/Clock.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/Clock.java
new file mode 100644
index 000..11e6ec1
--- /dev/null
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/Clock.java
@@ -0,0 +1,29 @@
+/*
+ * Copyright (C) 2015 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+package com.google.cloud.dataflow.sdk.runners.inprocess;
+
+import org.joda.time.Instant;
+
+/**
+ * Access to the current time.
+ */
+public interface Clock {
+  /**
+   * Returns the current time as an {@link Instant}.
+   */
+  Instant now();
+}
+

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManager.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManager.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManager.java
new file mode 100644
index 000..e280e22
--- /dev/null
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManager.java
@@ -0,0 +1,1316 @@
+/*
+ * Copyright (C) 2015 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ 

[38/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManagerTest.java
--
diff --git 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManagerTest.java
 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManagerTest.java
new file mode 100644
index 000..2425152
--- /dev/null
+++ 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/InMemoryWatermarkManagerTest.java
@@ -0,0 +1,1099 @@
+/*
+ * Copyright (C) 2015 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+package com.google.cloud.dataflow.sdk.runners.inprocess;
+
+import static org.hamcrest.Matchers.contains;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.emptyIterable;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.Assert.assertThat;
+
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InMemoryWatermarkManager.FiredTimers;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InMemoryWatermarkManager.TimerUpdate;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InMemoryWatermarkManager.TimerUpdate.TimerUpdateBuilder;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InMemoryWatermarkManager.TransformWatermarks;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.UncommittedBundle;
+import com.google.cloud.dataflow.sdk.testing.TestPipeline;
+import com.google.cloud.dataflow.sdk.transforms.AppliedPTransform;
+import com.google.cloud.dataflow.sdk.transforms.Create;
+import com.google.cloud.dataflow.sdk.transforms.DoFn;
+import com.google.cloud.dataflow.sdk.transforms.Filter;
+import com.google.cloud.dataflow.sdk.transforms.Flatten;
+import com.google.cloud.dataflow.sdk.transforms.ParDo;
+import com.google.cloud.dataflow.sdk.transforms.WithKeys;
+import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
+import com.google.cloud.dataflow.sdk.util.TimeDomain;
+import com.google.cloud.dataflow.sdk.util.TimerInternals.TimerData;
+import com.google.cloud.dataflow.sdk.util.WindowedValue;
+import com.google.cloud.dataflow.sdk.util.state.StateNamespaces;
+import com.google.cloud.dataflow.sdk.values.KV;
+import com.google.cloud.dataflow.sdk.values.PCollection;
+import com.google.cloud.dataflow.sdk.values.PCollectionList;
+import com.google.cloud.dataflow.sdk.values.PValue;
+import com.google.cloud.dataflow.sdk.values.TimestampedValue;
+import com.google.common.collect.ImmutableList;
+
+import org.hamcrest.BaseMatcher;
+import org.hamcrest.Description;
+import org.hamcrest.Matcher;
+import org.joda.time.Instant;
+import org.joda.time.ReadableInstant;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Tests for {@link InMemoryWatermarkManager}.
+ */
+@RunWith(JUnit4.class)
+public class InMemoryWatermarkManagerTest implements Serializable {
+  private transient MockClock clock;
+
+  private transient PCollection createdInts;
+
+  private transient PCollection filtered;
+  private transient PCollection filteredTimesTwo;
+  private transient PCollection> keyed;
+
+  private transient PCollection intsToFlatten;
+  private transient PCollection flattened;
+
+  private transient InMemoryWatermarkManager manager;
+
+  @Before
+  public void setup() {
+TestPipeline p = TestPipeline.create();
+
+createdInts = p.apply("createdInts", Create.of(1, 2, 3));
+
+filtered = createdInts.apply("filtered", Filter.greaterThan(1));
+filteredTimesTwo = filtered.apply("timesTwo", ParDo.of(new DoFn() {
+  @Override
+  public void processElement(DoFn.ProcessContext c) 
throws Exception {
+c.output(c.element() * 2);
+  }
+}));
+
+keyed = createdInts.apply("keyed", WithKeys.of("MyKey"));
+
+intsToFlatten = p.apply("intsToFlatten", Create.of(-1, 256, 65535));
+PCollectionList preFlatten = 
PCollectionList.of(createdInts).and(intsT

[39/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManager.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManager.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManager.java
deleted file mode 100644
index 4428e41..000
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManager.java
+++ /dev/null
@@ -1,1316 +0,0 @@
-/*
- * Copyright (C) 2015 Google Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy 
of
- * the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations 
under
- * the License.
- */
-package com.google.cloud.dataflow.sdk.runners.inprocess.util;
-
-import com.google.cloud.dataflow.sdk.Pipeline;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
-import com.google.cloud.dataflow.sdk.transforms.AppliedPTransform;
-import com.google.cloud.dataflow.sdk.transforms.PTransform;
-import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
-import com.google.cloud.dataflow.sdk.util.TimeDomain;
-import com.google.cloud.dataflow.sdk.util.TimerInternals;
-import com.google.cloud.dataflow.sdk.util.TimerInternals.TimerData;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.values.PCollection;
-import com.google.cloud.dataflow.sdk.values.PValue;
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.base.MoreObjects;
-import com.google.common.collect.ComparisonChain;
-import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
-import com.google.common.collect.Iterables;
-import com.google.common.collect.Ordering;
-import com.google.common.collect.SortedMultiset;
-import com.google.common.collect.TreeMultiset;
-
-import org.joda.time.Instant;
-
-import java.util.ArrayList;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.EnumMap;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.NavigableSet;
-import java.util.Objects;
-import java.util.PriorityQueue;
-import java.util.Set;
-import java.util.TreeSet;
-import java.util.concurrent.atomic.AtomicReference;
-
-import javax.annotation.Nullable;
-
-/**
- * Manages watermarks of {@link PCollection PCollections} and input and output 
watermarks of
- * {@link AppliedPTransform AppliedPTransforms} to provide event-time and 
completion tracking for
- * in-memory execution. {@link InMemoryWatermarkManager} is designed to update 
and return a
- * consistent view of watermarks in the presence of concurrent updates.
- *
- * An {@link InMemoryWatermarkManager} is provided with the collection of 
root
- * {@link AppliedPTransform AppliedPTransforms} and a map of {@link 
PCollection PCollections} to
- * all the {@link AppliedPTransform AppliedPTransforms} that consume them at 
construction time.
- *
- * Whenever a root {@link AppliedPTransform transform} produces elements, 
the
- * {@link InMemoryWatermarkManager} is provided with the produced elements and 
the output watermark
- * of the producing {@link AppliedPTransform transform}. The
- * {@link InMemoryWatermarkManager watermark manager} is responsible for 
computing the watermarks
- * of all {@link AppliedPTransform transforms} that consume one or more
- * {@link PCollection PCollections}.
- *
- * Whenever a non-root {@link AppliedPTransform} finishes processing one or 
more in-flight
- * elements (referred to as the input {@link CommittedBundle bundle}), the 
following occurs
- * atomically:
- * 
- *  All of the in-flight elements are removed from the collection of 
pending elements for the
- *  {@link AppliedPTransform}.
- *  All of the elements produced by the {@link AppliedPTransform} are 
added to the collection
- *  of pending elements for each {@link AppliedPTransform} that consumes 
them.
- *  The input watermark for the {@link AppliedPTransform} becomes the 
maximum value of
- *
- *  the previous input watermark
- *  the minimum of
- *
- *  the timestamps of all currently pending elements
- *  all input {@link PCollection} watermarks
- *
- *  
- *
- *  
- *  The output watermark for the {@link AppliedPTransform} becomes the 
maximum of
- *
-

[34/50] [abbrv] incubator-beam git commit: Update Timers and State in the InProcess ParDoEvaluator

[jamesmalone]

Update Timers and State in the InProcess ParDoEvaluator

If the ParDo accessed state, put the committed value into the transform
result, and likewise with timers.

Add a #commitState method to InProcessStepContext to return the
committed state.

Implement stateInternals() and timerInternals() to provide actual
implementations of StateInternals and TimerInternals. Use concrete types
due to implementation requirements. stateInternals() and
timerInternals() construct response values the first time they are
called based on the underlying data structure; #commitState returns null
if and only if stateInternals was not used by the transform, and
likewise for #getTimerUpdate

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115504122


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ca98da2a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ca98da2a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ca98da2a

Branch: refs/heads/master
Commit: ca98da2a372210325e1c8985292b4040d1ac8c62
Parents: 639e9d9
Author: tgroh 
Authored: Wed Feb 24 16:01:50 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../inprocess/InProcessExecutionContext.java| 108 +
 .../inprocess/InProcessPipelineRunner.java  |  47 +---
 .../inprocess/ParDoInProcessEvaluator.java  |  34 ++-
 .../inprocess/ParDoMultiEvaluatorFactory.java   |  25 +--
 .../inprocess/ParDoSingleEvaluatorFactory.java  |  30 +--
 .../util/InMemoryWatermarkManager.java  |  17 ++
 .../ParDoMultiEvaluatorFactoryTest.java | 223 ++-
 .../ParDoSingleEvaluatorFactoryTest.java| 184 ++-
 8 files changed, 556 insertions(+), 112 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ca98da2a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessExecutionContext.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessExecutionContext.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessExecutionContext.java
new file mode 100644
index 000..6342cd4
--- /dev/null
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessExecutionContext.java
@@ -0,0 +1,108 @@
+/*
+ * Copyright (C) 2016 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+package com.google.cloud.dataflow.sdk.runners.inprocess;
+
+import com.google.cloud.dataflow.sdk.runners.inprocess.util.Clock;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.TimerUpdate;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.TransformWatermarks;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InProcessTimerInternals;
+import com.google.cloud.dataflow.sdk.util.BaseExecutionContext;
+import com.google.cloud.dataflow.sdk.util.ExecutionContext;
+import com.google.cloud.dataflow.sdk.util.TimerInternals;
+import com.google.cloud.dataflow.sdk.util.common.worker.StateSampler;
+import 
com.google.cloud.dataflow.sdk.util.state.CopyOnAccessInMemoryStateInternals;
+
+/**
+ * Execution Context for the {@link InProcessPipelineRunner}.
+ *
+ * This implementation is not thread safe. A new {@link 
InProcessExecutionContext} must be created
+ * for each thread that requires it.
+ */
+class InProcessExecutionContext
+extends 
BaseExecutionContext {
+  private final Clock clock;
+  private final Object key;
+  private final CopyOnAccessInMemoryStateInternals existingState;
+  private final TransformWatermarks watermarks;
+
+  public InProcessExecutionContext(Clock clock, Object key,
+  CopyOnAccessInMemoryStateInternals existingState, 
TransformWatermarks watermarks) {
+this.clock = clock;
+this.key = key;
+this.existingState = existingState;
+this.watermarks = watermarks;
+  }
+
+  @Override
+  protected InProcessStepContext createStepContext(
+  String stepName, String transformName, StateSampler stateSampler) {
+return new InProcessStepContext(this, stepName, transformName);
+  }
+
+  /**
+   * Step Cont

[25/50] [abbrv] incubator-beam git commit: Add InProcess Override for CreatePCollectionView

[jamesmalone]

Add InProcess Override for CreatePCollectionView

Because windowing is used to retrieve values from a
PCollectionView, the elements must go through a GroupByKey.

Provide a PTransform override for use in the InProcess runner to apply
global grouping by window and pane, and a WriteView primitive to store
the contents of the view in a PCollectionView.

Update the View PTransform to make the view it returns available outside
of the application.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115409320


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/87b28e7d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/87b28e7d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/87b28e7d

Branch: refs/heads/master
Commit: 87b28e7dd1b2f8ca31eb155cc5bad4f98717664b
Parents: 6613031
Author: tgroh 
Authored: Tue Feb 23 19:12:37 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../inprocess/InProcessPipelineRunner.java  |  5 +-
 .../runners/inprocess/ViewEvaluatorFactory.java | 69 +---
 .../cloud/dataflow/sdk/transforms/View.java |  5 ++
 .../inprocess/ViewEvaluatorFactoryTest.java | 33 --
 4 files changed, 96 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87b28e7d/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessPipelineRunner.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessPipelineRunner.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessPipelineRunner.java
index 72642da..26c5061 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessPipelineRunner.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessPipelineRunner.java
@@ -20,10 +20,12 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 import com.google.cloud.dataflow.sdk.Pipeline;
 import com.google.cloud.dataflow.sdk.annotations.Experimental;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.GroupByKeyEvaluatorFactory.InProcessGroupByKey;
+import 
com.google.cloud.dataflow.sdk.runners.inprocess.ViewEvaluatorFactory.InProcessCreatePCollectionView;
 import com.google.cloud.dataflow.sdk.transforms.AppliedPTransform;
 import com.google.cloud.dataflow.sdk.transforms.GroupByKey;
 import com.google.cloud.dataflow.sdk.transforms.GroupByKey.GroupByKeyOnly;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
+import com.google.cloud.dataflow.sdk.transforms.View.CreatePCollectionView;
 import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
 import com.google.cloud.dataflow.sdk.transforms.windowing.Trigger;
 import com.google.cloud.dataflow.sdk.util.BaseExecutionContext;
@@ -60,6 +62,7 @@ public class InProcessPipelineRunner {
   defaultTransformOverrides =
   ImmutableMap., Class>builder()
   .put(GroupByKey.class, InProcessGroupByKey.class)
+  .put(CreatePCollectionView.class, 
InProcessCreatePCollectionView.class)
   .build();
 
   private static Map, TransformEvaluatorFactory> 
defaultEvaluatorFactories =
@@ -222,7 +225,7 @@ public class InProcessPipelineRunner {
  * Create a bundle whose elements will be used in a PCollectionView.
  */
  PCollectionViewWriter 
createPCollectionViewWriter(
-PCollection input, PCollectionView output);
+PCollection> input, PCollectionView output);
 
 /**
  * Get the options used by this {@link Pipeline}.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87b28e7d/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ViewEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ViewEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ViewEvaluatorFactory.java
index 654652c..f47cd1d 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ViewEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/ViewEvaluatorFactory.java
@@ -15,11 +15,16 @@
  */
 package com.google.cloud.dataflow.sdk.runners.inprocess;
 
+import com.google.cloud.dataflow.sdk.coders.KvCoder;
+import com.google.cloud.dataflow.sdk.coders.VoidCoder;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.InProcessEvaluationContext;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.PCollectionViewWriter;

[16/50] [abbrv] incubator-beam git commit: Add used-but-undeclared dependency on google-http-client

[jamesmalone]

Add used-but-undeclared dependency on google-http-client

Prior to this change, running

$ mvn dependency:analyze -pl sdk

yields

[WARNING] Used undeclared dependencies found:
[WARNING]com.google.code.findbugs:jsr305:jar:3.0.1:compile
[WARNING]javax.xml.stream:stax-api:jar:1.0-2:compile
[WARNING]com.google.http-client:google-http-client:jar:1.21.0:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.codehaus.woodstox:woodstox-core-asl:jar:4.1.2:compile
[WARNING]org.tukaani:xz:jar:1.5:compile

This is accurate. Many classes depend on com.google.api.client.util from
the google-http-client package.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115303825


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6b372ecb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6b372ecb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6b372ecb

Branch: refs/heads/master
Commit: 6b372ecb3dee573569dfbce415f45265eb3f6f67
Parents: 13a042a
Author: klk 
Authored: Mon Feb 22 21:42:17 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 sdk/pom.xml | 14 ++
 1 file changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6b372ecb/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index e3744fd..bdbdd5f 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -519,6 +519,20 @@
 
 
   com.google.http-client
+  google-http-client
+  ${google-clients.version}
+  
+
+
+  com.google.guava
+  guava-jdk5
+
+  
+
+
+
+  com.google.http-client
   google-http-client-jackson2
   ${google-clients.version}
   

[30/50] [abbrv] incubator-beam git commit: Use a static variable for CoderCalled in WriteTest

[jamesmalone]

Use a static variable for CoderCalled in WriteTest

The test is broken when Create serializes the created values at
apply-time rather than at execution-time, as the modification happens
after the Create is applied within the Write PTransform. Moving to a
static variable allows the state of the operation to be captured at any
time without breaking the test, so long as the captured (and executed)
operation uses the same ClassLoader as the test.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115515376


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8b5257fc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8b5257fc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8b5257fc

Branch: refs/heads/master
Commit: 8b5257fc64be2697d001bd335302e14591aca044
Parents: 7b28d23
Author: tgroh 
Authored: Wed Feb 24 17:58:23 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java  | 2 +-
 .../test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java  | 6 +-
 2 files changed, 6 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8b5257fc/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java
index 4d130cc..0b78b83 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/Write.java
@@ -117,7 +117,7 @@ public class Write {
 PCollection input, WriteOperation writeOperation) {
   Pipeline p = input.getPipeline();
 
-  // A coder to user for the WriteOperation.
+  // A coder to use for the WriteOperation.
   @SuppressWarnings("unchecked")
   Coder> operationCoder =
   (Coder>) 
SerializableCoder.of(writeOperation.getClass());

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8b5257fc/sdk/src/test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java
--
diff --git a/sdk/src/test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java
index 2340187..b92f2f1 100644
--- a/sdk/src/test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java
+++ b/sdk/src/test/java/com/google/cloud/dataflow/sdk/io/WriteTest.java
@@ -184,8 +184,12 @@ public class WriteTest {
   FINALIZED
 }
 
+// Must be static in case the WriteOperation is serialized before the its 
coder is obtained.
+// If this occurs, the value will be modified but not reflected in the 
WriteOperation that is
+// executed by the runner, and the finalize method will fail.
+private static volatile boolean coderCalled = false;
+
 private State state = State.INITIAL;
-private boolean coderCalled = false;
 
 private final TestSink sink;
 private final UUID id = UUID.randomUUID();

[33/50] [abbrv] incubator-beam git commit: Handle multiple requests in InProcess Read Primitives

[jamesmalone]

Handle multiple requests in InProcess Read Primitives

Each source that is invoked by a read should produce its elements in
serial. Using a queue of available sources enforces only one worker
having access to a source at a time.

Add EmptyTransformEvaluator, to be returned in the case that there are
no unused sources. EmptyTransformEvaluator ignores all input, produces
no output, and cannot advance the watermark.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115578920


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3eb30924
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3eb30924
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3eb30924

Branch: refs/heads/master
Commit: 3eb309242047f08afb105cc8a4e0c05f72131fec
Parents: 6c71040
Author: tgroh 
Authored: Thu Feb 25 10:44:41 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../inprocess/BoundedReadEvaluatorFactory.java  | 61 ++
 .../inprocess/EmptyTransformEvaluator.java  | 49 
 .../UnboundedReadEvaluatorFactory.java  | 75 +-
 .../BoundedReadEvaluatorFactoryTest.java| 75 ++
 .../UnboundedReadEvaluatorFactoryTest.java  | 83 +++-
 5 files changed, 273 insertions(+), 70 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3eb30924/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
index d11187c..1c02798 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/BoundedReadEvaluatorFactory.java
@@ -27,8 +27,10 @@ import com.google.cloud.dataflow.sdk.util.WindowedValue;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 
 import java.io.IOException;
-import java.util.Map;
+import java.util.Queue;
 import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ConcurrentMap;
 
 import javax.annotation.Nullable;
 
@@ -42,34 +44,62 @@ final class BoundedReadEvaluatorFactory implements 
TransformEvaluatorFactory {
* Evaluators are cached here to ensure that the reader is not restarted if 
the evaluator is
* retriggered.
*/
-  private final Map> sourceEvaluators =
-  new ConcurrentHashMap<>();
+  private final ConcurrentMap>>
+  sourceEvaluators = new ConcurrentHashMap<>();
 
   @SuppressWarnings({"unchecked", "rawtypes"})
   @Override
   public  TransformEvaluator forApplication(
   AppliedPTransform application,
   @Nullable CommittedBundle inputBundle,
-  InProcessEvaluationContext evaluationContext) {
+  InProcessEvaluationContext evaluationContext)
+  throws IOException {
 return getTransformEvaluator((AppliedPTransform) application, 
evaluationContext);
   }
 
   private  TransformEvaluator getTransformEvaluator(
   final AppliedPTransform, Bounded> 
transform,
-  final InProcessEvaluationContext evaluationContext) {
+  final InProcessEvaluationContext evaluationContext)
+  throws IOException {
+BoundedReadEvaluator evaluator =
+getTransformEvaluatorQueue(transform, evaluationContext).poll();
+if (evaluator == null) {
+  return EmptyTransformEvaluator.create(transform);
+}
+return evaluator;
+  }
+
+  /**
+   * Get the queue of {@link TransformEvaluator TransformEvaluators} that 
produce elements for the
+   * provided application of {@link Bounded Read.Bounded}, initializing it if 
required.
+   *
+   * This method is thread-safe, and will only produce new evaluators if no 
other invocation has
+   * already done so.
+   */
+  @SuppressWarnings("unchecked")
+  private  Queue> 
getTransformEvaluatorQueue(
+  final AppliedPTransform, Bounded> 
transform,
+  final InProcessEvaluationContext evaluationContext)
+  throws IOException {
+// Key by the application and the context the evaluation is occurring in 
(which call to
+// Pipeline#run).
 EvaluatorKey key = new EvaluatorKey(transform, evaluationContext);
-@SuppressWarnings("unchecked")
-BoundedReadEvaluator result =
-(BoundedReadEvaluator) sourceEvaluators.get(key);
-if (result == null) {
-  try {
-result = new BoundedReadEvaluator(transform, 
evaluationContext);
-  } catch (IOEx

[14/50] [abbrv] incubator-beam git commit: Migrate TextIO.Write to a custom sink

[jamesmalone]

Migrate TextIO.Write to a custom sink

Note for user requested sharding limits to be supported,
each pipeline runner must support applying those sharding limits.

Google Cloud Dataflow supports sharding limits.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115310814


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d7b5189c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d7b5189c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d7b5189c

Branch: refs/heads/master
Commit: d7b5189c5708b48308060dd40d6f3ab073759d28
Parents: 6b372ec
Author: lcwik 
Authored: Mon Feb 22 23:59:46 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../google/cloud/dataflow/sdk/io/TextIO.java| 187 +--
 .../sdk/runners/DataflowPipelineRunner.java | 108 ++-
 .../sdk/runners/DataflowPipelineTranslator.java |   5 -
 .../sdk/runners/dataflow/TextIOTranslator.java  |  91 -
 .../cloud/dataflow/sdk/io/TextIOTest.java   |  22 ---
 .../sdk/runners/DataflowPipelineRunnerTest.java |  21 +--
 .../runners/DataflowPipelineTranslatorTest.java |   4 +-
 .../dataflow/sdk/runners/TransformTreeTest.java |   9 +-
 8 files changed, 209 insertions(+), 238 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d7b5189c/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
index 0bb2861..d342f25 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
@@ -26,11 +26,9 @@ import com.google.cloud.dataflow.sdk.io.Read.Bounded;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner;
 import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
-import com.google.cloud.dataflow.sdk.runners.worker.TextSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
+import com.google.cloud.dataflow.sdk.util.MimeTypes;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 import com.google.cloud.dataflow.sdk.values.PInput;
@@ -39,10 +37,13 @@ import com.google.common.base.Preconditions;
 import com.google.protobuf.ByteString;
 
 import java.io.IOException;
+import java.io.OutputStream;
 import java.nio.ByteBuffer;
+import java.nio.channels.Channels;
 import java.nio.channels.ReadableByteChannel;
 import java.nio.channels.SeekableByteChannel;
-import java.util.List;
+import java.nio.channels.WritableByteChannel;
+import java.nio.charset.StandardCharsets;
 import java.util.NoSuchElementException;
 import java.util.regex.Pattern;
 
@@ -66,7 +67,7 @@ import javax.annotation.Nullable;
  *
  * See the following examples:
  *
- *  {@code
+ * {@code
  * Pipeline p = ...;
  *
  * // A simple Read of a local file (only runs locally):
@@ -79,7 +80,7 @@ import javax.annotation.Nullable;
  * p.apply(TextIO.Read.named("ReadNumbers")
  *.from("gs://my_bucket/path/to/numbers-*.txt")
  *.withCoder(TextualIntegerCoder.of()));
- * } 
+ * }
  *
  * To write a {@link PCollection} to one or more text files, use
  * {@link TextIO.Write}, specifying {@link TextIO.Write#to(String)} to specify
@@ -94,7 +95,7 @@ import javax.annotation.Nullable;
  * will be overwritten.
  *
  * For example:
- *  {@code
+ * {@code
  * // A simple Write to a local file (only runs locally):
  * PCollection lines = ...;
  * lines.apply(TextIO.Write.to("/path/to/file.txt"));
@@ -106,7 +107,7 @@ import javax.annotation.Nullable;
  *   .to("gs://my_bucket/path/to/numbers")
  *   .withSuffix(".txt")
  *   .withCoder(TextualIntegerCoder.of()));
- * } 
+ * }
  *
  * Permissions
  * When run using the {@link DirectPipelineRunner}, your pipeline can read 
and write text files
@@ -477,9 +478,6 @@ public class TextIO {
   /** Requested number of shards. 0 for automatic. */
   private final int numShards;
 
-  /** Insert a shuffle before writing to decouple parallelism when 
numShards != 0. */
-  private final boolean forceReshard;
-
   /** The shard template of each file written, combined with prefix and 
suffix. */
   private final String shardTemp

[50/50] [abbrv] incubator-beam git commit: This closes #1

[jamesmalone]

This closes #1


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/211e76ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/211e76ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/211e76ab

Branch: refs/heads/master
Commit: 211e76abf9ba34c35ef13cca279cbeefdad7c406
Parents: 11e8427 2efe761
Author: James Malone 
Authored: Fri Feb 26 14:51:08 2016 -0800
Committer: James Malone 
Committed: Fri Feb 26 14:51:08 2016 -0800

--
 .gitattributes  |   26 +
 .gitignore  |   16 +
 .travis.yml |   35 +
 LICENSE |  202 ++
 README.md   |   71 +-
 checkstyle.xml  |  413 +++
 contrib/README.md   |   53 +
 contrib/hadoop/AUTHORS.md   |7 +
 contrib/hadoop/README.md|   24 +
 contrib/hadoop/pom.xml  |  169 +
 .../contrib/hadoop/HadoopFileSource.java|  485 +++
 .../dataflow/contrib/hadoop/WritableCoder.java  |  110 +
 .../contrib/hadoop/HadoopFileSourceTest.java|  189 ++
 .../contrib/hadoop/WritableCoderTest.java   |   36 +
 contrib/join-library/AUTHORS.md |6 +
 contrib/join-library/README.md  |   33 +
 contrib/join-library/pom.xml|  185 ++
 .../dataflow/contrib/joinlibrary/Join.java  |  185 ++
 .../contrib/joinlibrary/InnerJoinTest.java  |  142 +
 .../contrib/joinlibrary/OuterLeftJoinTest.java  |  152 +
 .../contrib/joinlibrary/OuterRightJoinTest.java |  152 +
 examples/README.md  |   95 +
 examples/pom.xml|  521 
 .../dataflow/examples/DebuggingWordCount.java   |  182 ++
 .../dataflow/examples/MinimalWordCount.java |  117 +
 .../dataflow/examples/WindowedWordCount.java|  269 ++
 .../cloud/dataflow/examples/WordCount.java  |  206 ++
 .../examples/common/DataflowExampleOptions.java |   34 +
 .../examples/common/DataflowExampleUtils.java   |  485 +++
 .../common/ExampleBigQueryTableOptions.java |   53 +
 ...xamplePubsubTopicAndSubscriptionOptions.java |   44 +
 .../common/ExamplePubsubTopicOptions.java   |   44 +
 .../examples/common/PubsubFileInjector.java |  153 +
 .../examples/complete/AutoComplete.java |  510 +++
 .../cloud/dataflow/examples/complete/README.md  |   44 +
 .../examples/complete/StreamingWordExtract.java |  163 +
 .../cloud/dataflow/examples/complete/TfIdf.java |  431 +++
 .../examples/complete/TopWikipediaSessions.java |  223 ++
 .../examples/complete/TrafficMaxLaneFlow.java   |  425 +++
 .../examples/complete/TrafficRoutes.java|  459 +++
 .../examples/cookbook/BigQueryTornadoes.java|  179 ++
 .../cookbook/CombinePerKeyExamples.java |  223 ++
 .../examples/cookbook/DatastoreWordCount.java   |  269 ++
 .../examples/cookbook/DeDupExample.java |  100 +
 .../examples/cookbook/FilterExamples.java   |  266 ++
 .../examples/cookbook/JoinExamples.java |  185 ++
 .../examples/cookbook/MaxPerKeyExamples.java|  173 +
 .../cloud/dataflow/examples/cookbook/README.md  |   55 +
 .../examples/cookbook/TriggerExample.java   |  564 
 .../examples/MinimalWordCountJava8.java |   68 +
 .../examples/complete/game/GameStats.java   |  347 +++
 .../examples/complete/game/HourlyTeamScore.java |  193 ++
 .../examples/complete/game/LeaderBoard.java |  237 ++
 .../dataflow/examples/complete/game/README.md   |  119 +
 .../examples/complete/game/UserScore.java   |  239 ++
 .../complete/game/injector/Injector.java|  417 +++
 .../complete/game/injector/InjectorUtils.java   |  101 +
 .../injector/RetryHttpInitializerWrapper.java   |  127 +
 .../complete/game/utils/WriteToBigQuery.java|  134 +
 .../game/utils/WriteWindowedToBigQuery.java |   76 +
 .../examples/DebuggingWordCountTest.java|   45 +
 .../cloud/dataflow/examples/WordCountTest.java  |   85 +
 .../examples/complete/AutoCompleteTest.java |  181 ++
 .../dataflow/examples/complete/TfIdfTest.java   |   67 +
 .../complete/TopWikipediaSessionsTest.java  |   62 +
 .../cookbook/BigQueryTornadoesTest.java |   80 +
 .../cookbook/CombinePerKeyExamplesTest.java |   90 +
 .../examples/cookbook/DeDupExampleTest.java |   83 +
 .../examples/cookbook/FilterExamplesTest.java   |   85 +
 .../examples/cookbook/JoinExamplesTest.java |  114 +
 .../cookbook/MaxPerKeyExamplesTest.java |   85 +
 .../examples/cookbook/TriggerExampleTest.java   |  139 +
 .../examples/MinimalWordCountJava8Test.java |  103 +
 .../examples/complete/game/GameStatsTest.java   |   99 +
 .../complete/game/HourlyTeamScoreTest.java  | 

[07/50] [abbrv] incubator-beam git commit: Move over Google Cloud Dataflow worker utilities to worker module

[jamesmalone]

Move over Google Cloud Dataflow worker utilities to worker module

Note that CopyableSeekableByteChannel and LazyMultiReaderIterator
were deleted and not moved because they were only used by TextReader
which has now been removed as well.

This is for Apache Beam.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115271204


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7ff52a0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7ff52a0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7ff52a0e

Branch: refs/heads/master
Commit: 7ff52a0e184f984254bd9ad4dd71a9f67193e593
Parents: c857afa
Author: lcwik 
Authored: Mon Feb 22 14:33:36 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../worker/CopyableSeekableByteChannel.java | 270 ---
 .../runners/worker/LazyMultiReaderIterator.java |  87 -
 .../sdk/runners/worker/ReaderUtils.java |  74 -
 .../sdk/util/common/worker/ElementCounter.java  |  32 --
 .../sdk/util/common/worker/Operation.java   | 182 --
 .../sdk/util/common/worker/OutputReceiver.java  |  74 -
 .../sdk/util/common/worker/ProgressTracker.java |  38 ---
 .../common/worker/ProgressTrackerGroup.java |  71 
 .../sdk/util/common/worker/ReadOperation.java   | 333 ---
 .../sdk/util/common/worker/Receiver.java|  27 --
 10 files changed, 1188 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7ff52a0e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/CopyableSeekableByteChannel.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/CopyableSeekableByteChannel.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/CopyableSeekableByteChannel.java
deleted file mode 100644
index 55d25b9..000
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/CopyableSeekableByteChannel.java
+++ /dev/null
@@ -1,270 +0,0 @@
-/***
- * Copyright (C) 2015 Google Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy 
of
- * the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations 
under
- * the License.
- 
**/
-
-package com.google.cloud.dataflow.sdk.runners.worker;
-
-import static com.google.api.client.util.Preconditions.checkNotNull;
-import static com.google.api.client.util.Preconditions.checkState;
-
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.nio.channels.ClosedChannelException;
-import java.nio.channels.SeekableByteChannel;
-
-import javax.annotation.concurrent.GuardedBy;
-
-/**
- * A {@link SeekableByteChannel} that adds copy semantics.
- *
- * This implementation uses a lock to ensure that only one thread accesses
- * the underlying {@code SeekableByteChannel} at any given time.
- *
- * {@link SeekableByteChannel#close} is called on the underlying channel 
once
- * all {@code CopyableSeekableByteChannel} objects copied from the initial
- * {@code CopyableSeekableByteChannel} are closed.
- *
- * The implementation keeps track of the position of each
- * {@code CopyableSeekableByteChannel}; on access, it synchronizes with the
- * other {@code CopyableSeekableByteChannel} instances accessing the underlying
- * channel, seeks to its own position, performs the operation, updates its 
local
- * position, and returns the result.
- */
-final class CopyableSeekableByteChannel implements SeekableByteChannel {
-  /** This particular stream's position in the base stream. */
-  private long pos;
-
-  /**
-   * The synchronization object keeping track of the base
-   * {@link SeekableByteChannel}, its reference count, and its current 
position.
-   * This also doubles as the lock shared by all
-   * {@link CopyableSeekableByteChannel} instances derived from some original
-   * instance.
-   */
-  private final Sync sync;
-
-  /**
-   * Indicates whether this {@link CopyableSeekableByteChannel} is closed.
-   *
-   * Invariant: Unclosed channels own a reference to the base channel,
-   * allowing us to make {@link #close} idempotent.
-   *
-   * This i

[24/50] [abbrv] incubator-beam git commit: Proto2Coder: recompute the extension registry when mutated

[jamesmalone]

Proto2Coder: recompute the extension registry when mutated

Proto2Coder's deprecated methods mutate the existing coder's
extension host set, but it may already have been memoized.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115407399


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6926d8ee
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6926d8ee
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6926d8ee

Branch: refs/heads/master
Commit: 6926d8ee1d5fa2afece0024e64e18dc06ec857c4
Parents: 96b02f4
Author: dhalperi 
Authored: Tue Feb 23 18:38:04 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../google/cloud/dataflow/sdk/coders/Proto2Coder.java   | 12 +---
 1 file changed, 9 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6926d8ee/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
index b107f37..56ec0c7 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/coders/Proto2Coder.java
@@ -198,6 +198,11 @@ public class Proto2Coder extends 
AtomicCoder {
   }
   extensionHostClasses.add(extensionHost);
 }
+// The memoized extension registry needs to be recomputed because we have 
mutated this object.
+synchronized (this) {
+  memoizedExtensionRegistry = null;
+  getExtensionRegistry();
+}
 return this;
   }
 
@@ -293,18 +298,19 @@ public class Proto2Coder extends 
AtomicCoder {
 
   private transient ExtensionRegistry memoizedExtensionRegistry;
 
-  private ExtensionRegistry getExtensionRegistry() {
+  private synchronized ExtensionRegistry getExtensionRegistry() {
 if (memoizedExtensionRegistry == null) {
-  memoizedExtensionRegistry = ExtensionRegistry.newInstance();
+  ExtensionRegistry registry = ExtensionRegistry.newInstance();
   for (Class extensionHost : extensionHostClasses) {
 try {
   extensionHost
   .getDeclaredMethod("registerAllExtensions", 
ExtensionRegistry.class)
-  .invoke(null, memoizedExtensionRegistry);
+  .invoke(null, registry);
 } catch (IllegalAccessException | InvocationTargetException | 
NoSuchMethodException e) {
   throw new IllegalStateException(e);
 }
   }
+  memoizedExtensionRegistry = registry.getUnmodifiable();
 }
 return memoizedExtensionRegistry;
   }

[37/50] [abbrv] incubator-beam git commit: Finish Flattenning InProcess package

[jamesmalone]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/06c8911e/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManagerTest.java
--
diff --git 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManagerTest.java
 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManagerTest.java
deleted file mode 100644
index d4979f2..000
--- 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/util/InMemoryWatermarkManagerTest.java
+++ /dev/null
@@ -1,1099 +0,0 @@
-/*
- * Copyright (C) 2015 Google Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy 
of
- * the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations 
under
- * the License.
- */
-package com.google.cloud.dataflow.sdk.runners.inprocess.util;
-
-import static org.hamcrest.Matchers.contains;
-import static org.hamcrest.Matchers.containsInAnyOrder;
-import static org.hamcrest.Matchers.emptyIterable;
-import static org.hamcrest.Matchers.equalTo;
-import static org.hamcrest.Matchers.not;
-import static org.hamcrest.Matchers.nullValue;
-import static org.junit.Assert.assertThat;
-
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.UncommittedBundle;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.FiredTimers;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.TimerUpdate;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.TimerUpdate.TimerUpdateBuilder;
-import 
com.google.cloud.dataflow.sdk.runners.inprocess.util.InMemoryWatermarkManager.TransformWatermarks;
-import com.google.cloud.dataflow.sdk.testing.TestPipeline;
-import com.google.cloud.dataflow.sdk.transforms.AppliedPTransform;
-import com.google.cloud.dataflow.sdk.transforms.Create;
-import com.google.cloud.dataflow.sdk.transforms.DoFn;
-import com.google.cloud.dataflow.sdk.transforms.Filter;
-import com.google.cloud.dataflow.sdk.transforms.Flatten;
-import com.google.cloud.dataflow.sdk.transforms.ParDo;
-import com.google.cloud.dataflow.sdk.transforms.WithKeys;
-import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
-import com.google.cloud.dataflow.sdk.util.TimeDomain;
-import com.google.cloud.dataflow.sdk.util.TimerInternals.TimerData;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.util.state.StateNamespaces;
-import com.google.cloud.dataflow.sdk.values.KV;
-import com.google.cloud.dataflow.sdk.values.PCollection;
-import com.google.cloud.dataflow.sdk.values.PCollectionList;
-import com.google.cloud.dataflow.sdk.values.PValue;
-import com.google.cloud.dataflow.sdk.values.TimestampedValue;
-import com.google.common.collect.ImmutableList;
-
-import org.hamcrest.BaseMatcher;
-import org.hamcrest.Description;
-import org.hamcrest.Matcher;
-import org.joda.time.Instant;
-import org.joda.time.ReadableInstant;
-import org.junit.Before;
-import org.junit.Test;
-import org.junit.runner.RunWith;
-import org.junit.runners.JUnit4;
-
-import java.io.Serializable;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-
-/**
- * Tests for {@link InMemoryWatermarkManager}.
- */
-@RunWith(JUnit4.class)
-public class InMemoryWatermarkManagerTest implements Serializable {
-  private transient MockClock clock;
-
-  private transient PCollection createdInts;
-
-  private transient PCollection filtered;
-  private transient PCollection filteredTimesTwo;
-  private transient PCollection> keyed;
-
-  private transient PCollection intsToFlatten;
-  private transient PCollection flattened;
-
-  private transient InMemoryWatermarkManager manager;
-
-  @Before
-  public void setup() {
-TestPipeline p = TestPipeline.create();
-
-createdInts = p.apply("createdInts", Create.of(1, 2, 3));
-
-filtered = createdInts.apply("filtered", Filter.greaterThan(1));
-filteredTimesTwo = filtered.apply("timesTwo", ParDo.of(new DoFn() {
-  @Override
-  public void processElement(DoFn.ProcessContext c) 
throws Exception {
-c.output(c.element() * 2);
-  }
-}));
-
-keyed = createdInts.apply("keyed", WithKeys.of("MyKey"));
-
-intsToFlatten = p.apply("intsToFlatten", Create.of(-1, 256, 65535));
-PCollectionList preFla

[09/50] [abbrv] incubator-beam git commit: Revert "Migrate AvroIO.Write to a custom sink"

[jamesmalone]

Revert "Migrate AvroIO.Write to a custom sink"

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115355272


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2e89a4b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2e89a4b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2e89a4b3

Branch: refs/heads/master
Commit: 2e89a4b3ddfd5d395aa8d6c4f73b501712b907a7
Parents: 635541a
Author: sgmc 
Authored: Tue Feb 23 10:21:54 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../google/cloud/dataflow/sdk/io/AvroIO.java| 188 ++-
 .../sdk/runners/DataflowPipelineRunner.java | 179 --
 .../sdk/runners/DataflowPipelineTranslator.java |   5 +
 .../sdk/runners/dataflow/AvroIOTranslator.java  |  87 +
 .../sdk/io/AvroIOGeneratedClassTest.java|  48 ++---
 .../cloud/dataflow/sdk/io/AvroIOTest.java   |  45 -
 6 files changed, 215 insertions(+), 337 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2e89a4b3/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
index 7042010..9ee7e6b 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
@@ -22,25 +22,24 @@ import com.google.cloud.dataflow.sdk.coders.AvroCoder;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.VoidCoder;
 import com.google.cloud.dataflow.sdk.io.Read.Bounded;
-import com.google.cloud.dataflow.sdk.options.PipelineOptions;
+import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
 import com.google.cloud.dataflow.sdk.runners.PipelineRunner;
+import com.google.cloud.dataflow.sdk.runners.worker.AvroSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.MimeTypes;
+import com.google.cloud.dataflow.sdk.util.WindowedValue;
+import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 import com.google.cloud.dataflow.sdk.values.PInput;
-import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 
 import org.apache.avro.Schema;
-import org.apache.avro.file.DataFileWriter;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.reflect.ReflectData;
 
 import java.io.IOException;
-import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
+import java.util.List;
 import java.util.regex.Pattern;
 
 import javax.annotation.Nullable;
@@ -318,7 +317,7 @@ public class AvroIO {
 : com.google.cloud.dataflow.sdk.io.Read.from(
 AvroSource.from(filepattern).withSchema(type));
 
-PCollection pcol = input.getPipeline().apply("Read", read);
+PCollection pcol = input.getPipeline().apply(read);
 // Honor the default output coder that would have been used by this 
PTransform.
 pcol.setCoder(getDefaultOutputCoder());
 return pcol;
@@ -474,6 +473,8 @@ public class AvroIO {
   final int numShards;
   /** Shard template string. */
   final String shardTemplate;
+  /** Insert a shuffle before writing to decouple parallelism when 
numShards != 0. */
+  final boolean forceReshard;
   /** The class type of the records. */
   final Class type;
   /** The schema of the output file. */
@@ -483,16 +484,18 @@ public class AvroIO {
   final boolean validate;
 
   Bound(Class type) {
-this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, type, null, 
true);
+this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, true, type, 
null, true);
   }
 
   Bound(String name, String filenamePrefix, String filenameSuffix, int 
numShards,
-  String shardTemplate, Class type, Schema schema, boolean 
validate) {
+  String shardTemplate, boolean forceReshard, Class type, Schema 
schema,
+  boolean validate) {
 super(name);
 this.filenamePrefix = filenamePrefix;
 this.filenameSuffix = filenameSuffix;
 this.numShards = numShards;
 this.shardTemplate = shardTemplate;
+this.forceReshard = forceReshard;
 this.type = type;
 this.schema = schema;
 this.validate = validate;
@@ -506,7 +509,7 @@ public c

[36/50] [abbrv] incubator-beam git commit: Switch to the start state when lazily initializing

[jamesmalone]

Switch to the start state when lazily initializing

Previously, we would attribute time spent running the startBundle of a
DoFn as time spent in -process state.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115604508


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/31116460
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/31116460
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/31116460

Branch: refs/heads/master
Commit: 31116460fb4f6c47d48e01c507389f4eb8f8b3cd
Parents: fba9147
Author: bchambers 
Authored: Thu Feb 25 14:26:03 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../sdk/util/common/worker/StateSampler.java| 47 +---
 1 file changed, 32 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/31116460/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/common/worker/StateSampler.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/common/worker/StateSampler.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/common/worker/StateSampler.java
index df916a0..00d3b3b 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/common/worker/StateSampler.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/common/worker/StateSampler.java
@@ -67,7 +67,7 @@ public class StateSampler implements AutoCloseable {
   private volatile int currentState;
 
   /** Special value of {@code currentState} that means we do not sample. */
-  private static final int DO_NOT_SAMPLE = -1;
+  public static final int DO_NOT_SAMPLE = -1;
 
   /**
* A counter that increments with each state transition. May be used
@@ -113,6 +113,30 @@ public class StateSampler implements AutoCloseable {
 this.prefix = prefix;
 this.counterSetMutator = counterSetMutator;
 currentState = DO_NOT_SAMPLE;
+scheduleSampling(samplingPeriodMs);
+  }
+
+  /**
+   * Constructs a new {@link StateSampler} that can be used to obtain
+   * an approximate breakdown of the time spent by an execution
+   * context in various states, as a fraction of the total time.
+   *
+   * @param prefix the prefix of the counter names for the states
+   * @param counterSetMutator the {@link CounterSet.AddCounterMutator}
+   * used to create a counter for each distinct state
+   */
+  public StateSampler(String prefix,
+  CounterSet.AddCounterMutator counterSetMutator) {
+this(prefix, counterSetMutator, DEFAULT_SAMPLING_PERIOD_MS);
+  }
+
+  /**
+   * Called by the constructor to schedule sampling at the given period.
+   *
+   * Should not be overridden by sub-classes unless they want to change
+   * or disable the automatic sampling of state.
+   */
+  protected void scheduleSampling(final long samplingPeriodMs) {
 // Here "stratified sampling" is used, which makes sure that there's 1 
uniformly chosen sampled
 // point in every bucket of samplingPeriodMs, to prevent pathological 
behavior in case some
 // states happen to occur at a similar period.
@@ -148,20 +172,6 @@ public class StateSampler implements AutoCloseable {
 TimeUnit.MILLISECONDS);
   }
 
-  /**
-   * Constructs a new {@link StateSampler} that can be used to obtain
-   * an approximate breakdown of the time spent by an execution
-   * context in various states, as a fraction of the total time.
-   *
-   * @param prefix the prefix of the counter names for the states
-   * @param counterSetMutator the {@link CounterSet.AddCounterMutator}
-   * used to create a counter for each distinct state
-   */
-  public StateSampler(String prefix,
-  CounterSet.AddCounterMutator counterSetMutator) {
-this(prefix, counterSetMutator, DEFAULT_SAMPLING_PERIOD_MS);
-  }
-
   public synchronized void run() {
 long startTimestampNs = System.nanoTime();
 int state = currentState;
@@ -255,6 +265,13 @@ public class StateSampler implements AutoCloseable {
   }
 
   /**
+   * Returns the current state of this state sampler.
+   */
+  public int getCurrentState() {
+return currentState;
+  }
+
+  /**
* Sets the current thread state.
*
* @param state the new state to transition to

[06/50] [abbrv] incubator-beam git commit: Encode elements in InProcessCreate

[jamesmalone]

Encode elements in InProcessCreate

There is no requirement that the elements of a Create are seralizable -
however, they must be encodeable; because applying a read will ensure
the elements are seralizable, we must ensure that all elements succeed;
do so via encoding all of the available elements into the source, and
decoding them in the reader.

Implement getEstimatedSizeBytes and splitIntoBundles, now that we know
the size of the elements in bytes.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115256859


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/209364e6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/209364e6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/209364e6

Branch: refs/heads/master
Commit: 209364e6ac6890845c9a51253e6c97b31ad2191c
Parents: d58b7db
Author: tgroh 
Authored: Mon Feb 22 12:21:09 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../sdk/runners/inprocess/InProcessCreate.java  | 154 ---
 .../runners/inprocess/InProcessCreateTest.java  | 126 +++
 2 files changed, 223 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/209364e6/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessCreate.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessCreate.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessCreate.java
index 0ff881f..9023b7b 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessCreate.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/InProcessCreate.java
@@ -18,35 +18,39 @@ package com.google.cloud.dataflow.sdk.runners.inprocess;
 import com.google.cloud.dataflow.sdk.coders.CannotProvideCoderException;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.CoderException;
-import com.google.cloud.dataflow.sdk.coders.StandardCoder;
 import com.google.cloud.dataflow.sdk.io.BoundedSource;
 import com.google.cloud.dataflow.sdk.io.Read;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.transforms.Create;
 import com.google.cloud.dataflow.sdk.transforms.Create.Values;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
+import com.google.cloud.dataflow.sdk.util.CoderUtils;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PInput;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Optional;
+import com.google.common.base.Throwables;
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Iterators;
 import com.google.common.collect.PeekingIterator;
 
 import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.util.Collections;
+import java.util.ArrayList;
+import java.util.Collection;
 import java.util.List;
 import java.util.NoSuchElementException;
 
+import javax.annotation.Nullable;
+
 /**
- * An in memory implementation of the {@link Values Create.Values} {@link 
PTransform}, implemented'
+ * An in-process implementation of the {@link Values Create.Values} {@link 
PTransform}, implemented
  * using a {@link BoundedSource}.
  *
  * The coder is inferred via the {@link Values#getDefaultOutputCoder(PInput)} 
method on the original
  * transform.
  */
-class InProcessCreate extends PTransform> {
+class InProcessCreate extends ForwardingPTransform> {
   private final Create.Values original;
-  private final InMemorySource source;
 
   public static  InProcessCreate from(Create.Values original) {
 return new InProcessCreate<>(original);
@@ -54,38 +58,88 @@ class InProcessCreate extends PTransform> {
 
   private InProcessCreate(Values original) {
 this.original = original;
-this.source = new InMemorySource<>(original.getElements());
   }
 
   @Override
   public PCollection apply(PInput input) {
-input.getPipeline().getCoderRegistry();
-PCollection result = input.getPipeline().apply(Read.from(source));
+Coder elementCoder;
 try {
-  result.setCoder(original.getDefaultOutputCoder(input));
+  elementCoder = original.getDefaultOutputCoder(input);
 } catch (CannotProvideCoderException e) {
-  throw new IllegalArgumentException("Unable to infer a coder and no Coder 
was specified. "
-  + "Please set a coder by invoking Create.withCoder() explicitly.", 
e);
+  throw new IllegalArgumentException(
+  "Unable to infer a coder and no 

[23/50] [abbrv] incubator-beam git commit: Support CombineFnWithContext in GroupAlsoByWindows

[jamesmalone]

Support CombineFnWithContext in GroupAlsoByWindows

This requires plumbing pipeline options and side inputs to State API, including:
1. Adding bindKeyedCombiningValueWithContext() to StateTag.java
2. Adding StateContext to StateInternals.java
3. Plumbing through the remaining files

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115408034


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6613031b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6613031b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6613031b

Branch: refs/heads/master
Commit: 6613031bfed700567bdc2d4eab6b754c4392ce77
Parents: 6926d8e
Author: peihe 
Authored: Tue Feb 23 18:49:00 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../sdk/util/CombineContextFactory.java |  18 +++
 .../cloud/dataflow/sdk/util/CombineFnUtil.java  |  97 +
 .../cloud/dataflow/sdk/util/DoFnRunnerBase.java |   5 +
 .../util/GroupAlsoByWindowViaWindowSetDoFn.java |   3 +-
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   3 +-
 .../sdk/util/ReduceFnContextFactory.java|  64 +
 .../cloud/dataflow/sdk/util/ReduceFnRunner.java |   6 +-
 .../cloud/dataflow/sdk/util/SystemReduceFn.java |  19 +--
 .../sdk/util/TriggerContextFactory.java |   4 +-
 .../cloud/dataflow/sdk/util/TriggerRunner.java  |   9 +-
 .../dataflow/sdk/util/WindowingInternals.java   |   5 +
 .../CopyOnAccessInMemoryStateInternals.java |  72 +++---
 .../sdk/util/state/InMemoryStateInternals.java  |  35 +++--
 .../dataflow/sdk/util/state/StateContext.java   |  41 ++
 .../dataflow/sdk/util/state/StateContexts.java  | 107 +++
 .../dataflow/sdk/util/state/StateInternals.java |   7 +
 .../dataflow/sdk/util/state/StateTable.java |   8 +-
 .../cloud/dataflow/sdk/util/state/StateTag.java |   7 +
 .../dataflow/sdk/util/state/StateTags.java  |  83 ++-
 .../dataflow/sdk/util/CombineFnUtilTest.java|  62 +
 .../dataflow/sdk/util/ReduceFnRunnerTest.java   | 137 ++-
 .../cloud/dataflow/sdk/util/ReduceFnTester.java |  58 +++-
 22 files changed, 763 insertions(+), 87 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6613031b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineContextFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineContextFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineContextFactory.java
index bf09587..6f2b89b 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineContextFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineContextFactory.java
@@ -19,6 +19,7 @@ import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.transforms.CombineWithContext.Context;
 import com.google.cloud.dataflow.sdk.transforms.DoFn;
 import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
+import com.google.cloud.dataflow.sdk.util.state.StateContext;
 import com.google.cloud.dataflow.sdk.values.PCollectionView;
 
 /**
@@ -63,6 +64,23 @@ public class CombineContextFactory {
   }
 
   /**
+   * Returns a {@code Combine.Context} that wraps a {@link StateContext}.
+   */
+  public static Context createFromStateContext(final StateContext c) {
+return new Context() {
+  @Override
+  public PipelineOptions getPipelineOptions() {
+return c.getPipelineOptions();
+  }
+
+  @Override
+  public  T sideInput(PCollectionView view) {
+return c.sideInput(view);
+  }
+};
+  }
+
+  /**
* Returns a {@code Combine.Context} from {@code PipelineOptions}, {@code 
SideInputReader},
* and the main input window.
*/

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6613031b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineFnUtil.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineFnUtil.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineFnUtil.java
new file mode 100644
index 000..6201e6e
--- /dev/null
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/CombineFnUtil.java
@@ -0,0 +1,97 @@
+
+/*
+ * Copyright (C) 2016 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing

[15/50] [abbrv] incubator-beam git commit: Make TestPipeline slightly less DataflowPipelineRunner-centric

[jamesmalone]

Make TestPipeline slightly less DataflowPipelineRunner-centric

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115302769


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/13a042ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/13a042ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/13a042ae

Branch: refs/heads/master
Commit: 13a042aed7d01a126a1ff7ecb66e723474191fe0
Parents: c0a814b
Author: klk 
Authored: Mon Feb 22 21:17:40 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../dataflow/sdk/testing/TestPipeline.java  | 69 
 .../dataflow/sdk/testing/TestPipelineTest.java  | 18 +++--
 2 files changed, 53 insertions(+), 34 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/13a042ae/sdk/src/main/java/com/google/cloud/dataflow/sdk/testing/TestPipeline.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/testing/TestPipeline.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/testing/TestPipeline.java
index 05b5bad..a05a778 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/testing/TestPipeline.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/testing/TestPipeline.java
@@ -19,22 +19,23 @@ package com.google.cloud.dataflow.sdk.testing;
 import com.google.cloud.dataflow.sdk.Pipeline;
 import com.google.cloud.dataflow.sdk.PipelineResult;
 import com.google.cloud.dataflow.sdk.options.ApplicationNameOptions;
+import com.google.cloud.dataflow.sdk.options.GcpOptions;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions.CheckEnabled;
 import com.google.cloud.dataflow.sdk.options.PipelineOptionsFactory;
-import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
+import com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner;
 import com.google.cloud.dataflow.sdk.runners.PipelineRunner;
+import com.google.cloud.dataflow.sdk.util.TestCredential;
 import com.google.common.base.Optional;
 import com.google.common.collect.Iterators;
 
 import com.fasterxml.jackson.databind.ObjectMapper;
 
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
 import java.io.IOException;
 import java.util.Iterator;
 
+import javax.annotation.Nullable;
+
 /**
  * A creator of test pipelines that can be used inside of tests that can be
  * configured to run locally or against the live service.
@@ -67,7 +68,6 @@ import java.util.Iterator;
  */
 public class TestPipeline extends Pipeline {
   private static final String PROPERTY_DATAFLOW_OPTIONS = "dataflowOptions";
-  private static final Logger LOG = 
LoggerFactory.getLogger(TestPipeline.class);
   private static final ObjectMapper MAPPER = new ObjectMapper();
 
   /**
@@ -77,28 +77,22 @@ public class TestPipeline extends Pipeline {
* {@link Pipeline#run} to execute the pipeline and check the tests.
*/
   public static TestPipeline create() {
-if (isIntegrationTest()) {
-  TestDataflowPipelineOptions options = getPipelineOptions();
-  LOG.info("Using passed in options: " + options);
-  options.setStableUniqueNames(CheckEnabled.ERROR);
-  return new TestPipeline(TestDataflowPipelineRunner.fromOptions(options), 
options);
-} else {
-  DirectPipelineRunner directRunner = DirectPipelineRunner.createForTest();
-  directRunner.getPipelineOptions().setAppName(getAppName());
-  
directRunner.getPipelineOptions().setStableUniqueNames(CheckEnabled.ERROR);
-  return new TestPipeline(directRunner, directRunner.getPipelineOptions());
-}
+return fromOptions(testingPipelineOptions());
+  }
+
+  public static TestPipeline fromOptions(PipelineOptions options) {
+return new TestPipeline(PipelineRunner.fromOptions(options), options);
   }
 
   /**
-   * Returns whether this test is running on the Cloud Dataflow service as 
described
-   * in {@link TestPipeline}.
+   * Returns whether a {@link TestPipeline} supports dynamic work rebalancing, 
and thus tests
+   * of dynamic work rebalancing are expected to pass.
*/
-  public static boolean isIntegrationTest() {
-return 
Boolean.parseBoolean(System.getProperty("runIntegrationTestOnService"));
+  public boolean supportsDynamicWorkRebalancing() {
+return getRunner() instanceof DataflowPipelineRunner;
   }
 
-  TestPipeline(PipelineRunner runner, 
PipelineOptions options) {
+  private TestPipeline(PipelineRunner runner, 
PipelineOptions options) {
 super(runner, options);
   }
 
@@ -126,14 +120,28 @@ public class TestPipeline extends Pipeline {
   }
 
   /**
-   * Creates PipelineOptions for testing wit

[12/50] [abbrv] incubator-beam git commit: Change visibility of FileBasedSource subclass methods and fix return types for existing subclasses.

[jamesmalone]

Change visibility of FileBasedSource subclass methods and fix return types for 
existing subclasses.

Limit createForSubrangeOfFile and createSingleFileReader to
protected for FileBasedSource and all of its subclasses.
Also change the return type for subclasses to return
FileBasedSource and FileBasedReader instead of the specific
types.

Note that these methods were meant to be used only by the
FileBasedSource/BlockBasedSource and not called by users
but if they did, this change would be backwards incompatible
because of the return type and visibility changes.

This is for Apache Beam.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115301831


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c0a814b8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c0a814b8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c0a814b8

Branch: refs/heads/master
Commit: c0a814b89bcbcb3541591f7eccd7db5ada2b0a17
Parents: 24288e7
Author: lcwik 
Authored: Mon Feb 22 21:01:59 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java  | 4 ++--
 .../java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java | 4 ++--
 .../main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java   | 4 ++--
 .../com/google/cloud/dataflow/sdk/io/FileBasedSourceTest.java   | 5 +++--
 4 files changed, 9 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c0a814b8/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java
index da8458c..297663e 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroSource.java
@@ -242,7 +242,7 @@ public class AvroSource extends BlockBasedSource {
   }
 
   @Override
-  public AvroSource createForSubrangeOfFile(String fileName, long start, 
long end) {
+  public BlockBasedSource createForSubrangeOfFile(String fileName, long 
start, long end) {
 byte[] syncMarker = this.syncMarker;
 String codec = this.codec;
 String readSchemaString = this.readSchemaString;
@@ -274,7 +274,7 @@ public class AvroSource extends BlockBasedSource {
   }
 
   @Override
-  public AvroReader createSingleFileReader(PipelineOptions options) {
+  protected BlockBasedReader createSingleFileReader(PipelineOptions 
options) {
 return new AvroReader(this);
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c0a814b8/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java
index 9ef4cdc..f4a9c7d 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/BlockBasedSource.java
@@ -81,14 +81,14 @@ public abstract class BlockBasedSource extends 
FileBasedSource {
* Creates a {@code BlockBasedSource} for the specified range in a single 
file.
*/
   @Override
-  public abstract BlockBasedSource createForSubrangeOfFile(
+  protected abstract BlockBasedSource createForSubrangeOfFile(
   String fileName, long start, long end);
 
   /**
* Creates a {@code BlockBasedReader}.
*/
   @Override
-  public abstract BlockBasedReader createSingleFileReader(PipelineOptions 
options);
+  protected abstract BlockBasedReader 
createSingleFileReader(PipelineOptions options);
 
   /**
* A {@code Block} represents a block of records that can be read.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c0a814b8/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
index 668361b..d684d22 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
@@ -183,13 +183,13 @@ public class XmlSource extends FileBasedSource {
   }
 
   @Override
-  public FileBasedSource createForSubrangeOfFile(String fileName, long 
start, long end) {
+  protected FileBasedSource createForSubrangeOfFile(String fileName, long 
start, long end) {
 return ne

[03/50] [abbrv] incubator-beam git commit: Migrate AvroIO.Write to a custom sink

[jamesmalone]

Migrate AvroIO.Write to a custom sink

Note for user requested sharding limits to be supported,
each pipeline runner must support applying those sharding limits.

Google Cloud Dataflow supports sharding limits.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115282027


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c11af5fb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c11af5fb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c11af5fb

Branch: refs/heads/master
Commit: c11af5fbf37accb49d73bcf206b15f7cc59636bc
Parents: 2e171b5
Author: lcwik 
Authored: Mon Feb 22 16:09:35 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../google/cloud/dataflow/sdk/io/AvroIO.java| 188 +--
 .../sdk/runners/DataflowPipelineRunner.java | 179 ++
 .../sdk/runners/DataflowPipelineTranslator.java |   5 -
 .../sdk/runners/dataflow/AvroIOTranslator.java  |  87 -
 .../sdk/io/AvroIOGeneratedClassTest.java|  48 ++---
 .../cloud/dataflow/sdk/io/AvroIOTest.java   |  45 +
 6 files changed, 337 insertions(+), 215 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c11af5fb/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
index 9ee7e6b..7042010 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
@@ -22,24 +22,25 @@ import com.google.cloud.dataflow.sdk.coders.AvroCoder;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.VoidCoder;
 import com.google.cloud.dataflow.sdk.io.Read.Bounded;
-import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
+import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.runners.PipelineRunner;
-import com.google.cloud.dataflow.sdk.runners.worker.AvroSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
+import com.google.cloud.dataflow.sdk.util.MimeTypes;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 import com.google.cloud.dataflow.sdk.values.PInput;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 
 import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.reflect.ReflectData;
 
 import java.io.IOException;
-import java.util.List;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
 import java.util.regex.Pattern;
 
 import javax.annotation.Nullable;
@@ -317,7 +318,7 @@ public class AvroIO {
 : com.google.cloud.dataflow.sdk.io.Read.from(
 AvroSource.from(filepattern).withSchema(type));
 
-PCollection pcol = input.getPipeline().apply(read);
+PCollection pcol = input.getPipeline().apply("Read", read);
 // Honor the default output coder that would have been used by this 
PTransform.
 pcol.setCoder(getDefaultOutputCoder());
 return pcol;
@@ -473,8 +474,6 @@ public class AvroIO {
   final int numShards;
   /** Shard template string. */
   final String shardTemplate;
-  /** Insert a shuffle before writing to decouple parallelism when 
numShards != 0. */
-  final boolean forceReshard;
   /** The class type of the records. */
   final Class type;
   /** The schema of the output file. */
@@ -484,18 +483,16 @@ public class AvroIO {
   final boolean validate;
 
   Bound(Class type) {
-this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, true, type, 
null, true);
+this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, type, null, 
true);
   }
 
   Bound(String name, String filenamePrefix, String filenameSuffix, int 
numShards,
-  String shardTemplate, boolean forceReshard, Class type, Schema 
schema,
-  boolean validate) {
+  String shardTemplate, Class type, Schema schema, boolean 
validate) {
 super(name);
 this.filenamePrefix = filenamePrefix;
 this.filenameSuffix = filenameSuffix;
 this.numShards = numShards;
 this.shardTemplate = shardTemplate;

[18/50] [abbrv] incubator-beam git commit: Move Google Cloud Dataflow native sinks to worker module

[jamesmalone]

Move Google Cloud Dataflow native sinks to worker module

This is for Apache Beam.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115313244


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/9f546efe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/9f546efe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/9f546efe

Branch: refs/heads/master
Commit: 9f546efebc5c9307dd85e6eec79038285cffa12b
Parents: d7b5189
Author: lcwik 
Authored: Tue Feb 23 00:37:55 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../dataflow/sdk/runners/worker/AvroSink.java   | 135 -
 .../dataflow/sdk/runners/worker/TextSink.java   | 291 ---
 .../dataflow/sdk/util/common/worker/Sink.java   |  64 
 3 files changed, 490 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/9f546efe/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
deleted file mode 100644
index b101a2b..000
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Copyright (C) 2015 Google Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy 
of
- * the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations 
under
- * the License.
- */
-
-package com.google.cloud.dataflow.sdk.runners.worker;
-
-import com.google.cloud.dataflow.sdk.coders.AvroCoder;
-import com.google.cloud.dataflow.sdk.util.CoderUtils;
-import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.MimeTypes;
-import com.google.cloud.dataflow.sdk.util.ShardingWritableByteChannel;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import 
com.google.cloud.dataflow.sdk.util.WindowedValue.ValueOnlyWindowedValueCoder;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
-
-import org.apache.avro.Schema;
-import org.apache.avro.file.DataFileWriter;
-import org.apache.avro.io.DatumWriter;
-
-import java.io.IOException;
-import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
-import java.util.ArrayList;
-import java.util.Random;
-
-/**
- * A sink that writes Avro files.
- *
- * @param  the type of the elements written to the sink
- */
-public class AvroSink extends Sink> {
-
-  final String filenamePrefix;
-  final String shardFormat;
-  final String filenameSuffix;
-  final int shardCount;
-  final AvroCoder avroCoder;
-  final Schema schema;
-
-  public AvroSink(String filename, ValueOnlyWindowedValueCoder coder) {
-this(filename, "", "", 1, coder);
-  }
-
-  public AvroSink(String filenamePrefix, String shardFormat, String 
filenameSuffix, int shardCount,
-  ValueOnlyWindowedValueCoder coder) {
-if (!(coder.getValueCoder() instanceof AvroCoder)) {
-  throw new IllegalArgumentException(String.format(
-  "AvroSink requires an AvroCoder, not a %s", 
coder.getValueCoder().getClass()));
-}
-
-this.filenamePrefix = filenamePrefix;
-this.shardFormat = shardFormat;
-this.filenameSuffix = filenameSuffix;
-this.shardCount = shardCount;
-this.avroCoder = (AvroCoder) coder.getValueCoder();
-this.schema = this.avroCoder.getSchema();
-  }
-
-  public SinkWriter> writer(DatumWriter datumWriter) 
throws IOException {
-WritableByteChannel writer = IOChannelUtils.create(
-filenamePrefix, shardFormat, filenameSuffix, shardCount, 
MimeTypes.BINARY);
-
-if (writer instanceof ShardingWritableByteChannel) {
-  return new AvroShardingFileWriter(datumWriter, 
(ShardingWritableByteChannel) writer);
-} else {
-  return new AvroFileWriter(datumWriter, writer);
-}
-  }
-
-  @Override
-  public SinkWriter> writer() throws IOException {
-return writer(avroCoder.createDatumWriter());
-  }
-
-  /** The SinkWriter for an AvroSink. */
-  class AvroFileWriter implements SinkWriter> {
-DataFileWriter fileWriter;
-
-public AvroFileWriter(DatumWriter datumWriter, WritableByteChannel 
outputChannel)
-throws I

[10/50] [abbrv] incubator-beam git commit: Revert "Migrate TextIO.Write to a custom sink"

[jamesmalone]

Revert "Migrate TextIO.Write to a custom sink"

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115351833


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3904c907
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3904c907
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3904c907

Branch: refs/heads/master
Commit: 3904c9074e66733686285d09ce5068d28f303dd8
Parents: 45f5951
Author: sgmc 
Authored: Tue Feb 23 09:53:38 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../google/cloud/dataflow/sdk/io/TextIO.java| 187 ++-
 .../sdk/runners/DataflowPipelineRunner.java | 108 +--
 .../sdk/runners/DataflowPipelineTranslator.java |   5 +
 .../sdk/runners/dataflow/TextIOTranslator.java  |  91 +
 .../cloud/dataflow/sdk/io/TextIOTest.java   |  22 +++
 .../sdk/runners/DataflowPipelineRunnerTest.java |  21 ++-
 .../runners/DataflowPipelineTranslatorTest.java |   4 +-
 .../dataflow/sdk/runners/TransformTreeTest.java |   9 +-
 8 files changed, 238 insertions(+), 209 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3904c907/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
index d342f25..0bb2861 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
@@ -26,9 +26,11 @@ import com.google.cloud.dataflow.sdk.io.Read.Bounded;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner;
 import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
+import com.google.cloud.dataflow.sdk.runners.worker.TextSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.MimeTypes;
+import com.google.cloud.dataflow.sdk.util.WindowedValue;
+import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 import com.google.cloud.dataflow.sdk.values.PInput;
@@ -37,13 +39,10 @@ import com.google.common.base.Preconditions;
 import com.google.protobuf.ByteString;
 
 import java.io.IOException;
-import java.io.OutputStream;
 import java.nio.ByteBuffer;
-import java.nio.channels.Channels;
 import java.nio.channels.ReadableByteChannel;
 import java.nio.channels.SeekableByteChannel;
-import java.nio.channels.WritableByteChannel;
-import java.nio.charset.StandardCharsets;
+import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.regex.Pattern;
 
@@ -67,7 +66,7 @@ import javax.annotation.Nullable;
  *
  * See the following examples:
  *
- * {@code
+ *  {@code
  * Pipeline p = ...;
  *
  * // A simple Read of a local file (only runs locally):
@@ -80,7 +79,7 @@ import javax.annotation.Nullable;
  * p.apply(TextIO.Read.named("ReadNumbers")
  *.from("gs://my_bucket/path/to/numbers-*.txt")
  *.withCoder(TextualIntegerCoder.of()));
- * }
+ * } 
  *
  * To write a {@link PCollection} to one or more text files, use
  * {@link TextIO.Write}, specifying {@link TextIO.Write#to(String)} to specify
@@ -95,7 +94,7 @@ import javax.annotation.Nullable;
  * will be overwritten.
  *
  * For example:
- * {@code
+ *  {@code
  * // A simple Write to a local file (only runs locally):
  * PCollection lines = ...;
  * lines.apply(TextIO.Write.to("/path/to/file.txt"));
@@ -107,7 +106,7 @@ import javax.annotation.Nullable;
  *   .to("gs://my_bucket/path/to/numbers")
  *   .withSuffix(".txt")
  *   .withCoder(TextualIntegerCoder.of()));
- * }
+ * } 
  *
  * Permissions
  * When run using the {@link DirectPipelineRunner}, your pipeline can read 
and write text files
@@ -478,6 +477,9 @@ public class TextIO {
   /** Requested number of shards. 0 for automatic. */
   private final int numShards;
 
+  /** Insert a shuffle before writing to decouple parallelism when 
numShards != 0. */
+  private final boolean forceReshard;
+
   /** The shard template of each file written, combined with prefix and 
suffix. */
   private final String shardTemplate;
 
@@ -485,16 +487,17 @@ public class TextIO {
   private final boolean validate;
 
   Bound(Coder coder) {
-this(null, null, "", coder, 0, Shar

[45/50] [abbrv] incubator-beam git commit: Update worker harness container image

[jamesmalone]

Update worker harness container image

Pin version to the GitHub push 20160225-00.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115626265


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d4dcaaa0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d4dcaaa0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d4dcaaa0

Branch: refs/heads/master
Commit: d4dcaaa0500a6dbd314fe7119d73c461b2de17f7
Parents: c290b5e
Author: davor 
Authored: Thu Feb 25 18:01:30 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:29 2016 -0800

--
 .../cloud/dataflow/sdk/runners/DataflowPipelineRunner.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d4dcaaa0/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
index 06b2295..6eb6c2f 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
@@ -207,9 +207,9 @@ public class DataflowPipelineRunner extends 
PipelineRunner
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
   public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/java-batch:INVALID";
+  = "dataflow.gcr.io/v1beta3/java-batch:github-20160225-00";
   public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
-  = "dataflow.gcr.io/v1beta3/java-streaming:INVALID";
+  = "dataflow.gcr.io/v1beta3/java-streaming:github-20160225-00";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

[29/50] [abbrv] incubator-beam git commit: Set worker harness container image to INVALID until next release

[jamesmalone]

Set worker harness container image to INVALID until next release

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115473121


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1cc0211c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1cc0211c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1cc0211c

Branch: refs/heads/master
Commit: 1cc0211c19910d1d67d28ade1fd0c044bec8421d
Parents: d15d924
Author: herohde 
Authored: Wed Feb 24 11:20:32 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 pom.xml |   1 -
 .../sdk/runners/DataflowPipelineRunner.java |   9 +-
 worker/pom.xml  | 301 ---
 3 files changed, 4 insertions(+), 307 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cc0211c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index d946605..760a10d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -88,7 +88,6 @@
   pom
   
 sdk
-worker
 examples
 maven-archetypes/starter
 maven-archetypes/examples

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cc0211c/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
index ac0dcea..54fadea 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DataflowPipelineRunner.java
@@ -206,11 +206,10 @@ public class DataflowPipelineRunner extends 
PipelineRunner
 
   // Default Docker container images that execute Dataflow worker harness, 
residing in Google
   // Container Registry, separately for Batch and Streaming.
-  // TODO: Set these once versioned containers are ready.
-  public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE = null;
-  //"dataflow.gcr.io/v1beta3/java-batch:20160201-rc00---INVALID";
-  public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE = null;
-  //"dataflow.gcr.io/v1beta3/java-streaming:20160201-rc00---INVALID";
+  public static final String BATCH_WORKER_HARNESS_CONTAINER_IMAGE
+  = "dataflow.gcr.io/v1beta3/java-batch:INVALID";
+  public static final String STREAMING_WORKER_HARNESS_CONTAINER_IMAGE
+  = "dataflow.gcr.io/v1beta3/java-streaming:INVALID";
 
   // The limit of CreateJob request size.
   private static final int CREATE_JOB_REQUEST_LIMIT_BYTES = 10 * 1024 * 1024;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1cc0211c/worker/pom.xml
--
diff --git a/worker/pom.xml b/worker/pom.xml
deleted file mode 100644
index ab9e172..000
--- a/worker/pom.xml
+++ /dev/null
@@ -1,301 +0,0 @@
-
-
-http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
-  4.0.0
-
-  
-com.google.cloud.dataflow
-google-cloud-dataflow-java-sdk-parent
-1.5.0-SNAPSHOT
-  
-
-  com.google.cloud.dataflow
-  google-cloud-dataflow-java-worker-all
-  Google Cloud Dataflow Java Worker - All
-  Google Cloud Dataflow Java SDK provides a simple, Java-based
-interface for processing virtually any size data using Google cloud
-resources. This artifact includes entire Dataflow Java 
Worker.
-  http://cloud.google.com/dataflow
-
-  jar
-
-  
-${maven.build.timestamp}
--MM-dd 
HH:mm
-none
-  
-
-  
-
-  
-src/main/resources
-true
-  
-
-
-
-  
-org.apache.maven.plugins
-maven-compiler-plugin
-  
-
-  
-  
-org.apache.maven.plugins
-maven-checkstyle-plugin
-2.12
-
-  
-com.puppycrawl.tools
-checkstyle
-6.6
-  
-
-
-  ../checkstyle.xml
-  true
-  true
-  false
-  true
-  
${project.build.directory}/generated-test-sources/**
-
-
-  
-
-  check
-
-  
-
-  
-
-  
-org.apache.maven.plugins
-maven-jar-plugin
-
-  
-default-jar
-
-  jar
-
-  
-  
-default-test-jar
- 

[41/50] [abbrv] incubator-beam git commit: Rollback reverts "Move Google Cloud Dataflow native sinks to worker module"

[jamesmalone]

Rollback reverts "Move Google Cloud Dataflow native sinks to worker module"

This is for Apache Beam.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115507028


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7b28d235
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7b28d235
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7b28d235

Branch: refs/heads/master
Commit: 7b28d235000b9232a4fe55ebde76f77be26bd094
Parents: ca98da2
Author: lcwik 
Authored: Wed Feb 24 16:27:52 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../dataflow/sdk/runners/worker/AvroSink.java   | 135 -
 .../dataflow/sdk/runners/worker/TextSink.java   | 291 ---
 .../dataflow/sdk/util/common/worker/Sink.java   |  64 
 3 files changed, 490 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7b28d235/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
deleted file mode 100644
index b101a2b..000
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Copyright (C) 2015 Google Inc.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"); you may not
- * use this file except in compliance with the License. You may obtain a copy 
of
- * the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
- * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
- * License for the specific language governing permissions and limitations 
under
- * the License.
- */
-
-package com.google.cloud.dataflow.sdk.runners.worker;
-
-import com.google.cloud.dataflow.sdk.coders.AvroCoder;
-import com.google.cloud.dataflow.sdk.util.CoderUtils;
-import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.MimeTypes;
-import com.google.cloud.dataflow.sdk.util.ShardingWritableByteChannel;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import 
com.google.cloud.dataflow.sdk.util.WindowedValue.ValueOnlyWindowedValueCoder;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
-
-import org.apache.avro.Schema;
-import org.apache.avro.file.DataFileWriter;
-import org.apache.avro.io.DatumWriter;
-
-import java.io.IOException;
-import java.nio.channels.Channels;
-import java.nio.channels.WritableByteChannel;
-import java.util.ArrayList;
-import java.util.Random;
-
-/**
- * A sink that writes Avro files.
- *
- * @param  the type of the elements written to the sink
- */
-public class AvroSink extends Sink> {
-
-  final String filenamePrefix;
-  final String shardFormat;
-  final String filenameSuffix;
-  final int shardCount;
-  final AvroCoder avroCoder;
-  final Schema schema;
-
-  public AvroSink(String filename, ValueOnlyWindowedValueCoder coder) {
-this(filename, "", "", 1, coder);
-  }
-
-  public AvroSink(String filenamePrefix, String shardFormat, String 
filenameSuffix, int shardCount,
-  ValueOnlyWindowedValueCoder coder) {
-if (!(coder.getValueCoder() instanceof AvroCoder)) {
-  throw new IllegalArgumentException(String.format(
-  "AvroSink requires an AvroCoder, not a %s", 
coder.getValueCoder().getClass()));
-}
-
-this.filenamePrefix = filenamePrefix;
-this.shardFormat = shardFormat;
-this.filenameSuffix = filenameSuffix;
-this.shardCount = shardCount;
-this.avroCoder = (AvroCoder) coder.getValueCoder();
-this.schema = this.avroCoder.getSchema();
-  }
-
-  public SinkWriter> writer(DatumWriter datumWriter) 
throws IOException {
-WritableByteChannel writer = IOChannelUtils.create(
-filenamePrefix, shardFormat, filenameSuffix, shardCount, 
MimeTypes.BINARY);
-
-if (writer instanceof ShardingWritableByteChannel) {
-  return new AvroShardingFileWriter(datumWriter, 
(ShardingWritableByteChannel) writer);
-} else {
-  return new AvroFileWriter(datumWriter, writer);
-}
-  }
-
-  @Override
-  public SinkWriter> writer() throws IOException {
-return writer(avroCoder.createDatumWriter());
-  }
-
-  /** The SinkWriter for an AvroSink. */
-  class AvroFileWriter implements SinkWriter> {
-DataFileWriter fileWriter;
-
-public AvroFileWriter(DatumWriter datumWriter, WritableByteChannel 
outputChannel

[43/50] [abbrv] incubator-beam git commit: Fix worker dependencies and turn on strict checking

[jamesmalone]

Fix worker dependencies and turn on strict checking

Here is a pruned and edited tree diff of the dependency:tree output. It
is formatted to emphasize the packages that were SDK deps that are now
direct deps. Many of these are actually provided by lots of transitive
dependencies.

 [INFO] com.google.cloud.dataflow:google-cloud-dataflow-java-worker-all
 [INFO] +- com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all

-[INFO] |  +- com.google.api-client:google-api-client:jar:1.21.0
-[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev248-1...
-[INFO] |  +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev19-...
-[INFO] |  +- com.google.cloud.dataflow:google-cloud-dataflow-java-proto-l...
-[INFO] |  +- com.google.http-client:google-http-client:jar:1.19.0
-[INFO] |  +- com.google.http-client:google-http-client-jackson2:jar:1.19
-[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1
-[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0
-[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.7.0
-[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0
-[INFO] |  +- org.apache.avro:avro:jar:1.7.7
-[INFO] |  +- joda-time:joda-time:jar:2.4
-[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.14

+[INFO] +- com.google.api-client:google-api-client:jar:1.21.0
+[INFO] +- com.google.apis:google-api-services-bigquery:jar:v2-rev248-1.21...
+[INFO] +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev19-1.2...
+[INFO] +- com.google.apis:google-api-services-datastore-protobuf:jar:v1be...
+[INFO] +- com.google.cloud.dataflow:google-cloud-dataflow-java-proto-libr...
+[INFO] +- com.google.http-client:google-http-client:jar:1.21.0
+[INFO] +- com.google.http-client:google-http-client-jackson2:jar:1.21.0
+[INFO] +- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
+[INFO] +- com.fasterxml.jackson.core:jackson-annotations:jar:2.7.0
+[INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.7.0
+[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.7.0
+[INFO] +- joda-time:joda-time:jar:2.4
+[INFO] +- org.apache.avro:avro:jar:1.7.7
+[INFO] +- org.slf4j:slf4j-api:jar:1.7.14

One from gRPC:

 [INFO] +- io.grpc:grpc-all:jar:0.12.0
 [INFO] |  +- io.grpc:grpc-netty:jar:0.12.0
-[INFO] |  |  |  +- io.netty:netty-handler:jar:4.1.0.Beta8
+[INFO] +- io.netty:netty-handler:jar:4.1.0.Beta8

One from jetty:

 [INFO] +- org.eclipse.jetty:jetty-server:jar:9.2.10.v20150310
 [INFO] |  +- org.eclipse.jetty:jetty-http:jar:9.2.10.v20150310
-[INFO] |  |  \- org.eclipse.jetty:jetty-util:jar:9.2.10.v20150310
+[INFO] +- org.eclipse.jetty:jetty-util:jar:9.2.10.v20150310

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115612383


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c290b5e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c290b5e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c290b5e1

Branch: refs/heads/master
Commit: c290b5e1b4b65a74ef73889722fde285a168b346
Parents: 89e6241
Author: klk 
Authored: Thu Feb 25 15:37:18 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:29 2016 -0800

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c290b5e1/pom.xml
--
diff --git a/pom.xml b/pom.xml
index bfade1a..ba130d2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -71,6 +71,7 @@
 v2-rev248-1.21.0
 0.2.3
 v1b3-rev19-1.21.0
+0.5.160222
 v1beta2-rev1-4.0.0
 1.21.0
 19.0

[02/50] [abbrv] incubator-beam git commit: Move Google Cloud Dataflow worker utilities to worker module

[jamesmalone]

Move Google Cloud Dataflow worker utilities to worker module

SourceTranslationUtils had a component that was used for custom
source read translation. This component was moved directly into
the read translation so that the native reader source utilities
could move to the worker maven module along with the native reader
base class.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115288815


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1d0c6d0e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1d0c6d0e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1d0c6d0e

Branch: refs/heads/master
Commit: 1d0c6d0ef0ca24e2b1b7eb330dd0340f696f25ed
Parents: c11af5f
Author: lcwik 
Authored: Mon Feb 22 17:17:35 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../sdk/runners/dataflow/ReadTranslator.java|  41 ++-
 .../runners/worker/SourceTranslationUtils.java  | 150 
 .../sdk/util/common/worker/NativeReader.java| 364 ---
 3 files changed, 40 insertions(+), 515 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1d0c6d0e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/dataflow/ReadTranslator.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/dataflow/ReadTranslator.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/dataflow/ReadTranslator.java
index 47a1926..f110e84 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/dataflow/ReadTranslator.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/dataflow/ReadTranslator.java
@@ -16,8 +16,11 @@
 
 package com.google.cloud.dataflow.sdk.runners.dataflow;
 
-import static 
com.google.cloud.dataflow.sdk.runners.worker.SourceTranslationUtils.cloudSourceToDictionary;
+import static com.google.cloud.dataflow.sdk.util.Structs.addBoolean;
+import static com.google.cloud.dataflow.sdk.util.Structs.addDictionary;
+import static com.google.cloud.dataflow.sdk.util.Structs.addLong;
 
+import com.google.api.services.dataflow.model.SourceMetadata;
 import com.google.cloud.dataflow.sdk.io.FileBasedSource;
 import com.google.cloud.dataflow.sdk.io.Read;
 import com.google.cloud.dataflow.sdk.io.Source;
@@ -28,6 +31,9 @@ import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.PropertyNames;
 import com.google.cloud.dataflow.sdk.values.PValue;
 
+import java.util.HashMap;
+import java.util.Map;
+
 /**
  * Translator for the {@code Read} {@code PTransform} for the Dataflow 
back-end.
  */
@@ -61,4 +67,37 @@ public class ReadTranslator implements 
TransformTranslator> {
   throw new RuntimeException(e);
 }
   }
+
+  // Represents a cloud Source as a dictionary for encoding inside the {@code 
SOURCE_STEP_INPUT}
+  // property of CloudWorkflowStep.input.
+  private static Map cloudSourceToDictionary(
+  com.google.api.services.dataflow.model.Source source) {
+// Do not translate encoding - the source's encoding is translated 
elsewhere
+// to the step's output info.
+Map res = new HashMap<>();
+addDictionary(res, PropertyNames.SOURCE_SPEC, source.getSpec());
+if (source.getMetadata() != null) {
+  addDictionary(res, PropertyNames.SOURCE_METADATA,
+  cloudSourceMetadataToDictionary(source.getMetadata()));
+}
+if (source.getDoesNotNeedSplitting() != null) {
+  addBoolean(
+  res, PropertyNames.SOURCE_DOES_NOT_NEED_SPLITTING, 
source.getDoesNotNeedSplitting());
+}
+return res;
+  }
+
+  private static Map 
cloudSourceMetadataToDictionary(SourceMetadata metadata) {
+Map res = new HashMap<>();
+if (metadata.getProducesSortedKeys() != null) {
+  addBoolean(res, PropertyNames.SOURCE_PRODUCES_SORTED_KEYS, 
metadata.getProducesSortedKeys());
+}
+if (metadata.getEstimatedSizeBytes() != null) {
+  addLong(res, PropertyNames.SOURCE_ESTIMATED_SIZE_BYTES, 
metadata.getEstimatedSizeBytes());
+}
+if (metadata.getInfinite() != null) {
+  addBoolean(res, PropertyNames.SOURCE_IS_INFINITE, 
metadata.getInfinite());
+}
+return res;
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1d0c6d0e/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/SourceTranslationUtils.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/SourceTranslationUtils.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/SourceTranslationUtils.java
deleted file mode 100644
index 6c2c2c2..000
--- 
a/sdk/

[49/50] [abbrv] incubator-beam git commit: Remove Google-specific contribution rules

[jamesmalone]

Remove Google-specific contribution rules


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2efe7617
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2efe7617
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2efe7617

Branch: refs/heads/master
Commit: 2efe7617a595b8ac6167db40724d9874afd6df23
Parents: 3623a23
Author: Frances Perry 
Authored: Fri Feb 26 12:40:01 2016 -0800
Committer: Frances Perry 
Committed: Fri Feb 26 12:40:01 2016 -0800

--
 CONTRIBUTING.md | 31 ---
 1 file changed, 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2efe7617/CONTRIBUTING.md
--
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
deleted file mode 100644
index 8b0fcba..000
--- a/CONTRIBUTING.md
+++ /dev/null
@@ -1,31 +0,0 @@
-Want to contribute? Great! First, read this page (including the small print at
-the end).
-
-### Before you contribute
-Before we can use your code, you must sign the
-[Google Individual Contributor License 
Agreement](https://developers.google.com/open-source/cla/individual?csw=1)
-(CLA), which you can do online. The CLA is necessary mainly because you own the
-copyright to your changes, even after your contribution becomes part of our
-codebase, so we need your permission to use and distribute your code. We also
-need to be sure of various other things. For instance that you'll tell us if 
you
-know that your code infringes on other people's patents. You don't have to sign
-the CLA until after you've submitted your code for review and a member has
-approved it, but you must do it before we can put your code into our codebase.
-
-Before you start working on a larger contribution, we recommend to get in touch
-with us first through the issue tracker with your idea so that we can help out
-and possibly guide you. Coordinating up front makes it much easier to avoid
-frustration later on.
-
-### Code reviews
-All submissions, including submissions by project members, require review. We
-use GitHub pull requests for this purpose.
-
-### Organization
-During our review and triage of incoming pull requests, we'll advise whether to
-include your contribution into the mainline SDK, or to maintain it within the
-separate group of [community-contributed 
modules](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib).
-
-### The small print
-Contributions made by corporations are covered by a different agreement than
-the one above, the Software Grant and Corporate Contributor License Agreement.

[13/50] [abbrv] incubator-beam git commit: Add the slf4j-jdk bridge to the integration tests

[jamesmalone]

Add the slf4j-jdk bridge to the integration tests

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115350827


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/45f5951a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/45f5951a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/45f5951a

Branch: refs/heads/master
Commit: 45f5951a0a697b7044f15cd09418854641c9e98a
Parents: 51068d1
Author: bchambers 
Authored: Tue Feb 23 09:43:54 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/45f5951a/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 7d0cba9..9ff5cd2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -159,7 +159,7 @@
 
   org.codehaus.mojo
   exec-maven-plugin
-  1.1
+  1.4.0
   
 
   verify

[21/50] [abbrv] incubator-beam git commit: Add GroupByKey InProcess override

[jamesmalone]

Add GroupByKey InProcess override

This takes a GroupByKey primitive and implements it as a sequence of
composite transforms, including a simpler GroupByKeyOnly primitive.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115404181


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/96b02f47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/96b02f47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/96b02f47

Branch: refs/heads/master
Commit: 96b02f4737457d158db60ae467e7098839c383ab
Parents: 01a0da0
Author: tgroh 
Authored: Tue Feb 23 17:52:25 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../inprocess/GroupByKeyEvaluatorFactory.java   | 128 ---
 .../inprocess/InProcessPipelineRunner.java  |   9 ++
 .../dataflow/sdk/transforms/GroupByKey.java |   2 +-
 .../GroupByKeyEvaluatorFactoryTest.java | 114 ++---
 4 files changed, 190 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/96b02f47/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
index 44d6909..0347281 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/GroupByKeyEvaluatorFactory.java
@@ -19,18 +19,29 @@ import static 
com.google.cloud.dataflow.sdk.util.CoderUtils.encodeToByteArray;
 
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.CoderException;
+import com.google.cloud.dataflow.sdk.coders.IterableCoder;
 import com.google.cloud.dataflow.sdk.coders.KvCoder;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.CommittedBundle;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.InProcessEvaluationContext;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.InProcessPipelineRunner.UncommittedBundle;
 import 
com.google.cloud.dataflow.sdk.runners.inprocess.StepTransformResult.Builder;
 import com.google.cloud.dataflow.sdk.transforms.AppliedPTransform;
+import com.google.cloud.dataflow.sdk.transforms.DoFn;
 import com.google.cloud.dataflow.sdk.transforms.GroupByKey;
-import com.google.cloud.dataflow.sdk.transforms.GroupByKey.GroupByKeyOnly;
+import 
com.google.cloud.dataflow.sdk.transforms.GroupByKey.ReifyTimestampsAndWindows;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
+import com.google.cloud.dataflow.sdk.transforms.ParDo;
+import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
+import com.google.cloud.dataflow.sdk.util.GroupAlsoByWindowViaWindowSetDoFn;
+import com.google.cloud.dataflow.sdk.util.KeyedWorkItem;
+import com.google.cloud.dataflow.sdk.util.KeyedWorkItemCoder;
+import com.google.cloud.dataflow.sdk.util.KeyedWorkItems;
+import com.google.cloud.dataflow.sdk.util.SystemReduceFn;
 import com.google.cloud.dataflow.sdk.util.WindowedValue;
+import com.google.cloud.dataflow.sdk.util.WindowingStrategy;
 import com.google.cloud.dataflow.sdk.values.KV;
 import com.google.cloud.dataflow.sdk.values.PCollection;
+import com.google.common.annotations.VisibleForTesting;
 
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -53,52 +64,56 @@ class GroupByKeyEvaluatorFactory implements 
TransformEvaluatorFactory {
 (AppliedPTransform) application, (CommittedBundle) inputBundle, 
evaluationContext);
   }
 
-  private  TransformEvaluator> createEvaluator(
+  private  TransformEvaluator>> createEvaluator(
   final AppliedPTransform<
-  PCollection>, PCollection>>, 
GroupByKeyOnly>
+  PCollection>>, 
PCollection>,
+  InProcessGroupByKeyOnly>
   application,
   final CommittedBundle> inputBundle,
   final InProcessEvaluationContext evaluationContext) {
 return new GroupByKeyEvaluator(evaluationContext, inputBundle, 
application);
   }
 
-  private static class GroupByKeyEvaluator implements 
TransformEvaluator> {
+  private static class GroupByKeyEvaluator
+  implements TransformEvaluator>> {
 private final InProcessEvaluationContext evaluationContext;
 
 private final CommittedBundle> inputBundle;
 private final AppliedPTransform<
-PCollection>, PCollection>>, 
GroupByKeyOnly>
+PCollection>>, PCollection>,
+InProces

[27/50] [abbrv] incubator-beam git commit: Add used-but-undeclared findbugs JSR305 dependencies

[jamesmalone]

Add used-but-undeclared findbugs JSR305 dependencies

This artifact provides annotations such as @Nullable, @ThreadSafe,
and @NotThreadSafe, which we use throughout the SDK.

Relevant pieces of diff from `mvn dependency:tree -Dverbose`:

 [INFO] com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:...
+[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
 [INFO] +- com.google.cloud.bigtable:bigtable-protos:jar:0.2.3:compile
-[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.1:compile
+[INFO] |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted...)

And pieces that did not change but are simply other dependencies that also
depend on it so their entries were already omitted:

 [INFO] +- io.grpc:grpc-all:jar:0.12.0:compile
 [INFO] |  +- io.grpc:grpc-core:jar:0.12.0:compile
 [INFO] | +- (com.google.code.findbugs:jsr305:jar:3.0.0:compile - omit...)
 [INFO] +- com.google.cloud.bigdataoss:gcsio:jar:1.4.3:compile
 [INFO] |  +- (com.google.code.findbugs:jsr305:jar:2.0.3:compile - omitted...)
 [INFO] +- com.google.cloud.bigdataoss:util:jar:1.4.3:compile
 [INFO] |  +- (com.google.code.findbugs:jsr305:jar:2.0.3:compile - omitted...)
 [INFO] +- com.google.cloud.bigtable:bigtable-client-core:jar:0.2.3:compile
 [INFO] |  +- (com.google.code.findbugs:jsr305:jar:3.0.1:compile - omitted...)
 [INFO] +- com.google.oauth-client:google-oauth-client:jar:1.21.0:compile
 [INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted...)
 [INFO] +- com.google.http-client:google-http-client:jar:1.21.0:compile
 [INFO] |  +- (com.google.code.findbugs:jsr305:jar:1.3.9:compile - omitted...)

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115418405


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f7fc939c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f7fc939c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f7fc939c

Branch: refs/heads/master
Commit: f7fc939c1266caa0e7139c50904e66fca97b44c2
Parents: 01fd859
Author: klk 
Authored: Tue Feb 23 22:17:46 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 examples/pom.xml | 6 ++
 pom.xml  | 1 +
 sdk/pom.xml  | 6 ++
 worker/pom.xml   | 6 ++
 4 files changed, 19 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7fc939c/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index 3900a23..d7834cb 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -466,6 +466,12 @@
 
 
 
+  com.google.code.findbugs
+  jsr305
+  ${jsr305.version}
+
+
+
   joda-time
   joda-time
   ${joda.version}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7fc939c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index cf24c7b..d946605 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77,6 +77,7 @@
 1.3
 2.7.0
 2.4
+3.0.1
 4.11
 3.0.0-beta-1
 v1-rev7-1.21.0

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7fc939c/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index bc6d519..aeead72 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -613,6 +613,12 @@
 
 
 
+  com.google.code.findbugs
+  jsr305
+  ${jsr305.version}
+
+
+
   com.fasterxml.jackson.core
   jackson-core
   ${jackson.version}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f7fc939c/worker/pom.xml
--
diff --git a/worker/pom.xml b/worker/pom.xml
index e621ed4..ab9e172 100644
--- a/worker/pom.xml
+++ b/worker/pom.xml
@@ -263,6 +263,12 @@
   3.1.0
 
 
+
+  com.google.code.findbugs
+  jsr305
+  ${jsr305.version}
+
+
 
 
   org.hamcrest

[46/50] [abbrv] incubator-beam git commit: Revert "Add a first README.md file (at least to trigger the github mirroring)"

[jamesmalone]

Revert "Add a first README.md file (at least to trigger the github mirroring)"

This reverts commit 11e842717f70298a4ea8436363b3101117685f60 in preparation for 
the initial code drop.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/41e5cc9e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/41e5cc9e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/41e5cc9e

Branch: refs/heads/master
Commit: 41e5cc9e50be887e079ab746153745ee9e445664
Parents: 11e8427
Author: Frances Perry 
Authored: Fri Feb 26 12:00:44 2016 -0800
Committer: Frances Perry 
Committed: Fri Feb 26 12:00:44 2016 -0800

--
 README.md | 68 --
 1 file changed, 68 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41e5cc9e/README.md
--
diff --git a/README.md b/README.md
deleted file mode 100644
index f353381..000
--- a/README.md
+++ /dev/null
@@ -1,68 +0,0 @@
-# Apache Beam
-
-[Apache Beam](http://beam.incubator.apache.org) provides a simple, powerful
-programming model for building both batch and streaming parallel data 
processing
-pipelines. It also covers the data integration processus.
-
-[General 
usage](http://beam.incubator.apache.org/documentation/getting-started) is
-a good starting point for Apache Beam.
-
-You can take a look on the [Beam 
Examples](http://git-wip-us.apache.org/repos/asf/incubator-beam/examples).
-
-## Status [Build Status](http://builds.apache.org/job/beam-master)
-
-## Overview
-
-The key concepts in this programming model are:
-
-* `PCollection`: represents a collection of data, which could be bounded or 
unbounded in size.
-* `PTransform`: represents a computation that transforms input PCollections 
into output PCollections.
-* `Pipeline`: manages a directed acyclic graph of PTransforms and PCollections 
that is ready for execution.
-* `PipelineRunner`: specifies where and how the pipeline should execute.
-
-We provide the following PipelineRunners:
-
-  1. The `DirectPipelineRunner` runs the pipeline on your local machine.
-  2. The `BlockingDataflowPipelineRunner` submits the pipeline to the Dataflow 
Service via the `DataflowPipelineRunner`
-and then prints messages about the job status until the execution is complete.
-  3. The `SparkPipelineRunner` runs the pipeline on an Apache Spark cluster.
-  4. The `FlinkPipelineRunner` runs the pipeline on an Apache Flink cluster.
-
-## Getting Started
-
-The following command will build both the `sdk` and `example` modules and
-install them in your local Maven repository:
-
-mvn clean install
-
-You can speed up the build and install process by using the following options:
-
-  1. To skip execution of the unit tests, run:
-
-mvn install -DskipTests
-
-  2. While iterating on a specific module, use the following command to compile
-  and reinstall it. For example, to reinstall the `examples` module, run:
-
-mvn install -pl examples
-
-  Be careful, however, as this command will use the most recently installed SDK
-  from the local repository (or Maven Central) even if you have changed it
-  locally.
-
-After building and installing, you can execute the `WordCount` and other
-example pipelines by following the instructions in this 
[README](https://git-wip-us.apache.org/repos/asf/incubator-beam/examples/README.md).
-
-## Contact Us
-
-You can subscribe on the mailing lists to discuss and get involved in Apache 
Beam:
-
-* [Subscribe](mailto:user-subscr...@beam.incubator.apache.org) on the 
[u...@beam.incubator.apache.org](mailto:u...@beam.incubator.apache.org)
-* [Subscribe](mailto:dev-subscr...@beam.incubator.apache.org) on the 
[d...@beam.incubator.apache.org](mailto:d...@beam.incubator.apache.org)
-
-You can report issue on [Jira](https://issues.apache.org/jira/browse/BEAM).
-
-## More Information
-
-* [Apache Beam](http://beam.incubator.apache.org)
-* [Apache Beam Documentation](http://beam.incubator.apache.org/documentation)

[48/50] [abbrv] incubator-beam git commit: Update README for initial code drop.

[jamesmalone]

Update README for initial code drop.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3623a237
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3623a237
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3623a237

Branch: refs/heads/master
Commit: 3623a237fb1d40ace9d2a06690f89cb3ff3dbb20
Parents: 394390f
Author: Frances Perry 
Authored: Fri Feb 26 12:22:15 2016 -0800
Committer: Frances Perry 
Committed: Fri Feb 26 12:22:15 2016 -0800

--
 README.md | 138 ++---
 1 file changed, 43 insertions(+), 95 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3623a237/README.md
--
diff --git a/README.md b/README.md
index d5345a3..db4a13f 100644
--- a/README.md
+++ b/README.md
@@ -1,125 +1,73 @@
-# Google Cloud Dataflow SDK for Java
+# Apache Beam
 
-[Google Cloud Dataflow](https://cloud.google.com/dataflow/) provides a simple,
-powerful programming model for building both batch and streaming parallel data
-processing pipelines. This repository hosts the open-sourced Cloud Dataflow SDK
-for Java, which can be used to run pipelines against the Google Cloud Dataflow
-Service.
+[Apache Beam](http://beam.incubator.apache.org) is a unified model for 
defining both batch and streaming data-parallel processing pipelines, as well 
as a set of language-specific SDKs for constructing pipelines and Runners for 
executing them on distributed processing backends like [Apache 
Spark](http://spark.apache.org/), [Apache Flink](http://flink.apache.org), and 
[Google Cloud Dataflow](http://cloud.google.com/dataflow).
 
-[General usage](https://cloud.google.com/dataflow/getting-started) of Google
-Cloud Dataflow does **not** require use of this repository. Instead:
 
-1. depend directly on a specific
-[version](https://cloud.google.com/dataflow/release-notes/java) of the SDK in
-the [Maven Central 
Repository](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.google.cloud.dataflow%22)
-by adding the following dependency to development
-environments like Eclipse or Apache Maven:
+## Status 
 
-
-  com.google.cloud.dataflow
-  google-cloud-dataflow-java-sdk-all
-  version_number
-
+_**The Apache Beam project is in the process of bootstrapping. This includes 
the creation of project resources, the refactoring of the initial code 
submissions, and the formulation of project documentation, planning, and design 
documents. Please expect a significant amount of churn and breaking changes in 
the near future.**_
 
-1. download the example pipelines from the separate
-[DataflowJavaSDK-examples](https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples)
-repository.
+[Build Status](http://builds.apache.org/job/beam-master)
 
-However, if you'd like to contribute to the SDK, write your own PipelineRunner,
-or just dig in for the fun of it, please stay with us here!
 
-## Status [![Build 
Status](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK.svg?branch=master)](https://travis-ci.org/GoogleCloudPlatform/DataflowJavaSDK)
+## Overview
 
-Both the SDK and the Dataflow Service are generally available, open to all
-developers, and considered stable and fully qualified for production use.
+Beam provides a general approach to expressing [embarrassingly 
parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel) data 
processing pipelines and supports three categories of users, each of which have 
relatively disparate backgrounds and needs.
 
-## Overview
+1. _End Users_: Writing pipelines with an existing SDK, running it on an 
existing runner. These users want to focus on writing their application logic 
and have everything else just work.
+2. _SDK Writers_: Developing a Beam SDK targeted at a specific user community 
(Java, Python, Scala, Go, R, graphical, etc). These users are language geeks, 
and  would prefer to be shielded from all the details of various runners and 
their implementations.
+3. _Runner Writers_: Have an execution environment for distributed processing 
and would like to support programs written against the Beam Model. Would prefer 
to be shielded from details of multiple SDKs.
 
-The key concepts in this programming model are:
-
-* 
[`PCollection`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/PCollection.java):
-represents a collection of data, which could be bounded or unbounded in size.
-* 
[`PTransform`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/transforms/PTransform.java):
-represents a computation that transforms input PCo

[05/50] [abbrv] incubator-beam git commit: Expose base output file name on FileBasedSink

[jamesmalone]

Expose base output file name on FileBasedSink

Release Notes
Add the ability to get the base output filename to FileBasedSinks.

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115265994


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c857afaf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c857afaf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c857afaf

Branch: refs/heads/master
Commit: c857afaf1911175a6d907f53167e39e9b639c665
Parents: 209364e
Author: lcwik 
Authored: Mon Feb 22 13:49:38 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../com/google/cloud/dataflow/sdk/io/FileBasedSink.java  | 11 +++
 1 file changed, 11 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c857afaf/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
index f14f4bf..7c30167 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
@@ -121,6 +121,13 @@ public abstract class FileBasedSink extends Sink {
   }
 
   /**
+   * Returns the base output filename for this file based sink.
+   */
+  public String getBaseOutputFilename() {
+return baseOutputFilename;
+  }
+
+  /**
* Perform pipeline-construction-time validation. The default implementation 
is a no-op.
* Subclasses should override to ensure the sink is valid and can be written 
to. It is recommended
* to use {@link Preconditions} in the implementation of this method.
@@ -806,6 +813,7 @@ public abstract class FileBasedSink extends Sink {
   }
 
   static class ReshardForWrite extends PTransform, 
PCollection> {
+@Override
 public PCollection apply(PCollection input) {
   return input
   // TODO: This would need to be adapted to write per-window shards.
@@ -815,10 +823,12 @@ public abstract class FileBasedSink extends Sink {
   .apply("RandomKey", ParDo.of(
   new DoFn>() {
 transient long counter, step;
+@Override
 public void startBundle(Context c) {
   counter = (long) (Math.random() * Long.MAX_VALUE);
   step = 1 + 2 * (long) (Math.random() * Long.MAX_VALUE);
 }
+@Override
 public void processElement(ProcessContext c) {
   counter += step;
   c.output(KV.of(counter, c.element()));
@@ -827,6 +837,7 @@ public abstract class FileBasedSink extends Sink {
   .apply(GroupByKey.create())
   .apply("Ungroup", ParDo.of(
   new DoFn>, T>() {
+@Override
 public void processElement(ProcessContext c) {
   for (T item : c.element().getValue()) {
 c.output(item);

[31/50] [abbrv] incubator-beam git commit: Honor user requested shard limits for AvroIO.Write on DirectPipelineRunner

[jamesmalone]

Honor user requested shard limits for AvroIO.Write on DirectPipelineRunner

During the migration to custom sink within AvroIO, shard controls
were removed for DirectPipelineRunner. This change adds them
back.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115515647


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/510a55db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/510a55db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/510a55db

Branch: refs/heads/master
Commit: 510a55dbbf9b6d1a94817f7e8e78e8211dd559a4
Parents: 8b5257f
Author: lcwik 
Authored: Wed Feb 24 18:01:53 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../sdk/runners/DirectPipelineRunner.java   | 59 
 .../sdk/runners/DirectPipelineRunnerTest.java   | 53 ++
 2 files changed, 112 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/510a55db/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
index 4543b5a..872cfef 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/DirectPipelineRunner.java
@@ -25,6 +25,7 @@ import com.google.cloud.dataflow.sdk.PipelineResult;
 import com.google.cloud.dataflow.sdk.coders.CannotProvideCoderException;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.ListCoder;
+import com.google.cloud.dataflow.sdk.io.AvroIO;
 import com.google.cloud.dataflow.sdk.io.FileBasedSink;
 import com.google.cloud.dataflow.sdk.io.TextIO;
 import com.google.cloud.dataflow.sdk.options.DirectPipelineOptions;
@@ -240,6 +241,8 @@ public class DirectPipelineRunner
   return (OutputT) applyTestCombine((Combine.GroupedValues) transform, 
(PCollection) input);
 } else if (transform instanceof TextIO.Write.Bound) {
   return (OutputT) applyTextIOWrite((TextIO.Write.Bound) transform, 
(PCollection) input);
+} else if (transform instanceof AvroIO.Write.Bound) {
+  return (OutputT) applyAvroIOWrite((AvroIO.Write.Bound) transform, 
(PCollection) input);
 } else {
   return super.apply(transform, input);
 }
@@ -343,6 +346,62 @@ public class DirectPipelineRunner
   }
 
   /**
+   * Applies AvroIO.Write honoring user requested sharding controls (i.e. 
withNumShards)
+   * by applying a partition function based upon the number of shards the user 
requested.
+   */
+  private static class DirectAvroIOWrite extends PTransform, 
PDone> {
+private final AvroIO.Write.Bound transform;
+
+private DirectAvroIOWrite(AvroIO.Write.Bound transform) {
+  this.transform = transform;
+}
+
+@Override
+public PDone apply(PCollection input) {
+  checkState(transform.getNumShards() > 1,
+  "DirectAvroIOWrite is expected to only be used when sharding 
controls are required.");
+
+  // Evenly distribute all the elements across the partitions.
+  PCollectionList partitionedElements =
+  input.apply(Partition.of(transform.getNumShards(),
+   new 
ElementProcessingOrderPartitionFn()));
+
+  // For each input PCollection partition, create a write transform that 
represents
+  // one of the specific shards.
+  for (int i = 0; i < transform.getNumShards(); ++i) {
+/*
+ * This logic mirrors the file naming strategy within
+ * {@link FileBasedSink#generateDestinationFilenames()}
+ */
+String outputFilename = IOChannelUtils.constructName(
+transform.getFilenamePrefix(),
+transform.getShardNameTemplate(),
+getFileExtension(transform.getFilenameSuffix()),
+i,
+transform.getNumShards());
+
+String transformName = String.format("%s(Shard:%s)", 
transform.getName(), i);
+partitionedElements.get(i).apply(transformName,
+
transform.withNumShards(1).withShardNameTemplate("").withSuffix("").to(outputFilename));
+  }
+  return PDone.in(input.getPipeline());
+}
+  }
+
+  /**
+   * Apply the override for AvroIO.Write.Bound if the user requested sharding 
controls
+   * greater than one.
+   */
+  private  PDone applyAvroIOWrite(AvroIO.Write.Bound transform, 
PCollection input) {
+if (transform.getNumShards() <= 1) {
+  // By default, the DirectPipelineRunner outputs to only 1 shard. Sin

[42/50] [abbrv] incubator-beam git commit: Ensure a TypedPValue has a Coder on finishSpecifying

[jamesmalone]

Ensure a TypedPValue has a Coder on finishSpecifying

Coders cannot be set on a PValue that is is marked as finished
specifying, and a coder is required for every TypedPValue in a pipeline.

Ensure that a coder is always available when a TypedPValue has been
finished by invoking getCoder() (which will throw an exception if no
coder is available)

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115601351


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fba91473
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fba91473
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fba91473

Branch: refs/heads/master
Commit: fba914736cc3f3401aa96c252a1336e9e5865b1e
Parents: 06c8911
Author: tgroh 
Authored: Thu Feb 25 13:59:52 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../cloud/dataflow/sdk/values/TypedPValue.java  |  3 ++
 .../dataflow/sdk/values/TypedPValueTest.java| 51 +++-
 2 files changed, 31 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fba91473/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/TypedPValue.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/TypedPValue.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/TypedPValue.java
index 9b210b2..29fd639 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/TypedPValue.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/values/TypedPValue.java
@@ -83,6 +83,9 @@ public abstract class TypedPValue extends PValueBase 
implements PValue {
   return;
 }
 super.finishSpecifying();
+// Ensure that this TypedPValue has a coder by inferring the coder if none 
exists; If not,
+// this will throw an exception.
+getCoder();
   }
 
   /

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fba91473/sdk/src/test/java/com/google/cloud/dataflow/sdk/values/TypedPValueTest.java
--
diff --git 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/values/TypedPValueTest.java 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/values/TypedPValueTest.java
index 4c62111..b0a13ec 100644
--- 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/values/TypedPValueTest.java
+++ 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/values/TypedPValueTest.java
@@ -20,7 +20,6 @@ import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.hamcrest.Matchers.not;
 import static org.junit.Assert.assertThat;
-import static org.junit.Assert.fail;
 
 import com.google.cloud.dataflow.sdk.Pipeline;
 import com.google.cloud.dataflow.sdk.coders.VarIntCoder;
@@ -132,28 +131,34 @@ public class TypedPValueTest {
   @Test
   public void testParDoWithNoSideOutputsErrorDoesNotMentionTupleTag() {
 Pipeline p = TestPipeline.create();
-PCollection input = p
-.apply(Create.of(1, 2, 3))
-.apply(ParDo.of(new EmptyClassDoFn()));
-
-try {
-  input.getCoder();
-} catch (IllegalStateException exc) {
-  String message = exc.getMessage();
-
-  // Output specific to ParDo TupleTag side outputs should not be present.
-  assertThat(message, not(containsString("erasure")));
-  assertThat(message, not(containsString("see TupleTag Javadoc")));
-  // Instead, expect output suggesting other possible fixes.
-  assertThat(message,
-  containsString("Building a Coder using a registered CoderFactory 
failed"));
-  assertThat(message,
-  containsString("Building a Coder from the @DefaultCoder annotation 
failed"));
-  assertThat(message,
-  containsString("Building a Coder from the fallback CoderProvider 
failed"));
-  return;
-}
-fail("Should have thrown IllegalStateException due to failure to infer a 
coder.");
+PCollection input =
+p.apply(Create.of(1, 2, 3)).apply(ParDo.of(new EmptyClassDoFn()));
+
+thrown.expect(IllegalStateException.class);
+
+// Output specific to ParDo TupleTag side outputs should not be present.
+thrown.expectMessage(not(containsString("erasure")));
+thrown.expectMessage(not(containsString("see TupleTag Javadoc")));
+// Instead, expect output suggesting other possible fixes.
+thrown.expectMessage(containsString("Building a Coder using a registered 
CoderFactory failed"));
+thrown.expectMessage(
+containsString("Building a Coder from the @DefaultCoder annotation 
failed"));
+thrown.expectMessage

[22/50] [abbrv] incubator-beam git commit: Resubmit "Migrate AvroIO.Write to a custom sink"

[jamesmalone]

Resubmit "Migrate AvroIO.Write to a custom sink"

Note for user requested sharding limits to be supported,
each pipeline runner must support applying those sharding limits.

Google Cloud Dataflow supports sharding limits.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115402880


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/01a0da02
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/01a0da02
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/01a0da02

Branch: refs/heads/master
Commit: 01a0da02daed5f1609237ae85c82fd056ea76339
Parents: 2e89a4b
Author: dhalperi 
Authored: Tue Feb 23 17:35:40 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../google/cloud/dataflow/sdk/io/AvroIO.java| 223 ++-
 .../sdk/runners/DataflowPipelineRunner.java | 189 
 .../sdk/runners/DataflowPipelineTranslator.java |   5 -
 .../sdk/runners/dataflow/AvroIOTranslator.java  |  87 
 .../sdk/io/AvroIOGeneratedClassTest.java| 186 
 .../cloud/dataflow/sdk/io/AvroIOTest.java   |  34 ++-
 6 files changed, 433 insertions(+), 291 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/01a0da02/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
index 9ee7e6b..f016b5b 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/AvroIO.java
@@ -22,24 +22,25 @@ import com.google.cloud.dataflow.sdk.coders.AvroCoder;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.VoidCoder;
 import com.google.cloud.dataflow.sdk.io.Read.Bounded;
-import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
+import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.runners.PipelineRunner;
-import com.google.cloud.dataflow.sdk.runners.worker.AvroSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
+import com.google.cloud.dataflow.sdk.util.MimeTypes;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 import com.google.cloud.dataflow.sdk.values.PInput;
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 
 import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.reflect.ReflectData;
 
 import java.io.IOException;
-import java.util.List;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
 import java.util.regex.Pattern;
 
 import javax.annotation.Nullable;
@@ -317,7 +318,7 @@ public class AvroIO {
 : com.google.cloud.dataflow.sdk.io.Read.from(
 AvroSource.from(filepattern).withSchema(type));
 
-PCollection pcol = input.getPipeline().apply(read);
+PCollection pcol = input.getPipeline().apply("Read", read);
 // Honor the default output coder that would have been used by this 
PTransform.
 pcol.setCoder(getDefaultOutputCoder());
 return pcol;
@@ -473,8 +474,6 @@ public class AvroIO {
   final int numShards;
   /** Shard template string. */
   final String shardTemplate;
-  /** Insert a shuffle before writing to decouple parallelism when 
numShards != 0. */
-  final boolean forceReshard;
   /** The class type of the records. */
   final Class type;
   /** The schema of the output file. */
@@ -484,18 +483,23 @@ public class AvroIO {
   final boolean validate;
 
   Bound(Class type) {
-this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, true, type, 
null, true);
+this(null, null, "", 0, ShardNameTemplate.INDEX_OF_MAX, type, null, 
true);
   }
 
-  Bound(String name, String filenamePrefix, String filenameSuffix, int 
numShards,
-  String shardTemplate, boolean forceReshard, Class type, Schema 
schema,
+  Bound(
+  String name,
+  String filenamePrefix,
+  String filenameSuffix,
+  int numShards,
+  String shardTemplate,
+  Class type,
+  Schema schema,
   boolean validate) {
 super(name);
 this.filename

[17/50] [abbrv] incubator-beam git commit: Reverts "Move Google Cloud Dataflow native sinks to worker module"

[jamesmalone]

Reverts "Move Google Cloud Dataflow native sinks to worker module"

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115349469


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/51068d16
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/51068d16
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/51068d16

Branch: refs/heads/master
Commit: 51068d1635f4bb3143412a7708852e21816a1d27
Parents: 9f546ef
Author: sgmc 
Authored: Tue Feb 23 09:29:49 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../dataflow/sdk/runners/worker/AvroSink.java   | 135 +
 .../dataflow/sdk/runners/worker/TextSink.java   | 291 +++
 .../dataflow/sdk/util/common/worker/Sink.java   |  64 
 3 files changed, 490 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/51068d16/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
new file mode 100644
index 000..b101a2b
--- /dev/null
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/worker/AvroSink.java
@@ -0,0 +1,135 @@
+/*
+ * Copyright (C) 2015 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+
+package com.google.cloud.dataflow.sdk.runners.worker;
+
+import com.google.cloud.dataflow.sdk.coders.AvroCoder;
+import com.google.cloud.dataflow.sdk.util.CoderUtils;
+import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
+import com.google.cloud.dataflow.sdk.util.MimeTypes;
+import com.google.cloud.dataflow.sdk.util.ShardingWritableByteChannel;
+import com.google.cloud.dataflow.sdk.util.WindowedValue;
+import 
com.google.cloud.dataflow.sdk.util.WindowedValue.ValueOnlyWindowedValueCoder;
+import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
+
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileWriter;
+import org.apache.avro.io.DatumWriter;
+
+import java.io.IOException;
+import java.nio.channels.Channels;
+import java.nio.channels.WritableByteChannel;
+import java.util.ArrayList;
+import java.util.Random;
+
+/**
+ * A sink that writes Avro files.
+ *
+ * @param  the type of the elements written to the sink
+ */
+public class AvroSink extends Sink> {
+
+  final String filenamePrefix;
+  final String shardFormat;
+  final String filenameSuffix;
+  final int shardCount;
+  final AvroCoder avroCoder;
+  final Schema schema;
+
+  public AvroSink(String filename, ValueOnlyWindowedValueCoder coder) {
+this(filename, "", "", 1, coder);
+  }
+
+  public AvroSink(String filenamePrefix, String shardFormat, String 
filenameSuffix, int shardCount,
+  ValueOnlyWindowedValueCoder coder) {
+if (!(coder.getValueCoder() instanceof AvroCoder)) {
+  throw new IllegalArgumentException(String.format(
+  "AvroSink requires an AvroCoder, not a %s", 
coder.getValueCoder().getClass()));
+}
+
+this.filenamePrefix = filenamePrefix;
+this.shardFormat = shardFormat;
+this.filenameSuffix = filenameSuffix;
+this.shardCount = shardCount;
+this.avroCoder = (AvroCoder) coder.getValueCoder();
+this.schema = this.avroCoder.getSchema();
+  }
+
+  public SinkWriter> writer(DatumWriter datumWriter) 
throws IOException {
+WritableByteChannel writer = IOChannelUtils.create(
+filenamePrefix, shardFormat, filenameSuffix, shardCount, 
MimeTypes.BINARY);
+
+if (writer instanceof ShardingWritableByteChannel) {
+  return new AvroShardingFileWriter(datumWriter, 
(ShardingWritableByteChannel) writer);
+} else {
+  return new AvroFileWriter(datumWriter, writer);
+}
+  }
+
+  @Override
+  public SinkWriter> writer() throws IOException {
+return writer(avroCoder.createDatumWriter());
+  }
+
+  /** The SinkWriter for an AvroSink. */
+  class AvroFileWriter implements SinkWriter> {
+DataFileWriter fileWriter;
+
+public AvroFileWriter(DatumWriter datumWriter, WritableByteChannel 
outputChannel)
+throws IOException {
+  

[35/50] [abbrv] incubator-beam git commit: Rollback revert "Migrate TextIO.Write to a custom sink"

[jamesmalone]

Rollback revert "Migrate TextIO.Write to a custom sink"

Note for user requested sharding limits to be supported,
each pipeline runner must support applying those sharding limits.

DirectPipelineRunner and Google Cloud Dataflow supports sharding limits.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115500204


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/639e9d95
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/639e9d95
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/639e9d95

Branch: refs/heads/master
Commit: 639e9d95b61704ae1740a0a1f02f76c3d480fa48
Parents: 045e343
Author: lcwik 
Authored: Wed Feb 24 15:24:22 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:28 2016 -0800

--
 .../cloud/dataflow/sdk/io/FileBasedSink.java|  17 +-
 .../google/cloud/dataflow/sdk/io/TextIO.java| 187 +--
 .../sdk/runners/DataflowPipelineRunner.java | 131 +++--
 .../sdk/runners/DataflowPipelineTranslator.java |   6 -
 .../sdk/runners/DirectPipelineRunner.java   |  89 +
 .../sdk/runners/dataflow/TextIOTranslator.java  |  91 -
 .../dataflow/sdk/io/FileBasedSinkTest.java  |  29 ++-
 .../cloud/dataflow/sdk/io/TextIOTest.java   |  22 ---
 .../sdk/runners/DataflowPipelineRunnerTest.java |  21 +--
 .../runners/DataflowPipelineTranslatorTest.java |   4 +-
 .../sdk/runners/DirectPipelineRunnerTest.java   |  80 +++-
 .../dataflow/sdk/runners/TransformTreeTest.java |   9 +-
 12 files changed, 414 insertions(+), 272 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/639e9d95/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
index 7c30167..dda500c 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/FileBasedSink.java
@@ -355,7 +355,7 @@ public abstract class FileBasedSink extends Sink {
   String baseOutputFilename = getSink().baseOutputFilename;
   String fileNamingTemplate = getSink().fileNamingTemplate;
 
-  String suffix = (extension.length() == 0) ? extension : ("." + 
extension);
+  String suffix = getFileExtension(extension);
   for (int i = 0; i < numFiles; i++) {
 destFilenames.add(IOChannelUtils.constructName(
 baseOutputFilename, fileNamingTemplate, suffix, i, numFiles));
@@ -364,6 +364,21 @@ public abstract class FileBasedSink extends Sink {
 }
 
 /**
+ * Returns the file extension to be used. If the user did not request a 
file
+ * extension then this method returns the empty string. Otherwise this 
method
+ * adds a {@code "."} to the beginning of the users extension if one is 
not present.
+ */
+private String getFileExtension(String usersExtension) {
+  if (usersExtension == null || usersExtension.isEmpty()) {
+return "";
+  }
+  if (usersExtension.startsWith(".")) {
+return usersExtension;
+  }
+  return "." + usersExtension;
+}
+
+/**
  * Removes temporary output files. Uses the temporary filename to find 
files to remove.
  *
  * Can be called from subclasses that override {@link 
FileBasedWriteOperation#finalize}.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/639e9d95/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
index 0bb2861..d342f25 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/TextIO.java
@@ -26,11 +26,9 @@ import com.google.cloud.dataflow.sdk.io.Read.Bounded;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
 import com.google.cloud.dataflow.sdk.runners.DataflowPipelineRunner;
 import com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner;
-import com.google.cloud.dataflow.sdk.runners.worker.TextSink;
 import com.google.cloud.dataflow.sdk.transforms.PTransform;
 import com.google.cloud.dataflow.sdk.util.IOChannelUtils;
-import com.google.cloud.dataflow.sdk.util.WindowedValue;
-import com.google.cloud.dataflow.sdk.util.common.worker.Sink;
+import com.google.cloud.dataflow.sdk.util.MimeTypes;
 import com.google.cloud.dataflow.sdk.values.PCollection;
 import com.google.cloud.dataflow.sdk.values.PDone;
 im

[11/50] [abbrv] incubator-beam git commit: Move some worker-and-example-only dependencies out of sdk

[jamesmalone]

Move some worker-and-example-only dependencies out of sdk

Following the separation of the worker module, running

$ mvn dependency:analyze -pl sdk

yields

[WARNING] Used undeclared dependencies found:
[WARNING]com.google.code.findbugs:jsr305:jar:3.0.1:compile
[WARNING]javax.xml.stream:stax-api:jar:1.0-2:compile
[WARNING]com.google.http-client:google-http-client:jar:1.21.0:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.eclipse.jetty:jetty-servlet:jar:9.2.10.v20150310:compile
[WARNING]org.codehaus.woodstox:woodstox-core-asl:jar:4.1.2:compile
[WARNING]org.eclipse.jetty:jetty-server:jar:9.2.10.v20150310:compile
[WARNING]javax.servlet:javax.servlet-api:jar:3.1.0:compile
[WARNING]org.tukaani:xz:jar:1.5:compile

This change removes unused dependencies. The `xz` and `woodstox` dependencies
are for optional features of AvroSource and XmlSource.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115300250


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/24288e7b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/24288e7b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/24288e7b

Branch: refs/heads/master
Commit: 24288e7ba5be86d927cffaccbd4c768797f4361d
Parents: 7871cbb
Author: klk 
Authored: Mon Feb 22 20:25:49 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 examples/pom.xml  |  6 ++
 .../main/resources/archetype-resources/pom.xml|  6 ++
 sdk/pom.xml   | 18 --
 worker/pom.xml| 18 ++
 4 files changed, 30 insertions(+), 18 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/24288e7b/examples/pom.xml
--
diff --git a/examples/pom.xml b/examples/pom.xml
index adea4b1..3900a23 100644
--- a/examples/pom.xml
+++ b/examples/pom.xml
@@ -484,6 +484,12 @@
   runtime
 
 
+
+  javax.servlet
+  javax.servlet-api
+  3.1.0
+
+
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/24288e7b/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
--
diff --git 
a/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml 
b/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 8531087..bffa376 100644
--- a/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ b/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -166,6 +166,12 @@
   18.0
 
 
+ 
+  javax.servlet
+  javax.servlet-api
+  3.1.0
+
+
 
 
   org.slf4j

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/24288e7b/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index 1d74580..e3744fd 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -640,24 +640,6 @@
   ${joda.version}
 
 
-
-  org.eclipse.jetty
-  jetty-server
-  9.2.10.v20150310
-
-
-
-  org.eclipse.jetty
-  jetty-servlet
-  9.2.10.v20150310
-
-
-
-  javax.servlet
-  javax.servlet-api
-  3.1.0
-
-
 
 
   org.hamcrest

[20/50] [abbrv] incubator-beam git commit: Update maven-dependency-plugin to latest

[jamesmalone]

Update maven-dependency-plugin to latest

Prior to version 2.10 configuration options we require are
unavailable. Specifically, to have a strict dependency policy with
our optional dependency configurations, we will need:

 - ignoredUnusedDeclaredDependencies
 - ignoredUsedUndeclaredDependencies

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115411144


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/db708bb9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/db708bb9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/db708bb9

Branch: refs/heads/master
Commit: db708bb9a157d0b86519f9309d5c0b945a72ebde
Parents: 87b28e7
Author: klk 
Authored: Tue Feb 23 19:53:21 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 pom.xml | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/db708bb9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 9ff5cd2..cf24c7b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -293,6 +293,7 @@
 
   org.apache.maven.plugins
   maven-dependency-plugin
+  2.10
   
 true
   

[01/50] [abbrv] incubator-beam git commit: Allow TransformEvaluatorFactory#forApplication to throw

[jamesmalone]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 11e842717 -> 211e76abf


Allow TransformEvaluatorFactory#forApplication to throw

Certain Transforms (like sources) can throw an exception while being
constructed. This allows the transforms to throw an exception while
being constructed rather than on first application.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115122695


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/015e1afe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/015e1afe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/015e1afe

Branch: refs/heads/master
Commit: 015e1afede0abb3fc087d4c5d4072f7801ed5d45
Parents: 234d5eb
Author: tgroh 
Authored: Fri Feb 19 20:08:10 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 .../sdk/runners/inprocess/TransformEvaluatorFactory.java | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/015e1afe/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformEvaluatorFactory.java
index 7941c16..3b672e0 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/TransformEvaluatorFactory.java
@@ -34,8 +34,10 @@ public interface TransformEvaluatorFactory {
* Any work that must be done before input elements are processed (such as 
calling
* {@link DoFn#startBundle(DoFn.Context)}) must be done before the {@link 
TransformEvaluator} is
* made available to the caller.
+   *
+   * @throws Exception whenever constructing the underlying evaluator throws 
an exception
*/
TransformEvaluator forApplication(
   AppliedPTransform application, @Nullable CommittedBundle 
inputBundle,
-  InProcessEvaluationContext evaluationContext);
+  InProcessEvaluationContext evaluationContext) throws Exception;
 }

[04/50] [abbrv] incubator-beam git commit: Add new test proto messages that use map fields

[jamesmalone]

Add new test proto messages that use map fields

Directly, and transitively.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115280894


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2e171b50
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2e171b50
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2e171b50

Branch: refs/heads/master
Commit: 2e171b509ebe163ac54332eec626dccdfa18c9ad
Parents: 7ff52a0
Author: dhalperi 
Authored: Mon Feb 22 15:59:23 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:25 2016 -0800

--
 sdk/src/main/proto/proto2_coder_test_messages.proto | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2e171b50/sdk/src/main/proto/proto2_coder_test_messages.proto
--
diff --git a/sdk/src/main/proto/proto2_coder_test_messages.proto 
b/sdk/src/main/proto/proto2_coder_test_messages.proto
index 56efd89..eb3c3df 100644
--- a/sdk/src/main/proto/proto2_coder_test_messages.proto
+++ b/sdk/src/main/proto/proto2_coder_test_messages.proto
@@ -41,3 +41,11 @@ extend MessageC {
   optional MessageA field1 = 101;
   optional MessageB field2 = 102;
 }
+
+message MessageWithMap {
+  map field1 = 1;
+}
+
+message ReferencesMessageWithMap {
+  repeated MessageWithMap field1 = 1;
+}

[44/50] [abbrv] incubator-beam git commit: Fix SDK deps and enable strict enforcement

[jamesmalone]

Fix SDK deps and enable strict enforcement

After this change, the analysis by `mvn dependency:analyze` for the SDK
will fail if there are any transitive dependency violations.

To enable this, we take the following scope interpretations and adjust
the pom.xml accordingly:

A "compile" scope optional dependency is required for build but left out
of transitive dependencies. We have one of these:

 - org.codehaus.woodstox:stax2-api is an API that the SDK references
   directly, but it will not throw a class loading error until XmlSource
   is used.

A "runtime" scope optional dependency is never reference directly from the
SDK, but is an implementation referenced indirectly and loaded lazily.
We have two of these:

 - org.tukaani:xz is an optional dependency to support xz-compressed
   streams at runtime we use from Apache commons, though the SDK does not
   directly use the library. Our use of Apache commons is an implementation
   detail so xz is really our dependency.
 - org.codehaus.woodstox:woodstox-core-asl is an implementation of stax2-api.
   Such an implementation must be present on the classpath when XmlSource is
   used, but the SDK deliberately does not reference a specific
   implementation.

Release Notes

[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115604996


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/89e62414
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/89e62414
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/89e62414

Branch: refs/heads/master
Commit: 89e624141c589be950d792508705966b56f2bc61
Parents: 3111646
Author: klk 
Authored: Thu Feb 25 14:30:23 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:29 2016 -0800

--
 sdk/pom.xml | 25 +---
 .../google/cloud/dataflow/sdk/io/XmlSource.java | 18 --
 2 files changed, 33 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89e62414/sdk/pom.xml
--
diff --git a/sdk/pom.xml b/sdk/pom.xml
index 1f15b02..4995da0 100644
--- a/sdk/pom.xml
+++ b/sdk/pom.xml
@@ -141,6 +141,19 @@
 maven-compiler-plugin
   
 
+  
+org.apache.maven.plugins
+maven-dependency-plugin
+
+  
+analyze-only
+
+  true
+
+  
+
+  
+
   
   
@@ -661,8 +674,11 @@
 
 
 
 
   org.codehaus.woodstox
@@ -675,6 +691,7 @@
   org.codehaus.woodstox
   woodstox-core-asl
   ${woodstox.version}
+  runtime
   true
   
 
@@ -687,12 +704,14 @@
 
 
 
   org.tukaani
   xz
   1.5
+  runtime
   true
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/89e62414/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
--
diff --git a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
index d684d22..1ead391 100644
--- a/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
+++ b/sdk/src/main/java/com/google/cloud/dataflow/sdk/io/XmlSource.java
@@ -17,6 +17,7 @@ package com.google.cloud.dataflow.sdk.io;
 import com.google.cloud.dataflow.sdk.coders.Coder;
 import com.google.cloud.dataflow.sdk.coders.JAXBCoder;
 import com.google.cloud.dataflow.sdk.options.PipelineOptions;
+import com.google.cloud.dataflow.sdk.runners.PipelineRunner;
 import com.google.common.base.Preconditions;
 
 import org.codehaus.stax2.XMLInputFactory2;
@@ -94,18 +95,21 @@ import javax.xml.stream.XMLStreamReader;
  * Currently, only XML files that use single-byte characters are supported. 
Using a file that
  * contains multi-byte characters may result in data loss or duplication.
  *
- * To use {@code XmlSource}, explicitly declare dependencies on following 
two jars from Woodstox
- * StAX XML parser.
- * (1) stax2-api-3.1.1.jar
- * (2) woodstox-core-asl-4.1.2.jar
- * These dependencies have been declared as optional in Maven sdk/pom.xml file 
of Google Cloud
- * Dataflow.
+ * To use {@link XmlSource}:
+ * 
+ *   Explicitly declare a dependency on 
org.codehaus.woodstox:stax2-api
+ *   Include a compatible implementation on the classpath at run-time,
+ *   such as org.codehaus.woodstox:woodstox-core-asl
+ * 
+ *
+ * These dependencies have been declared as optional in Maven sdk/pom.xml 
file of
+ * Google Cloud Dataflow.
  *
  * Permissions
  * Permission requirements depend on the
  * {@link com.google.cloud.dataflow.sdk.runners.PipelineRunner PipelineRunner} 

[28/50] [abbrv] incubator-beam git commit: Handle PCollectionList.empty() in FlattenEvaluatorFactory

[jamesmalone]

Handle PCollectionList.empty() in FlattenEvaluatorFactory

PCollectionList.empty() is a valid argument to a Flatten#pCollections
PTransform. It should succeed and produce no output.

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115455733


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/d15d924d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/d15d924d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/d15d924d

Branch: refs/heads/master
Commit: d15d924d5dae18a07067cc3a71ba3b50431fe3d7
Parents: f7fc939
Author: tgroh 
Authored: Wed Feb 24 08:42:55 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:27 2016 -0800

--
 .../inprocess/FlattenEvaluatorFactory.java  |  8 ++-
 .../inprocess/FlattenEvaluatorFactoryTest.java  | 23 
 2 files changed, 30 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d15d924d/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
index d8b5312..1442888 100644
--- 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactory.java
@@ -1,5 +1,5 @@
 /*
- * Copyright (C) 2015 Google Inc.
+ * Copyright (C) 2016 Google Inc.
  *
  * Licensed under the Apache License, Version 2.0 (the "License"); you may not
  * use this file except in compliance with the License. You may obtain a copy 
of
@@ -46,6 +46,12 @@ class FlattenEvaluatorFactory implements 
TransformEvaluatorFactory {
   application,
   final CommittedBundle inputBundle,
   final InProcessEvaluationContext evaluationContext) {
+if (inputBundle == null) {
+  // it is impossible to call processElement on a flatten with no input 
bundle. A Flatten with
+  // no input bundle occurs as an output of 
Flatten.pcollections(PCollectionList.empty())
+  return new FlattenEvaluator<>(
+  null, StepTransformResult.withoutHold(application).build());
+}
 final UncommittedBundle outputBundle =
 evaluationContext.createBundle(inputBundle, application.getOutput());
 final InProcessTransformResult result =

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/d15d924d/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactoryTest.java
--
diff --git 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactoryTest.java
 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactoryTest.java
index c2b9995..dac42b6 100644
--- 
a/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactoryTest.java
+++ 
b/sdk/src/test/java/com/google/cloud/dataflow/sdk/runners/inprocess/FlattenEvaluatorFactoryTest.java
@@ -16,6 +16,7 @@
 package com.google.cloud.dataflow.sdk.runners.inprocess;
 
 import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.hamcrest.Matchers.emptyIterable;
 import static org.junit.Assert.assertThat;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
@@ -112,4 +113,26 @@ public class FlattenEvaluatorFactoryTest {
 WindowedValue.timestampedValueInGlobalWindow(-4, new 
Instant(-4096)),
 WindowedValue.valueInGlobalWindow(-1)));
   }
+
+  @Test
+  public void testFlattenInMemoryEvaluatorWithEmptyPCollectionList() throws 
Exception {
+TestPipeline p = TestPipeline.create();
+PCollectionList list = PCollectionList.empty(p);
+
+PCollection flattened = 
list.apply(Flatten.pCollections());
+
+InProcessEvaluationContext context = 
mock(InProcessEvaluationContext.class);
+
+FlattenEvaluatorFactory factory = new FlattenEvaluatorFactory();
+TransformEvaluator emptyEvaluator =
+factory.forApplication(flattened.getProducingTransformInternal(), 
null, context);
+
+InProcessTransformResult leftSideResult = emptyEvaluator.finishBundle();
+
+assertThat(leftSideResult.getOutputBundles(), emptyIterable());
+assertThat(
+leftSideResult.getTransform(),
+Matchers.>equalTo(flattened.getProducingTransformInternal()));
+  }
+
 }

[08/50] [abbrv] incubator-beam git commit: Add KeyedWorkItemCoder

[jamesmalone]

Add KeyedWorkItemCoder

Release Notes
[]
-
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=115354168


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/635541a7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/635541a7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/635541a7

Branch: refs/heads/master
Commit: 635541a74438a02d18910fba6a76c5aacc2c7816
Parents: 3904c90
Author: tgroh 
Authored: Tue Feb 23 10:12:17 2016 -0800
Committer: Davor Bonaci 
Committed: Thu Feb 25 23:58:26 2016 -0800

--
 .../dataflow/sdk/util/KeyedWorkItemCoder.java   | 120 +++
 .../cloud/dataflow/sdk/util/KeyedWorkItems.java |  29 -
 .../sdk/util/KeyedWorkItemCoderTest.java|  61 ++
 3 files changed, 209 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/635541a7/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/KeyedWorkItemCoder.java
--
diff --git 
a/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/KeyedWorkItemCoder.java 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/KeyedWorkItemCoder.java
new file mode 100644
index 000..398e82a
--- /dev/null
+++ 
b/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/KeyedWorkItemCoder.java
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) 2016 Google Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"); you may not
+ * use this file except in compliance with the License. You may obtain a copy 
of
+ * the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations 
under
+ * the License.
+ */
+package com.google.cloud.dataflow.sdk.util;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.cloud.dataflow.sdk.coders.Coder;
+import com.google.cloud.dataflow.sdk.coders.CoderException;
+import com.google.cloud.dataflow.sdk.coders.IterableCoder;
+import com.google.cloud.dataflow.sdk.coders.StandardCoder;
+import com.google.cloud.dataflow.sdk.transforms.windowing.BoundedWindow;
+import com.google.cloud.dataflow.sdk.util.TimerInternals.TimerData;
+import com.google.cloud.dataflow.sdk.util.TimerInternals.TimerDataCoder;
+import com.google.cloud.dataflow.sdk.util.WindowedValue.FullWindowedValueCoder;
+import com.google.common.collect.ImmutableList;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.List;
+
+/**
+ * A {@link Coder} for {@link KeyedWorkItem KeyedWorkItems}.
+ */
+public class KeyedWorkItemCoder extends 
StandardCoder> {
+  /**
+   * Create a new {@link KeyedWorkItemCoder} with the provided key coder, 
element coder, and window
+   * coder.
+   */
+  public static  KeyedWorkItemCoder of(
+  Coder keyCoder, Coder elemCoder, Coder windowCoder) {
+return new KeyedWorkItemCoder<>(keyCoder, elemCoder, windowCoder);
+  }
+
+  @JsonCreator
+  public static  KeyedWorkItemCoder of(
+  @JsonProperty(PropertyNames.COMPONENT_ENCODINGS) List> 
components) {
+checkArgument(components.size() == 3, "Expecting 3 components, got %s", 
components.size());
+@SuppressWarnings("unchecked")
+Coder keyCoder = (Coder) components.get(0);
+@SuppressWarnings("unchecked")
+Coder elemCoder = (Coder) components.get(1);
+@SuppressWarnings("unchecked")
+Coder windowCoder = (Coder) components.get(2);
+return new KeyedWorkItemCoder<>(keyCoder, elemCoder, windowCoder);
+  }
+
+  private final Coder keyCoder;
+  private final Coder elemCoder;
+  private final Coder windowCoder;
+  private final Coder> timersCoder;
+  private final Coder>> elemsCoder;
+
+  private KeyedWorkItemCoder(
+  Coder keyCoder, Coder elemCoder, Coder windowCoder) {
+this.keyCoder = keyCoder;
+this.elemCoder = elemCoder;
+this.windowCoder = windowCoder;
+this.timersCoder = IterableCoder.of(TimerDataCoder.of(windowCoder));
+this.elemsCoder = IterableCoder.of(FullWindowedValueCoder.of(elemCoder, 
windowCoder));
+  }
+
+  @Override
+  public void encode(KeyedWorkItem value, OutputStream outStream, 
Coder.Context context)
+  throws CoderException, IOException {
+Coder.Context nestedContext = context.nested();
+keyCoder.encode(value.key(), outStream, nestedContext);
+timersCoder.encode(value.timersIterable(), ou

svn commit: r1732428 - /incubator/beam/website/index.html

[jamesmalone]

Author: jamesmalone
Date: Fri Feb 26 06:49:19 2016
New Revision: 1732428

URL: http://svn.apache.org/viewvc?rev=1732428&view=rev
Log:
Update to index.

Modified:
incubator/beam/website/index.html

Modified: incubator/beam/website/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/index.html?rev=1732428&r1=1732427&r2=1732428&view=diff
==
--- incubator/beam/website/index.html (original)
+++ incubator/beam/website/index.html Fri Feb 26 06:49:19 2016
@@ -85,7 +85,8 @@
 
 
 Apache Beam (incubating)
-Apache Beam is an open source, unified model and set of language-specific 
SDKs for defining and executing data processing workflows, and also data 
ingestion and integration flows, supporting Enterprise Integration Patterns 
(EIPs) and Domain Specific Languages (DSLs). Beam pipelines simplify the 
mechanics of large-scale batch and streaming data processing and can run on a 
number of runtimes like Apache Flink, Apache Spark, and Google Cloud Dataflow 
(a cloud service). Beam also brings DSL in different languages, allowing users 
to easily implement their data integration processes.
+
+Apache Beam is an open source, unified model and set of language-specific 
SDKs for defining data processing workflows that may then be executed on top of 
a set of supported runners, currently including http://flink.apache.org";>Apache Flink, http://spark.apache.org";>Apache Spark, and https://cloud.google.com/dataflow";>Google Cloud Dataflow.
 
 Using Apache Beam
 You can use Beam for nearly any kind of data processing task, including 
both batch and streaming data processing. Beam provides a unified data model 
that can represent any size data set, including an unbounded or infinite data 
set from a continuously updating data source such as Kafka.

svn commit: r1732393 - /incubator/beam/website/blog/index.html

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:05:22 2016
New Revision: 1732393

URL: http://svn.apache.org/viewvc?rev=1732393&view=rev
Log:

New blog post

Modified:
incubator/beam/website/blog/index.html

Modified: incubator/beam/website/blog/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/blog/index.html?rev=1732393&r1=1732392&r2=1732393&view=diff
==
--- incubator/beam/website/blog/index.html (original)
+++ incubator/beam/website/blog/index.html Thu Feb 25 23:05:22 2016
@@ -86,7 +86,7 @@
 This is the blog for the Apache Beam project. This blog contains news 
and updates
 for the project.
 
-XcX
+Dataflow Python SDK is 
now public!
 Feb 25, 2016 - posted by James Malone [https://twitter.com/chimerasaurus";>@chimerasaurus]
 
 

svn commit: r1732394 - /incubator/beam/website/feed.xml

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:05:29 2016
New Revision: 1732394

URL: http://svn.apache.org/viewvc?rev=1732394&view=rev
Log:



Modified:
incubator/beam/website/feed.xml

Modified: incubator/beam/website/feed.xml
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/feed.xml?rev=1732394&r1=1732393&r2=1732394&view=diff
==
--- incubator/beam/website/feed.xml (original)
+++ incubator/beam/website/feed.xml Thu Feb 25 23:05:29 2016
@@ -6,12 +6,12 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Thu, 25 Feb 2016 15:01:53 -0800
-Thu, 25 Feb 2016 15:01:53 -0800
+Thu, 25 Feb 2016 15:04:09 -0800
+Thu, 25 Feb 2016 15:04:09 -0800
 Jekyll v3.1.1
 
   
-XcX
+Dataflow Python SDK is now public!
 <p>When the Apache Beam project proposed entry into 
the <a 
href="http://wiki.apache.org/incubator/BeamProposal">Apache 
Incubator</a> the proposal
 included the <a 
href="https://github.com/GoogleCloudPlatform/DataflowJavaSDK">Dataflow
 Java SDK</a>. In the long term, however, Apache Beam aims to support 
SDKs implemented in multiple languages, such as Python.</p>
 

svn commit: r1732395 - /incubator/beam/website/index.html

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:05:34 2016
New Revision: 1732395

URL: http://svn.apache.org/viewvc?rev=1732395&view=rev
Log:


Modified:
incubator/beam/website/index.html

Modified: incubator/beam/website/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/index.html?rev=1732395&r1=1732394&r2=1732395&view=diff
==
--- incubator/beam/website/index.html (original)
+++ incubator/beam/website/index.html Thu Feb 25 23:05:34 2016
@@ -115,7 +115,7 @@
 Blog
 
 
-Feb 25, 2016 - XcX
+Feb 25, 2016 - 
Dataflow Python SDK is now public!
 
 Feb 22, 2016 - 
Apache Beam has a logo!
 

svn commit: r1732392 - /incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:04:58 2016
New Revision: 1732392

URL: http://svn.apache.org/viewvc?rev=1732392&view=rev
Log:
Update to blog post

Modified:
incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html

Modified: 
incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html?rev=1732392&r1=1732391&r2=1732392&view=diff
==
--- incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html 
(original)
+++ incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html Thu 
Feb 25 23:04:58 2016
@@ -6,7 +6,7 @@
   
   
 
-  XcX
+  Dataflow Python SDK is now public!
   
 
   
@@ -78,7 +78,7 @@
 http://schema.org/BlogPosting";>
 
   
-XcX
+Dataflow Python SDK is now 
public!
 Feb 25, 2016 • http://schema.org/Person";>jamesmalone
   
 

svn commit: r1732390 - in /incubator/beam/website/beam/python: ./ sdk/ sdk/2016/ sdk/2016/02/ sdk/2016/02/25/ sdk/2016/02/25/beam-has-a-logo0.html

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:03:18 2016
New Revision: 1732390

URL: http://svn.apache.org/viewvc?rev=1732390&view=rev
Log:

New blog post

Added:
incubator/beam/website/beam/python/
incubator/beam/website/beam/python/sdk/
incubator/beam/website/beam/python/sdk/2016/
incubator/beam/website/beam/python/sdk/2016/02/
incubator/beam/website/beam/python/sdk/2016/02/25/
incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html

Added: incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html?rev=1732390&view=auto
==
--- incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html 
(added)
+++ incubator/beam/website/beam/python/sdk/2016/02/25/beam-has-a-logo0.html Thu 
Feb 25 23:03:18 2016
@@ -0,0 +1,123 @@
+
+
+
+  
+  
+  
+  
+
+  XcX
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"</a>;>
+  
+  http://beam.incubator.apache.org/beam/python/sdk/2016/02/25/beam-has-a-logo0.html";>
+  http://beam.incubator.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+
+
+  
+
+  Documentation 
+  
+Getting Started
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+  
+
+
+  Community 
+  
+Community
+Mailing Lists
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+Apache Beam Team
+
+Contribute
+Source Repository
+Issue Tracking
+  
+
+    Blog
+  
+
+  
+
+
+
+
+
+
+
+
+  
+http://schema.org/BlogPosting";>
+
+  
+XcX
+Feb 25, 2016 • http://schema.org/Person";>jamesmalone
+  
+
+  
+When the Apache Beam project proposed entry into the http://wiki.apache.org/incubator/BeamProposal";>Apache Incubator the 
proposal
+included the https://github.com/GoogleCloudPlatform/DataflowJavaSDK";>Dataflow Java 
SDK. In the long term, however, Apache Beam aims to support SDKs 
implemented in multiple languages, such as Python.
+
+
+
+Today, Google submitted the http://github.com/GoogleCloudPlatform/DataflowPythonSDK";>Dataflow Python 
(2.x) SDK on GitHub. Google is committed to including the in progress 
python SDK in Apache Beam and, in that spirit, we’ve moved development of the 
Python SDK to a public repository. While this SDK will not be included with the 
initial (incubating) releases of Apache Beam, our we plan on incorporating the 
Python SDK into beam during incubation. We want to take the time to implement 
changes from the https://goo.gl/nk5OM0";>technical vision into the 
Java SDK before we introduce a Python SDK for Apache Beam. We believe this will 
allow us to work on the model and SDKs in an ordered fashion.
+
+You can look for the Apache Beam Python SDK in the coming months once we 
finish forking and refactoring the Java SDK.
+
+Best,
+
+Apache Beam Team
+
+  
+
+
+
+  
+
+
+
+  
+  
+  
+  © Copyright 2016
+http://www.apache.org";>The Apache Software 
Foundation. All Rights Reserved.
+Privacy 
Policy |
+RSS Feed
+  
+  
+  
+  
+
+
+
+  
+
+

svn commit: r1732389 - in /incubator/beam/website: blog/index.html feed.xml index.html

[jamesmalone]

Author: jamesmalone
Date: Thu Feb 25 23:02:52 2016
New Revision: 1732389

URL: http://svn.apache.org/viewvc?rev=1732389&view=rev
Log:
Edits to page; new blog post.

Modified:
incubator/beam/website/blog/index.html
incubator/beam/website/feed.xml
incubator/beam/website/index.html

Modified: incubator/beam/website/blog/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/blog/index.html?rev=1732389&r1=1732388&r2=1732389&view=diff
==
--- incubator/beam/website/blog/index.html (original)
+++ incubator/beam/website/blog/index.html Thu Feb 25 23:02:52 2016
@@ -86,6 +86,15 @@
 This is the blog for the Apache Beam project. This blog contains news 
and updates
 for the project.
 
+XcX
+Feb 25, 2016 - posted by James Malone [https://twitter.com/chimerasaurus";>@chimerasaurus]
+
+
+When the Apache Beam project proposed entry into the http://wiki.apache.org/incubator/BeamProposal";>Apache Incubator the 
proposal
+included the https://github.com/GoogleCloudPlatform/DataflowJavaSDK";>Dataflow Java 
SDK. In the long term, however, Apache Beam aims to support SDKs 
implemented in multiple languages, such as Python.
+
+
+
 Apache Beam has a 
logo!
 Feb 22, 2016 - posted by James Malone [https://twitter.com/chimerasaurus";>@chimerasaurus]
 

Modified: incubator/beam/website/feed.xml
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/feed.xml?rev=1732389&r1=1732388&r2=1732389&view=diff
==
--- incubator/beam/website/feed.xml (original)
+++ incubator/beam/website/feed.xml Thu Feb 25 23:02:52 2016
@@ -6,11 +6,39 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Wed, 24 Feb 2016 09:34:19 -0800
-Wed, 24 Feb 2016 09:34:19 -0800
+Thu, 25 Feb 2016 15:01:53 -0800
+Thu, 25 Feb 2016 15:01:53 -0800
 Jekyll v3.1.1
 
   
+XcX
+<p>When the Apache Beam project proposed entry into 
the <a 
href="http://wiki.apache.org/incubator/BeamProposal">Apache 
Incubator</a> the proposal
+included the <a 
href="https://github.com/GoogleCloudPlatform/DataflowJavaSDK">Dataflow
 Java SDK</a>. In the long term, however, Apache Beam aims to support 
SDKs implemented in multiple languages, such as Python.</p>
+
+<!--more-->
+
+<p>Today, Google submitted the <a 
href="http://github.com/GoogleCloudPlatform/DataflowPythonSDK">Dataflow
 Python (2.x) SDK</a> on GitHub. Google is committed to including the in 
progress python SDK in Apache Beam and, in that spirit, we’ve moved 
development of the Python SDK to a public repository. While this SDK will not 
be included with the initial (incubating) releases of Apache Beam, our we plan 
on incorporating the Python SDK into beam during incubation. We want to take 
the time to implement changes from the <a 
href="https://goo.gl/nk5OM0">technical vision</a> into the 
Java SDK before we introduce a Python SDK for Apache Beam. We believe this will 
allow us to work on the model and SDKs in an ordered fashion.</p>
+
+<p>You can look for the Apache Beam Python SDK in the coming months once 
we finish forking and refactoring the Java SDK.</p>
+
+<p>Best,</p>
+
+<p>Apache Beam Team</p>
+
+Thu, 25 Feb 2016 13:00:00 -0800
+
http://beam.incubator.apache.org/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
+http://beam.incubator.apache.org/beam/python/sdk/2016/02/25/beam-has-a-logo0.html
+
+
+beam
+
+python
+
+sdk
+
+  
+
+  
 Apache Beam has a logo!
 <p>One of the major benefits of Apache Beam is the 
fact that it unifies both
 both batch and stream processing into one powerful model. In fact, this 
unification

Modified: incubator/beam/website/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/index.html?rev=1732389&r1=1732388&r2=1732389&view=diff
==
--- incubator/beam/website/index.html (original)
+++ incubator/beam/website/index.html Thu Feb 25 23:02:52 2016
@@ -115,6 +115,8 @@
 Blog
 
 
+Feb 25, 2016 - XcX
+
 Feb 22, 2016 - 
Apache Beam has a logo!
 
 

svn commit: r1732196 - /incubator/beam/website/images/favicon.ico

[jamesmalone]

Author: jamesmalone
Date: Wed Feb 24 17:59:00 2016
New Revision: 1732196

URL: http://svn.apache.org/viewvc?rev=1732196&view=rev
Log:
Favicon

Added:
incubator/beam/website/images/favicon.ico   (with props)

Added: incubator/beam/website/images/favicon.ico
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/images/favicon.ico?rev=1732196&view=auto
==
Binary file - no diff available.

Propchange: incubator/beam/website/images/favicon.ico
--
svn:mime-type = application/octet-stream

svn commit: r1732194 - in /incubator/beam/website: ./ beam/update/website/2016/02/22/ blog/ getting_started/ images/ issue_tracking/ mailing_lists/ privacy_policy/ source_repository/ team/

[jamesmalone]

Author: jamesmalone
Date: Wed Feb 24 17:35:05 2016
New Revision: 1732194

URL: http://svn.apache.org/viewvc?rev=1732194&view=rev
Log:
Logo updates

Modified:
incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html
incubator/beam/website/blog/index.html
incubator/beam/website/feed.xml
incubator/beam/website/getting_started/index.html
incubator/beam/website/images/beam_logo_navbar.png
incubator/beam/website/index.html
incubator/beam/website/issue_tracking/index.html
incubator/beam/website/mailing_lists/index.html
incubator/beam/website/privacy_policy/index.html
incubator/beam/website/source_repository/index.html
incubator/beam/website/team/index.html

Modified: 
incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html 
(original)
+++ incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html 
Wed Feb 24 17:35:05 2016
@@ -25,6 +25,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/blog/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/blog/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/blog/index.html (original)
+++ incubator/beam/website/blog/index.html Wed Feb 24 17:35:05 2016
@@ -26,6 +26,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/feed.xml
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/feed.xml?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/feed.xml (original)
+++ incubator/beam/website/feed.xml Wed Feb 24 17:35:05 2016
@@ -6,8 +6,8 @@
 
 http://beam.incubator.apache.org/
 http://beam.incubator.apache.org/feed.xml"; rel="self" 
type="application/rss+xml"/>
-Tue, 23 Feb 2016 21:53:13 -0800
-Tue, 23 Feb 2016 21:53:13 -0800
+Wed, 24 Feb 2016 09:34:19 -0800
+Wed, 24 Feb 2016 09:34:19 -0800
 Jekyll v3.1.1
 
   

Modified: incubator/beam/website/getting_started/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/getting_started/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/getting_started/index.html (original)
+++ incubator/beam/website/getting_started/index.html Wed Feb 24 17:35:05 2016
@@ -26,6 +26,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/images/beam_logo_navbar.png
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/images/beam_logo_navbar.png?rev=1732194&r1=1732193&r2=1732194&view=diff
==
Binary files - no diff available.

Modified: incubator/beam/website/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/index.html (original)
+++ incubator/beam/website/index.html Wed Feb 24 17:35:05 2016
@@ -26,6 +26,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/issue_tracking/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/issue_tracking/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/issue_tracking/index.html (original)
+++ incubator/beam/website/issue_tracking/index.html Wed Feb 24 17:35:05 2016
@@ -26,6 +26,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/mailing_lists/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/mailing_lists/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff
==
--- incubator/beam/website/mailing_lists/index.html (original)
+++ incubator/beam/website/mailing_lists/index.html Wed Feb 24 17:35:05 2016
@@ -26,6 +26,7 @@
 ga('send', 'pageview');
 
   
+  
 
 
 

Modified: incubator/beam/website/privacy_policy/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/privacy_policy/index.html?rev=1732194&r1=1732193&r2=1732194&view=diff

svn commit: r1732191 - /incubator/beam/website/team/index.html

[jamesmalone]

Author: jamesmalone
Date: Wed Feb 24 17:08:46 2016
New Revision: 1732191

URL: http://svn.apache.org/viewvc?rev=1732191&view=rev
Log:
Update to teams page

Modified:
incubator/beam/website/team/index.html

Modified: incubator/beam/website/team/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/team/index.html?rev=1732191&r1=1732190&r2=1732191&view=diff
==
--- incubator/beam/website/team/index.html (original)
+++ incubator/beam/website/team/index.html Wed Feb 24 17:08:46 2016
@@ -202,7 +202,7 @@
   
 
   Kenneth Knowles
-  
+  kenn
   mailto:k...@google.com";>k...@google.com
   Google
   committer, PPMC

svn commit: r1732190 - in /incubator/beam/website: index.html team/index.html

[jamesmalone]

Author: jamesmalone
Date: Wed Feb 24 16:58:27 2016
New Revision: 1732190

URL: http://svn.apache.org/viewvc?rev=1732190&view=rev
Log:
Fixes and additions to website

Modified:
incubator/beam/website/index.html
incubator/beam/website/team/index.html

Modified: incubator/beam/website/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/index.html?rev=1732190&r1=1732189&r2=1732190&view=diff
==
--- incubator/beam/website/index.html (original)
+++ incubator/beam/website/index.html Wed Feb 24 16:58:27 2016
@@ -107,6 +107,24 @@
 
 See the https://cloud.google.com/dataflow/model/programming-model";>programming 
model documentation to learn more about how Beam implements these 
concepts.
 
+
+
+
+  
+Blog
+
+
+Feb 22, 2016 - 
Apache Beam has a logo!
+
+
+  
+  
+Twitter
+https://twitter.com/ApacheBeam"; 
data-widget-id="697809684422533120">Tweets by @ApacheBeam
+!function(d,s,id){var 
js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");
+  
+
+
 Apache Project
 Apache Beam is an http://www.apache.org";>Apache Software 
Foundation project,
 available under the Apache v2 license.

Modified: incubator/beam/website/team/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/team/index.html?rev=1732190&r1=1732189&r2=1732190&view=diff
==
--- incubator/beam/website/team/index.html (original)
+++ incubator/beam/website/team/index.html Wed Feb 24 16:58:27 2016
@@ -104,16 +104,16 @@
 
   Aljoscha Krettek
   aljoscha
-  mailto:member.email";>aljos...@apache.org
+  mailto:aljos...@apache.org";>aljos...@apache.org
   data Artisans
   committer, PPMC
-  1
+  +1
 
   
 
   Amit Sela
   amitsela
-  mailto:member.email";>amits...@apache.org
+  mailto:amits...@apache.org";>amits...@apache.org
   PayPal
   committer, PPMC
   +2
@@ -122,7 +122,7 @@
 
   Ben Chambers
   
-  mailto:member.email";>bchamb...@google.com
+  mailto:bchamb...@google.com";>bchamb...@google.com
   Google
   committer, PPMC
   -8
@@ -131,7 +131,7 @@
 
   Craig Chambers
   
-  mailto:member.email";>chamb...@google.com
+  mailto:chamb...@google.com";>chamb...@google.com
   Google
   committer, PPMC
   -8
@@ -140,7 +140,7 @@
 
   Dan Halperin
   
-  mailto:member.email";>dhalp...@google.com
+  mailto:dhalp...@google.com";>dhalp...@google.com
   Google
   committer, PPMC
   -8
@@ -149,7 +149,7 @@
 
   Davor Bonaci
   
-  mailto:member.email";>da...@google.com
+  mailto:da...@google.com";>da...@google.com
   Google
   committer, PPMC
   -8
@@ -158,7 +158,7 @@
 
   Frances Perry
   
-  mailto:member.email";>f...@google.com
+  mailto:f...@google.com";>f...@google.com
   Google
   committer, PPMC
   -8
@@ -167,7 +167,7 @@
 
   James Malone
   jamesmalone
-  mailto:member.email";>jamesmal...@google.com
+  mailto:jamesmal...@google.com";>jamesmal...@google.com
   Google
   committer, PPMC
   -8
@@ -176,7 +176,7 @@
 
   Jean-Baptiste Onofré
   jbonofre
-  mailto:member.email";>da...@google.com
+  mailto:da...@google.com";>da...@google.com
   Talend
   champion, committer, PPMC
   +1
@@ -185,7 +185,7 @@
 
   Josh Wills
   jwills
-  mailto:member.email";>jwi...@apache.org
+  mailto:jwi...@apache.org";>jwi...@apache.org
   
   committer, PPMC
   -8
@@ -194,7 +194,7 @@
 
   Kostas Tzoumas
   
-  mailto:member.email";>kos...@data-artisans.com
+  mailto:kos...@data-artisans.com";>kos...@data-artisans.com
   data Artisans
   committer, PPMC
   +1
@@ -203,7 +203,7 @@
 
   Kenneth Knowles
   
-  mailto:member.email";>k...@google.com
+  mailto:k...@google.com";>k...@google.com
   Google
   com

svn commit: r1732024 [3/8] - in /incubator/beam/website: ./ beam/ beam/update/ beam/update/website/ beam/update/website/2016/ beam/update/website/2016/02/ beam/update/website/2016/02/22/ blog/ css/ fo

[jamesmalone]

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.svg
--
svn:executable = 

Added: incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.ttf
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.ttf?rev=1732024&view=auto
==
Binary file - no diff available.

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.ttf
--
svn:executable = 

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.ttf
--
svn:mime-type = application/octet-stream

Added: incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff?rev=1732024&view=auto
==
Binary file - no diff available.

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff
--
svn:executable = 

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff
--
svn:mime-type = application/octet-stream

Added: incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff2
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff2?rev=1732024&view=auto
==
Binary file - no diff available.

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff2
--
svn:executable = 

Propchange: 
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff2
--
svn:mime-type = application/octet-stream

Added: incubator/beam/website/getting_started/index.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/getting_started/index.html?rev=1732024&view=auto
==
--- incubator/beam/website/getting_started/index.html (added)
+++ incubator/beam/website/getting_started/index.html Wed Feb 24 05:56:18 2016
@@ -0,0 +1,143 @@
+
+
+
+  
+  
+  
+  
+
+  Getting started with Apache Beam
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  http://beam.incubator.apache.org/getting_started/";>
+  http://beam.incubator.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+
+
+  
+
+  Documentation 
+  
+Getting Started
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+  
+
+
+  Community 
+  
+Community
+Mailing Lists
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+Apache Beam Team
+
+Contribute
+Source Repository
+Issue Tracking
+  
+
+Blog
+  
+
+  
+
+
+
+
+
+
+
+
+  
+
+
+  
+Getting started with Apache Beam
+  
+
+  
+The Apache Beam project is in the process of bootstrapping. This 
includes the creation of project resources, the refactoring of the initial code 
submission, and the formulation of project documentation, planning, and design 
documents. Until the project is fully initialized, this page contains useful 
resources to learn more about the model and tools which comprise Apache 
Beam.
+
+Articles & slides
+
+  https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101";>The 
world beyond batch: Streaming 101
+  https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102";>The 
world beyong batch: Streaming 102
+  https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison";>Dataflow/Beam
 & Spark: A Programming Model Comparison
+  http://blog.nanthrax.net/2016/01/introduci

svn commit: r1732024 [5/8] - in /incubator/beam/website: ./ beam/ beam/update/ beam/update/website/ beam/update/website/2016/ beam/update/website/2016/02/ beam/update/website/2016/02/22/ blog/ css/ fo

[jamesmalone]

Added: incubator/beam/website/js/bootstrap.min.js
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/js/bootstrap.min.js?rev=1732024&view=auto
==
--- incubator/beam/website/js/bootstrap.min.js (added)
+++ incubator/beam/website/js/bootstrap.min.js Wed Feb 24 05:56:18 2016
@@ -0,0 +1,7 @@
+/*!
+ * Bootstrap v3.3.6 (http://getbootstrap.com)
+ * Copyright 2011-2015 Twitter, Inc.
+ * Licensed under the MIT license
+ */
+if("undefined"==typeof jQuery)throw new Error("Bootstrap's JavaScript requires 
jQuery");+function(a){"use strict";var b=a.fn.jquery.split(" 
")[0].split(".");if(b[0]<2&&b[1]<9||1==b[0]&&9==b[1]&&b[2]<1||b[0]>2)throw new 
Error("Bootstrap's JavaScript requires jQuery version 1.9.1 or higher, but 
lower than version 3")}(jQuery),+function(a){"use strict";function b(){var 
a=document.createElement("bootstrap"),b={WebkitTransition:"webkitTransitionEnd",MozTransition:"transitionend",OTransition:"oTransitionEnd
 otransitionend",transition:"transitionend"};for(var c in b)if(void 
0!==a.style[c])return{end:b[c]};return!1}a.fn.emulateTransitionEnd=function(b){var
 c=!1,d=this;a(this).one("bsTransitionEnd",function(){c=!0});var 
e=function(){c||a(d).trigger(a.support.transition.end)};return 
setTimeout(e,b),this},a(function(){a.support.transition=b(),a.support.transition&&(a.event.special.bsTransitionEnd={bindType:a.support.transition.end,delegateType:a.support.transition.end,handle:function(b){return
  a(b.target).is(this)?b.handleObj.handler.apply(this,arguments):void 
0}})})}(jQuery),+function(a){"use strict";function b(b){return 
this.each(function(){var 
c=a(this),e=c.data("bs.alert");e||c.data("bs.alert",e=new 
d(this)),"string"==typeof b&&e[b].call(c)})}var 
c='[data-dismiss="alert"]',d=function(b){a(b).on("click",c,this.close)};d.VERSION="3.3.6",d.TRANSITION_DURATION=150,d.prototype.close=function(b){function
 c(){g.detach().trigger("closed.bs.alert").remove()}var 
e=a(this),f=e.attr("data-target");f||(f=e.attr("href"),f=f&&f.replace(/.*(?=#[^\s]*$)/,""));var
 
g=a(f);b&&b.preventDefault(),g.length||(g=e.closest(".alert")),g.trigger(b=a.Event("close.bs.alert")),b.isDefaultPrevented()||(g.removeClass("in"),a.support.transition&&g.hasClass("fade")?g.one("bsTransitionEnd",c).emulateTransitionEnd(d.TRANSITION_DURATION):c())};var
 
e=a.fn.alert;a.fn.alert=b,a.fn.alert.Constructor=d,a.fn.alert.noConflict=function(){return
 a.fn.alert=e,this},a(document).on("click.bs.alert.data-api",c,d.prot
 otype.close)}(jQuery),+function(a){"use strict";function b(b){return 
this.each(function(){var d=a(this),e=d.data("bs.button"),f="object"==typeof 
b&&b;e||d.data("bs.button",e=new 
c(this,f)),"toggle"==b?e.toggle():b&&e.setState(b)})}var 
c=function(b,d){this.$element=a(b),this.options=a.extend({},c.DEFAULTS,d),this.isLoading=!1};c.VERSION="3.3.6",c.DEFAULTS={loadingText:"loading..."},c.prototype.setState=function(b){var
 
c="disabled",d=this.$element,e=d.is("input")?"val":"html",f=d.data();b+="Text",null==f.resetText&&d.data("resetText",d[e]()),setTimeout(a.proxy(function(){d[e](null==f[b]?this.options[b]:f[b]),"loadingText"==b?(this.isLoading=!0,d.addClass(c).attr(c,c)):this.isLoading&&(this.isLoading=!1,d.removeClass(c).removeAttr(c))},this),0)},c.prototype.toggle=function(){var
 a=!0,b=this.$element.closest('[data-toggle="buttons"]');if(b.length){var 
c=this.$element.find("input");"radio"==c.prop("type")?(c.prop("checked")&&(a=!1),b.find(".active").removeClass("active"),this.$element.ad
 
dClass("active")):"checkbox"==c.prop("type")&&(c.prop("checked")!==this.$element.hasClass("active")&&(a=!1),this.$element.toggleClass("active")),c.prop("checked",this.$element.hasClass("active")),a&&c.trigger("change")}else
 
this.$element.attr("aria-pressed",!this.$element.hasClass("active")),this.$element.toggleClass("active")};var
 
d=a.fn.button;a.fn.button=b,a.fn.button.Constructor=c,a.fn.button.noConflict=function(){return
 
a.fn.button=d,this},a(document).on("click.bs.button.data-api",'[data-toggle^="button"]',function(c){var
 
d=a(c.target);d.hasClass("btn")||(d=d.closest(".btn")),b.call(d,"toggle"),a(c.target).is('input[type="radio"]')||a(c.target).is('input[type="checkbox"]')||c.preventDefault()}).on("focus.bs.button.data-api
 
blur.bs.button.data-api",'[data-toggle^="button"]',function(b){a(b.target).closest(".btn").toggleClass("focus",/^focus(in)?$/.test(b.type))})}(jQuery),+function(a){"use
 strict";function b(b){return this.each(function(){var 
d=a(this),e=d.data("bs.carousel"),f=
 a.extend({},c.DEFAULTS,d.data(),"object"==typeof b&&b),g="string"==typeof 
b?b:f.slide;e||d.data("bs.carousel",e=new c(this,f)),"number"==typeof 
b?e.to(b):g?e[g]():f.interval&&e.pause().cycle()})}var 
c=function(b,c){this.$element=a(b),this.$indicators=this.$element.find(".carousel-indicators"),this.options=c,this.paused=null,this.sliding=null,this.interval=null,this.$active=null,this.$items=null,this.options.keyboard&&this.$element.on("keydown.bs.caro

svn commit: r1732024 [4/8] - in /incubator/beam/website: ./ beam/ beam/update/ beam/update/website/ beam/update/website/2016/ beam/update/website/2016/02/ beam/update/website/2016/02/22/ blog/ css/ fo

[jamesmalone]

Added: incubator/beam/website/js/bootstrap.js
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/js/bootstrap.js?rev=1732024&view=auto
==
--- incubator/beam/website/js/bootstrap.js (added)
+++ incubator/beam/website/js/bootstrap.js Wed Feb 24 05:56:18 2016
@@ -0,0 +1,2363 @@
+/*!
+ * Bootstrap v3.3.6 (http://getbootstrap.com)
+ * Copyright 2011-2015 Twitter, Inc.
+ * Licensed under the MIT license
+ */
+
+if (typeof jQuery === 'undefined') {
+  throw new Error('Bootstrap\'s JavaScript requires jQuery')
+}
+
++function ($) {
+  'use strict';
+  var version = $.fn.jquery.split(' ')[0].split('.')
+  if ((version[0] < 2 && version[1] < 9) || (version[0] == 1 && version[1] == 
9 && version[2] < 1) || (version[0] > 2)) {
+throw new Error('Bootstrap\'s JavaScript requires jQuery version 1.9.1 or 
higher, but lower than version 3')
+  }
+}(jQuery);
+
+/* 
+ * Bootstrap: transition.js v3.3.6
+ * http://getbootstrap.com/javascript/#transitions
+ * 
+ * Copyright 2011-2015 Twitter, Inc.
+ * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
+ *  */
+
+
++function ($) {
+  'use strict';
+
+  // CSS TRANSITION SUPPORT (Shoutout: http://www.modernizr.com/)
+  // 
+
+  function transitionEnd() {
+var el = document.createElement('bootstrap')
+
+var transEndEventNames = {
+  WebkitTransition : 'webkitTransitionEnd',
+  MozTransition: 'transitionend',
+  OTransition  : 'oTransitionEnd otransitionend',
+  transition   : 'transitionend'
+}
+
+for (var name in transEndEventNames) {
+  if (el.style[name] !== undefined) {
+return { end: transEndEventNames[name] }
+  }
+}
+
+return false // explicit for ie8 (  ._.)
+  }
+
+  // http://blog.alexmaccaw.com/css-transitions
+  $.fn.emulateTransitionEnd = function (duration) {
+var called = false
+var $el = this
+$(this).one('bsTransitionEnd', function () { called = true })
+var callback = function () { if (!called) 
$($el).trigger($.support.transition.end) }
+setTimeout(callback, duration)
+return this
+  }
+
+  $(function () {
+$.support.transition = transitionEnd()
+
+if (!$.support.transition) return
+
+$.event.special.bsTransitionEnd = {
+  bindType: $.support.transition.end,
+  delegateType: $.support.transition.end,
+  handle: function (e) {
+if ($(e.target).is(this)) return e.handleObj.handler.apply(this, 
arguments)
+  }
+}
+  })
+
+}(jQuery);
+
+/* 
+ * Bootstrap: alert.js v3.3.6
+ * http://getbootstrap.com/javascript/#alerts
+ * 
+ * Copyright 2011-2015 Twitter, Inc.
+ * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)
+ *  */
+
+
++function ($) {
+  'use strict';
+
+  // ALERT CLASS DEFINITION
+  // ==
+
+  var dismiss = '[data-dismiss="alert"]'
+  var Alert   = function (el) {
+$(el).on('click', dismiss, this.close)
+  }
+
+  Alert.VERSION = '3.3.6'
+
+  Alert.TRANSITION_DURATION = 150
+
+  Alert.prototype.close = function (e) {
+var $this= $(this)
+var selector = $this.attr('data-target')
+
+if (!selector) {
+  selector = $this.attr('href')
+  selector = selector && selector.replace(/.*(?=#[^\s]*$)/, '') // strip 
for ie7
+}
+
+var $parent = $(selector)
+
+if (e) e.preventDefault()
+
+if (!$parent.length) {
+  $parent = $this.closest('.alert')
+}
+
+$parent.trigger(e = $.Event('close.bs.alert'))
+
+if (e.isDefaultPrevented()) return
+
+$parent.removeClass('in')
+
+function removeElement() {
+  // detach from parent, fire event then clean up data
+  $parent.detach().trigger('closed.bs.alert').remove()
+}
+
+$.support.transition && $parent.hasClass('fade') ?
+  $parent
+.one('bsTransitionEnd', removeElement)
+.emulateTransitionEnd(Alert.TRANSITION_DURATION) :
+  removeElement()
+  }
+
+
+  // ALERT PLUGIN DEFINITION
+  // ===
+
+  function Plugin(option) {
+return this.each(function () {
+  var $this = $(this)
+  var data  = $this.data('bs.alert')
+
+  if (!data) $this.data('bs.alert', (data = new Alert(this)))
+  if (typeof option == 'string') data[option].call($this)
+})
+  }
+
+  var old = $.fn.alert
+
+  $.fn.alert = Plugin
+  $.fn.alert.Constructor = Alert
+
+
+  // ALERT NO CONFLICT
+  // =
+
+  $.fn.alert.noConflict = funct

svn commit: r1732024 [1/8] - in /incubator/beam/website: ./ beam/ beam/update/ beam/update/website/ beam/update/website/2016/ beam/update/website/2016/02/ beam/update/website/2016/02/22/ blog/ css/ fo

[jamesmalone]

Author: jamesmalone
Date: Wed Feb 24 05:56:18 2016
New Revision: 1732024

URL: http://svn.apache.org/viewvc?rev=1732024&view=rev
Log:
Addition of new Beam site (hopefully)

Added:
incubator/beam/website/beam/
incubator/beam/website/beam/update/
incubator/beam/website/beam/update/website/
incubator/beam/website/beam/update/website/2016/
incubator/beam/website/beam/update/website/2016/02/
incubator/beam/website/beam/update/website/2016/02/22/
incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html
incubator/beam/website/blog/
incubator/beam/website/blog/index.html
incubator/beam/website/css/
incubator/beam/website/css/main.css
incubator/beam/website/css/theme.css
incubator/beam/website/feed.xml
incubator/beam/website/fonts/
incubator/beam/website/fonts/bootstrap/
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.eot   
(with props)
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.svg   
(with props)
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.ttf   
(with props)
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff   
(with props)
incubator/beam/website/fonts/bootstrap/glyphicons-halflings-regular.woff2   
(with props)
incubator/beam/website/getting_started/
incubator/beam/website/getting_started/index.html
incubator/beam/website/images/
incubator/beam/website/images/beam_logo_navbar.png   (with props)
incubator/beam/website/images/beam_logo_s.png   (with props)
incubator/beam/website/index.html
incubator/beam/website/issue_tracking/
incubator/beam/website/issue_tracking/index.html
incubator/beam/website/js/
incubator/beam/website/js/bootstrap/
incubator/beam/website/js/bootstrap-sprockets.js   (with props)
incubator/beam/website/js/bootstrap.js   (with props)
incubator/beam/website/js/bootstrap.min.js   (with props)
incubator/beam/website/js/bootstrap/affix.js   (with props)
incubator/beam/website/js/bootstrap/alert.js   (with props)
incubator/beam/website/js/bootstrap/button.js   (with props)
incubator/beam/website/js/bootstrap/carousel.js   (with props)
incubator/beam/website/js/bootstrap/collapse.js   (with props)
incubator/beam/website/js/bootstrap/dropdown.js   (with props)
incubator/beam/website/js/bootstrap/modal.js   (with props)
incubator/beam/website/js/bootstrap/popover.js   (with props)
incubator/beam/website/js/bootstrap/scrollspy.js   (with props)
incubator/beam/website/js/bootstrap/tab.js   (with props)
incubator/beam/website/js/bootstrap/tooltip.js   (with props)
incubator/beam/website/js/bootstrap/transition.js   (with props)
incubator/beam/website/mailing_lists/
incubator/beam/website/mailing_lists/index.html
incubator/beam/website/privacy_policy/
incubator/beam/website/privacy_policy/index.html
incubator/beam/website/source_repository/
incubator/beam/website/source_repository/index.html
incubator/beam/website/styles/
incubator/beam/website/styles/site.css
incubator/beam/website/team/
incubator/beam/website/team/index.html

Added: 
incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html
URL: 
http://svn.apache.org/viewvc/incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html?rev=1732024&view=auto
==
--- incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html 
(added)
+++ incubator/beam/website/beam/update/website/2016/02/22/beam-has-a-logo.html 
Wed Feb 24 05:56:18 2016
@@ -0,0 +1,128 @@
+
+
+
+  
+  
+  
+  
+
+  Apache Beam has a logo!
+  
+
+  
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js"</a>;>
+  
+  http://beam.incubator.apache.org/beam/update/website/2016/02/22/beam-has-a-logo.html";>
+  http://beam.incubator.apache.org/feed.xml";>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+
+  
+
+
+
+  
+
+
+  
+
+  
+
+  
+
+
+  
+
+  Documentation 
+  
+Getting Started
+https://goo.gl/ps8twC";>Technical Docs
+https://goo.gl/nk5OM0";>Technical Vision
+  
+
+
+  Community 
+  
+Community
+Mailing Lists
+https://goo.gl/ps8twC";>Tech