[GitHub] incubator-beam-site pull request #86: Adjust main page header to look like v...

[francesperry]

Github user francesperry closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/86


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666251#comment-15666251
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1360

[BEAM-498] Rename DoFn.ExtraContextFactory to DoFn.Arguments

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

I think we've discussed this before. Big picture: now we can treat all 
dynamic methods uniformly, and use `DoFnSignatures` and `ParDo` to do the 
needed validation. I think since all arguments to methods fall into the "extra" 
category, we can save some typing and use more conventional vocabulary.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam Arguments

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1360


commit 4afc16fad9eeaeee1d640822000da6a1062c4056
Author: Kenneth Knowles 
Date:   2016-11-15T06:19:35Z

Rename DoFn.ExtraContextFactory to DoFn.Arguments




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1360: [BEAM-498] Rename DoFn.ExtraContextFactor...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1360

[BEAM-498] Rename DoFn.ExtraContextFactory to DoFn.Arguments

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

I think we've discussed this before. Big picture: now we can treat all 
dynamic methods uniformly, and use `DoFnSignatures` and `ParDo` to do the 
needed validation. I think since all arguments to methods fall into the "extra" 
category, we can save some typing and use more conventional vocabulary.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam Arguments

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1360


commit 4afc16fad9eeaeee1d640822000da6a1062c4056
Author: Kenneth Knowles 
Date:   2016-11-15T06:19:35Z

Rename DoFn.ExtraContextFactory to DoFn.Arguments




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666237#comment-15666237
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1355


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Treat ProcessContext and Context like other DoFn parameters

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 0117a2834 -> 13ad8f68a


Treat ProcessContext and Context like other DoFn parameters

Before this change, ProcessContext and Context were special-cased, while
other parameters were treated generically. After this change, all parameters
receive the same dynamic treatment.

The reasons/benefits for this:

 - The code to invoke a parameterized DoFn method can be shared across all
   DoFn methods. Currently the special-case for ProcessContext complicates
   sharing with @StartBundle, @SetUp, @OnTimer, etc.
 - The user need not have these "all-in-one" parameters, so we can start
   to replace them with simpler parameters, like "Element" and individual
   side input parameters. It is a backwards-compatible migration path.
 - At this point a runner need not construct complex objects unless the
   user requests it.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f03b8e77
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f03b8e77
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f03b8e77

Branch: refs/heads/master
Commit: f03b8e77f200e7a0ff8f148631808e46e62cfbfc
Parents: 8bfa085
Author: Kenneth Knowles 
Authored: Fri Nov 11 14:55:16 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 14 20:44:22 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 102 +++
 .../beam/runners/core/SplittableParDo.java  |  24 ++-
 .../org/apache/beam/sdk/transforms/DoFn.java|  20 +++
 .../beam/sdk/transforms/DoFnAdapters.java   |  28 ++-
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 177 +++
 .../reflect/ByteBuddyOnTimerInvokerFactory.java |  14 +-
 .../sdk/transforms/reflect/DoFnInvoker.java |   4 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   7 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |  52 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |  61 ---
 .../transforms/reflect/DoFnInvokersTest.java|  73 
 .../DoFnSignaturesProcessElementTest.java   |  22 ---
 .../DoFnSignaturesSplittableDoFnTest.java   |   2 +-
 .../transforms/reflect/DoFnSignaturesTest.java  |  25 ++-
 .../transforms/DoFnInvokersBenchmark.java   |   2 +-
 15 files changed, 363 insertions(+), 250 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f03b8e77/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index 3b784d1..1550303 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -34,8 +34,10 @@ import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.Aggregator.AggregatorFactory;
 import org.apache.beam.sdk.transforms.Combine.CombineFn;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.Context;
 import org.apache.beam.sdk.transforms.DoFn.InputProvider;
 import org.apache.beam.sdk.transforms.DoFn.OutputReceiver;
+import org.apache.beam.sdk.transforms.DoFn.ProcessContext;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
@@ -136,16 +138,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner.ProcessContext processContext = 
createProcessContext(elem);
-
-// Note that if the element must be exploded into all its windows, that 
has to be done outside
-// of this runner.
-final DoFn.ExtraContextFactory extraContextFactory =
-createExtraContextFactory(elem);
+DoFnProcessContext processContext = 
createProcessContext(elem);
 
 // This can contain user code. Wrap it in case it throws an exception.
 try {
-  invoker.invokeProcessElement(processContext, extraContextFactory);
+  invoker.invokeProcessElement(processContext);
 } catch (Exception ex) {
   throw wrapUserCodeException(ex);
 }
@@ -163,15 +160,10 @@ public class SimpleDoFnRunner implements 
DoFnRunner.ProcessContext 
createProcessContext(WindowedValue elem) {

[2/2] incubator-beam git commit: This closes #1355

[kenn]

This closes #1355


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/13ad8f68
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/13ad8f68
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/13ad8f68

Branch: refs/heads/master
Commit: 13ad8f68abbded45f2a026d7b3338202cc3ede68
Parents: 0117a28 f03b8e7
Author: Kenneth Knowles 
Authored: Mon Nov 14 22:10:23 2016 -0800
Committer: Kenneth Knowles 
Committed: Mon Nov 14 22:10:23 2016 -0800

--
 .../beam/runners/core/SimpleDoFnRunner.java | 102 +++
 .../beam/runners/core/SplittableParDo.java  |  24 ++-
 .../org/apache/beam/sdk/transforms/DoFn.java|  20 +++
 .../beam/sdk/transforms/DoFnAdapters.java   |  28 ++-
 .../reflect/ByteBuddyDoFnInvokerFactory.java| 177 +++
 .../reflect/ByteBuddyOnTimerInvokerFactory.java |  14 +-
 .../sdk/transforms/reflect/DoFnInvoker.java |   4 +-
 .../sdk/transforms/reflect/DoFnInvokers.java|   7 +-
 .../sdk/transforms/reflect/DoFnSignature.java   |  52 +-
 .../sdk/transforms/reflect/DoFnSignatures.java  |  61 ---
 .../transforms/reflect/DoFnInvokersTest.java|  73 
 .../DoFnSignaturesProcessElementTest.java   |  22 ---
 .../DoFnSignaturesSplittableDoFnTest.java   |   2 +-
 .../transforms/reflect/DoFnSignaturesTest.java  |  25 ++-
 .../transforms/DoFnInvokersBenchmark.java   |   2 +-
 15 files changed, 363 insertions(+), 250 deletions(-)
--

[GitHub] incubator-beam pull request #1355: [BEAM-498] Treat ProcessContext and Conte...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1355


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #86: Adjust main page header to look like v...

[francesperry]

GitHub user francesperry opened a pull request:

https://github.com/apache/incubator-beam-site/pull/86

Adjust main page header to look like v2's, only with pulldowns

Starting the process of merging in the new v2/ landing page by porting the 
header to the new style.

Still todo: center beam image, center menus, add top orange bar.

Reviewer: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/francesperry/incubator-beam-site v2-trademarks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/86.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #86


commit 811231739a5bdaf099af9d6d2d784db064d40e5a
Author: Frances Perry 
Date:   2016-11-15T05:26:43Z

Adjust main header to look like v2's, only with pulldowns

TODO: center beam image, center menus, add top orange bar.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #85: Structure

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/85


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] incubator-beam-site git commit: Regenerate html after #85

[frances]

Regenerate html after #85


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/9f62b6e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/9f62b6e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/9f62b6e5

Branch: refs/heads/asf-site
Commit: 9f62b6e53c7b694518b637cdeb69629978264a98
Parents: fe3b381
Author: Frances Perry 
Authored: Mon Nov 14 19:01:05 2016 -0800
Committer: Frances Perry 
Committed: Mon Nov 14 19:01:05 2016 -0800

--
 .../2016/03/17/capability-matrix.html   |   2 +-
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../sdk/2016/02/25/python-sdk-now-public.html   |   2 +-
 .../beam/release/2016/06/15/first-release.html  |   2 +-
 .../10/11/strata-hadoop-world-and-beam.html |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   2 +-
 .../blog/2016/05/18/splitAtFraction-method.html |   2 +-
 .../05/27/where-is-my-pcollection-dot-map.html  |   2 +-
 .../06/13/flink-batch-runner-milestone.html |   2 +-
 content/blog/2016/08/03/six-months.html |   2 +-
 content/blog/2016/10/20/test-stream.html|   2 +-
 content/blog/index.html |   2 +-
 content/coming-soon.html|   2 +-
 .../contribute/contribution-guide/index.html|   2 +-
 content/contribute/design-principles/index.html |   2 +-
 content/contribute/index.html   |   2 +-
 content/contribute/logos/index.html |   2 +-
 .../presentation-materials/index.html   |   2 +-
 content/contribute/release-guide/index.html |   2 +-
 content/contribute/source-repository/index.html |   2 +-
 content/contribute/team/index.html  |   2 +-
 content/contribute/testing/index.html   |   2 +-
 content/contribute/work-in-progress/index.html  |   2 +-
 content/documentation/index.html|   4 +-
 .../pipelines/create-your-pipeline/index.html   |   2 +-
 .../pipelines/design-your-pipeline/index.html   |   2 +-
 .../pipelines/test-your-pipeline/index.html |   2 +-
 .../documentation/programming-guide/index.html  |   2 +-
 content/documentation/resources/index.html  |   2 +-
 content/documentation/runners/apex/index.html   |   2 +-
 .../runners/capability-matrix/index.html|   2 +-
 .../documentation/runners/dataflow/index.html   |   2 +-
 content/documentation/runners/direct/index.html |   2 +-
 content/documentation/runners/flink/index.html  |   2 +-
 content/documentation/runners/spark/index.html  |   2 +-
 content/documentation/sdks/java/index.html  |   2 +-
 content/documentation/sdks/python/index.html| 171 +++
 content/get-started/beam-overview/index.html|   2 +-
 content/get-started/downloads/index.html|  56 -
 content/get-started/index.html  |   2 +-
 .../mobile-gaming-example/index.html|   2 +-
 content/get-started/quickstart/index.html   |   2 +-
 content/get-started/releases/index.html | 219 +--
 content/get-started/support/index.html  |   2 +-
 .../get-started/wordcount-example/index.html|   2 +-
 content/index.html  |   2 +-
 content/privacy_policy/index.html   |   2 +-
 content/releases/index.html |   8 +-
 content/use/releases/index.html |   8 +-
 49 files changed, 282 insertions(+), 270 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/9f62b6e5/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index 405c68b..4cfb68a 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -59,7 +59,6 @@
   
   Resources
   Downloads
-  Release Notes
   Support
  

@@ -80,6 +79,7 @@
  SDKs
  Java 
SDK
  Java SDK 
API Reference
+Python SDK
  
  Runners
  Capability Matrix

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/9f62b6e5/content/beam/capability/2016/04/03/presentation-materials.html
--
diff --git a/content/beam/capability/2016/04/03/presentation-materials.html 
b/content/beam/capability/2016/04/03/presentation-materials.html
index 766c6d3..22dfbc2 100644
--- 

[2/4] incubator-beam-site git commit: Rename /get-started/releases to get-started/downloads. Delete old get-started/downloads page.

[frances]

Rename /get-started/releases to get-started/downloads. Delete old 
get-started/downloads page.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/fe3b3810
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/fe3b3810
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/fe3b3810

Branch: refs/heads/asf-site
Commit: fe3b38100bcd6aa9801af289cf5aad89791a4cd4
Parents: c505247
Author: Hadar Hod 
Authored: Mon Nov 14 17:20:40 2016 -0800
Committer: Frances Perry 
Committed: Mon Nov 14 18:55:13 2016 -0800

--
 src/_includes/header.html|  1 -
 src/get-started/downloads.md | 50 +++
 src/get-started/releases.md  | 50 ---
 3 files changed, 46 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/fe3b3810/src/_includes/header.html
--
diff --git a/src/_includes/header.html b/src/_includes/header.html
index 465fc42..6df7bbe 100644
--- a/src/_includes/header.html
+++ b/src/_includes/header.html
@@ -25,7 +25,6 @@
   
   Resources
   Downloads
-  Release 
Notes
   Support
  


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/fe3b3810/src/get-started/downloads.md
--
diff --git a/src/get-started/downloads.md b/src/get-started/downloads.md
index 95bbffc..0015d77 100644
--- a/src/get-started/downloads.md
+++ b/src/get-started/downloads.md
@@ -1,9 +1,51 @@
 ---
 layout: default
-title: "Downloads"
-permalink: /get-started/downloads/
+title: "Beam Releases"
+permalink: get-started/downloads/
+redirect_from:
+  - /get-started/releases/
+  - /use/releases/
+  - /releases/
 ---
 
-# Downloads
+# Apache Beam Downloads
 
-> **Note:** There is an open JIRA issue to fill in this page 
([BEAM-906](https://issues.apache.org/jira/browse/BEAM-906)).
+The easiest way to use Apache Beam is via one of the released versions in the
+[Maven Central 
Repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22).
+
+For example, if you are developing using Maven and want to use the SDK for
+Java with the `DirectRunner`, add the following dependencies to your
+`pom.xml` file:
+
+
+  org.apache.beam
+  beam-sdks-java-core
+  0.3.0-incubating
+
+
+  org.apache.beam
+  beam-runners-direct-java
+  0.3.0-incubating
+  runtime
+
+
+Additionally, you may want to depend on additional SDK modules, such as IO
+connectors or other extensions, and additional runners to execute your pipeline
+at scale.
+
+## Release Notes
+
+### 0.3.0-incubating
+[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.3.0-incubating/apache-beam-0.3.0-incubating-source-release.zip=download)
+
+* Release notes are available [in 
JIRA](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12338051).
+
+### 0.2.0-incubating
+[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.2.0-incubating/apache-beam-0.2.0-incubating-source-release.zip=download)
+
+* Release notes are available [in 
JIRA](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527=12335766).
+
+### 0.1.0-incubating
+[Source code 
download](https://www.apache.org/dyn/closer.cgi?filename=incubator/beam/0.1.0-incubating/apache-beam-0.1.0-incubating-source-release.zip=download)
+
+* The first incubating release of Apache Beam.

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/fe3b3810/src/get-started/releases.md
--
diff --git a/src/get-started/releases.md b/src/get-started/releases.md
deleted file mode 100644
index 4241b0d..000
--- a/src/get-started/releases.md
+++ /dev/null
@@ -1,50 +0,0 @@

-layout: default
-title: "Beam Releases"
-permalink: get-started/releases/
-redirect_from:
-  - /use/releases/
-  - /releases/

-
-# Apache Beam Releases
-
-The easiest way to use Apache Beam is via one of the released versions in the
-[Maven Central 
Repository](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.beam%22).
-
-For example, if you are developing using Maven and want to use the SDK for
-Java with the `DirectRunner`, add the following dependencies to your
-`pom.xml` file:
-
-
-  org.apache.beam
-  beam-sdks-java-core
-  0.3.0-incubating
-
-
-  org.apache.beam
-  beam-runners-direct-java
-  0.3.0-incubating
-  runtime
-
-
-Additionally, you may want to 

[4/4] incubator-beam-site git commit: This closes #85

[frances]

This closes #85


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/3b93d1b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/3b93d1b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/3b93d1b5

Branch: refs/heads/asf-site
Commit: 3b93d1b5655a16088571b59d678412854d245bbb
Parents: 160a5d3 9f62b6e
Author: Frances Perry 
Authored: Mon Nov 14 19:02:04 2016 -0800
Committer: Frances Perry 
Committed: Mon Nov 14 19:02:04 2016 -0800

--
 .../2016/03/17/capability-matrix.html   |   2 +-
 .../2016/04/03/presentation-materials.html  |   2 +-
 .../sdk/2016/02/25/python-sdk-now-public.html   |   2 +-
 .../beam/release/2016/06/15/first-release.html  |   2 +-
 .../10/11/strata-hadoop-world-and-beam.html |   2 +-
 .../website/2016/02/22/beam-has-a-logo.html |   2 +-
 .../blog/2016/05/18/splitAtFraction-method.html |   2 +-
 .../05/27/where-is-my-pcollection-dot-map.html  |   2 +-
 .../06/13/flink-batch-runner-milestone.html |   2 +-
 content/blog/2016/08/03/six-months.html |   2 +-
 content/blog/2016/10/20/test-stream.html|   2 +-
 content/blog/index.html |   2 +-
 content/coming-soon.html|   2 +-
 .../contribute/contribution-guide/index.html|   2 +-
 content/contribute/design-principles/index.html |   2 +-
 content/contribute/index.html   |   2 +-
 content/contribute/logos/index.html |   2 +-
 .../presentation-materials/index.html   |   2 +-
 content/contribute/release-guide/index.html |   2 +-
 content/contribute/source-repository/index.html |   2 +-
 content/contribute/team/index.html  |   2 +-
 content/contribute/testing/index.html   |   2 +-
 content/contribute/work-in-progress/index.html  |   2 +-
 content/documentation/index.html|   4 +-
 .../pipelines/create-your-pipeline/index.html   |   2 +-
 .../pipelines/design-your-pipeline/index.html   |   2 +-
 .../pipelines/test-your-pipeline/index.html |   2 +-
 .../documentation/programming-guide/index.html  |   2 +-
 content/documentation/resources/index.html  |   2 +-
 content/documentation/runners/apex/index.html   |   2 +-
 .../runners/capability-matrix/index.html|   2 +-
 .../documentation/runners/dataflow/index.html   |   2 +-
 content/documentation/runners/direct/index.html |   2 +-
 content/documentation/runners/flink/index.html  |   2 +-
 content/documentation/runners/spark/index.html  |   2 +-
 content/documentation/sdks/java/index.html  |   2 +-
 content/documentation/sdks/python/index.html| 171 +++
 content/get-started/beam-overview/index.html|   2 +-
 content/get-started/downloads/index.html|  56 -
 content/get-started/index.html  |   2 +-
 .../mobile-gaming-example/index.html|   2 +-
 content/get-started/quickstart/index.html   |   2 +-
 content/get-started/releases/index.html | 219 +--
 content/get-started/support/index.html  |   2 +-
 .../get-started/wordcount-example/index.html|   2 +-
 content/index.html  |   2 +-
 content/privacy_policy/index.html   |   2 +-
 content/releases/index.html |   8 +-
 content/use/releases/index.html |   8 +-
 src/_includes/header.html   |   2 +-
 src/documentation/index.md  |   2 +-
 src/documentation/sdks/python.md|  12 +
 src/get-started/downloads.md|  50 -
 src/get-started/releases.md |  50 -
 54 files changed, 342 insertions(+), 326 deletions(-)
--

[1/4] incubator-beam-site git commit: Add Python SDK page and add it to /documentation/index.md

[frances]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 160a5d330 -> 3b93d1b56


Add Python SDK page and add it to /documentation/index.md


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/c505247e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/c505247e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/c505247e

Branch: refs/heads/asf-site
Commit: c505247eec633acfbc693e508cfb7a6d267d1356
Parents: 160a5d3
Author: Hadar Hod 
Authored: Mon Nov 14 17:12:35 2016 -0800
Committer: Frances Perry 
Committed: Mon Nov 14 18:55:05 2016 -0800

--
 src/_includes/header.html|  1 +
 src/documentation/index.md   |  2 +-
 src/documentation/sdks/python.md | 12 
 3 files changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c505247e/src/_includes/header.html
--
diff --git a/src/_includes/header.html b/src/_includes/header.html
index e39e9d1..465fc42 100644
--- a/src/_includes/header.html
+++ b/src/_includes/header.html
@@ -46,6 +46,7 @@
  SDKs
  Java SDK
  Java SDK API Reference
+Python 
SDK
  
  Runners
  Capability Matrix

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c505247e/src/documentation/index.md
--
diff --git a/src/documentation/index.md b/src/documentation/index.md
index c4bbd83..87ba7b8 100644
--- a/src/documentation/index.md
+++ b/src/documentation/index.md
@@ -23,7 +23,7 @@ Learn about the Beam Programming Model and the concepts 
common to all Beam SDKs
 Find status and reference information on all of the available Beam SDKs.
 
 * [Java SDK]({{ site.baseurl }}/documentation/sdks/java/) 
-* _[Under Development]_ [Python SDK]({{ site.baseurl 
}}/contribute/work-in-progress/#feature-branches)
+* _[Under Development]_ [Python SDK]({{ site.baseurl 
}}/documentation/sdks/python/)
 
 ## Runners
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/c505247e/src/documentation/sdks/python.md
--
diff --git a/src/documentation/sdks/python.md b/src/documentation/sdks/python.md
new file mode 100644
index 000..41dbea7
--- /dev/null
+++ b/src/documentation/sdks/python.md
@@ -0,0 +1,12 @@
+---
+layout: default
+title: "Beam Python SDK"
+permalink: /documentation/sdks/python/
+---
+# Apache Beam Python SDK _[Under Development]_
+
+This page is under construction 
([BEAM-977](https://issues.apache.org/jira/browse/BEAM-977)).
+
+The Beam Python SDK is currently under development on a feature branch. Would 
you like to contribute? See the Beam [Work in Progress]({{ site.baseurl 
}}/contribute/work-in-progress/#feature-branches) page to find out how you can 
help!
+
+Get started with the [Beam Programming Guide]({{ site.baseurl 
}}/learn/programming-guide) to learn the basic concepts that hold for all SDKs 
in the Beam Model.

[jira] [Commented] (BEAM-978) Support bulk get file sizes in GcsUtil

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665801#comment-15665801
 ] 

ASF GitHub Bot commented on BEAM-978:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1359

[BEAM-978] Support bulk get file size in GcsUtil.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bulk-size

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1359.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1359


commit 259f4ac3d709637cd140baf7b30ab8632b04fee3
Author: Pei He 
Date:   2016-11-11T05:53:57Z

[BEAM-978] Support bulk get file size in GcsUtil.




> Support bulk get file sizes in GcsUtil
> --
>
> Key: BEAM-978
> URL: https://issues.apache.org/jira/browse/BEAM-978
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> Bulk get-file-sizes could be useful in FileBasedSource (It currently does 
> sampling and total size estimation with one RPC request per file.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1359: [BEAM-978] Support bulk get file size in ...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1359

[BEAM-978] Support bulk get file size in GcsUtil.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bulk-size

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1359.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1359


commit 259f4ac3d709637cd140baf7b30ab8632b04fee3
Author: Pei He 
Date:   2016-11-11T05:53:57Z

[BEAM-978] Support bulk get file size in GcsUtil.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-978) Support bulk get file sizes in GcsUtil

[Pei He (JIRA)]

Pei He created BEAM-978:
---

 Summary: Support bulk get file sizes in GcsUtil
 Key: BEAM-978
 URL: https://issues.apache.org/jira/browse/BEAM-978
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-gcp
Reporter: Pei He
Assignee: Pei He


Bulk get-file-sizes could be useful in FileBasedSource (It currently does 
sampling and total size estimation with one RPC request per file.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #83: Landing page

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/83


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d8cdb81a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d8cdb81a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d8cdb81a

Branch: refs/heads/asf-site
Commit: d8cdb81af56f2b5ba3b8411aa6951ca2688c880f
Parents: 748fe82
Author: Davor Bonaci 
Authored: Mon Nov 14 18:23:16 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 18:23:16 2016 -0800

--
 content/v2/css/beam.css |   6 +--
 content/v2/images/beam-logo.png | Bin 3382 -> 0 bytes
 content/v2/index.html   |  78 ---
 3 files changed, 29 insertions(+), 55 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/d8cdb81a/content/v2/css/beam.css
--
diff --git a/content/v2/css/beam.css b/content/v2/css/beam.css
index 7d318e3..11ef08f 100644
--- a/content/v2/css/beam.css
+++ b/content/v2/css/beam.css
@@ -245,9 +245,9 @@ code{font-size:14px}body{background-color:#f1f1f1}
 .header--navbar .navbar-inner .navbar-link.nav-search.js-highlight 
.navbar-search--icon{display:none}
 .header--navbar .navbar-inner .navbar-link.nav-search.js-highlight 
.search-input-close{display:inline-block;color:#eee;padding:4px 24px 
3px;cursor:pointer}.header--navbar 
.search-dropdown--container{position:absolute;z-index:998;margin-top:-90px;background-color:#fff;width:100%;border-bottom:1px
 solid #ddd;transition:margin .25s;-webkit-transition:margin 
.25s;-moz-transition:margin .25s;-o-transition:margin 
.25s;-ms-transition:margin .25s}.header--navbar 
.search-dropdown--container.no-animation{transition:none;-webkit-transition:none;-moz-transition:none;-o-transition:none;-ms-transition:none}.header--navbar
 .search-dropdown--container .form-search{margin:0}.header--navbar 
.search-dropdown--container .form-search 
.search-form--form{background:rgba(0,0,0,0);border:0;box-shadow:none;color:#34302D;font-size:21px;line-height:29px;height:30px;padding:25px
 0;width:93%}.header--navbar .search-dropdown--container .form-search 
.search-form--form::-webkit-input-placeholder{font-style:italic}.
 header--navbar .search-dropdown--container .form-search 
.search-form--form:-moz-placeholder{font-style:italic}.header--navbar 
.search-dropdown--container .form-search 
.search-form--form::-moz-placeholder{font-style:italic}.header--navbar 
.search-dropdown--container .form-search 
.search-form--form:-ms-input-placeholder{font-style:italic}.header--navbar 
.search-dropdown--container .form-search 
.search-form--submit{background-color:transparent;border:0;float:right;padding:28px
 16px 24px 0}.header--navbar .search-dropdown--container .form-search 
.icon-search{font-size:22px;color:#34302d}.header--navbar 
.search-dropdown--container.js-show{margin-top:0}
 
-a.beam-logo{background:url(../images/beam-logo.png) -1px -1px no-repeat}
+a.beam-logo{background:url(../../images/logos/full-color/name-right/beam-logo-full-color-name-right.svg)
 -1px -1px no-repeat}
 
-a.beam-logo 
span{display:block;width:160px;height:46px;background:url(../images/beam-logo.png)
 -1px -48px no-repeat;opacity:0;-moz-transition:opacity .12s 
ease-in-out;-webkit-transition:opacity .12s ease-in-out;-o-transition:opacity 
.12s ease-in-out}
+a.beam-logo 
span{display:block;width:160px;height:46px;background:url(../../images/logos/full-color/name-right/beam-logo-full-color-name-right.svg)
 -1px -48px no-repeat;opacity:0;-moz-transition:opacity .12s 
ease-in-out;-webkit-transition:opacity .12s ease-in-out;-o-transition:opacity 
.12s ease-in-out}
 .right-pane-widget--container{border:1px solid #ccc;box-shadow:0 -4px 0 
#34302d;margin:4px 0 40px}.right-pane-widget--container 
li{display:block;padding:12px 6.2%;line-height:14px;border-top:1px solid 
#fff}.right-pane-widget--container li 
[class^=icon-]{font-size:16px}.right-pane-widget--container li 
a{font-size:14px;line-height:14px;color:#5fa134}.right-pane-widget--container 
li:first-child{border-bottom:0}.right-pane-widget--container.secondary-nav li 
a{text-transform:uppercase;color:#888;font-family:Montserrat,sans-serif;font-weight:400;text-decoration:none}.right-pane-widget--container.secondary-nav
 li a:hover{color:#5fa134}.right-pane-widget--container.secondary-nav li.active 
a{font-weight:700;color:#34302d}.right-pane-widget--container.with-icon 
li{padding:12px 6.2% 12px 3%}.right-pane-widget--container.with-icon li 
a{margin-left:2%}.right-pane-widget--container.no-top-border{box-shadow:none}.header--content{background-color:#dedede;padding:60px
 0}.header--content .header--con
 tent-subtitle--link{color:#8cc63f}.header--content 

[GitHub] incubator-beam pull request #1043: Datastore Connector prototype using googl...

[vikkyrk]

Github user vikkyrk closed the pull request at:

https://github.com/apache/incubator-beam/pull/1043


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #80: Add Jenkins job definitions for existi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/80


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam-site git commit: This closes #80

[davor]

This closes #80


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/8ba89346
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/8ba89346
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/8ba89346

Branch: refs/heads/asf-site
Commit: 8ba89346db60c59499bfddedd7234c63ff8ff842
Parents: 3ec3a49 3596eef
Author: Davor Bonaci 
Authored: Mon Nov 14 18:09:00 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 18:09:00 2016 -0800

--
 tools/common_job_properties.groovy | 106 
 tools/job_precommit_stage.groovy   |  62 +++
 tools/job_precommit_test.groovy|  47 ++
 tools/job_seed.groovy  |  32 ++
 4 files changed, 247 insertions(+)
--

[1/2] incubator-beam-site git commit: Add Jenkins job definitions for existing projects using Jenkins DSL

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 3ec3a49ef -> 8ba89346d


Add Jenkins job definitions for existing projects using Jenkins DSL


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/3596eefa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/3596eefa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/3596eefa

Branch: refs/heads/asf-site
Commit: 3596eefaf149ce85f38cfa364c4c3363b1691ff1
Parents: 3ec3a49
Author: Davor Bonaci 
Authored: Sun Nov 13 01:49:37 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:56:10 2016 -0800

--
 tools/common_job_properties.groovy | 106 
 tools/job_precommit_stage.groovy   |  62 +++
 tools/job_precommit_test.groovy|  47 ++
 tools/job_seed.groovy  |  32 ++
 4 files changed, 247 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/3596eefa/tools/common_job_properties.groovy
--
diff --git a/tools/common_job_properties.groovy 
b/tools/common_job_properties.groovy
new file mode 100644
index 000..6e30023
--- /dev/null
+++ b/tools/common_job_properties.groovy
@@ -0,0 +1,106 @@
+// Contains functions that help build Jenkins projects. Functions typically set
+// common properties that are shared among all Jenkins projects.
+class common_job_properties {
+
+  // Sets common top-level job properties.
+  static def setTopLevelJobProperties(def context) {
+
+// GitHub project.
+context.properties {
+  githubProjectUrl('https://github.com/apache/incubator-beam-site/')
+}
+
+// Restrict this project to run only on Jenkins executors dedicated to the
+// Apache Beam project.
+context.label('beam')
+
+// Discard old builds. Build records are only kept up to this number of 
days.
+context.logRotator {
+  daysToKeep(14)
+}
+
+// Source code management.
+context.scm {
+  git {
+remote {
+  url('https://github.com/apache/incubator-beam-site.git')
+  refspec('+refs/heads/*:refs/remotes/origin/* ' +
+  '+refs/pull/*:refs/remotes/origin/pr/*')
+}
+branch('${sha1}')
+extensions {
+  cleanAfterCheckout()
+}
+  }
+}
+
+context.parameters {
+  // This is a recommended setup if you want to run the job manually. The
+  // ${sha1} parameter needs to be provided, and defaults to the main 
branch.
+  stringParam(
+  'sha1',
+  'asf-site',
+  'Commit id or refname (eg: origin/pr/9/head) you want to build.')
+}
+
+context.wrappers {
+  // Abort the build if it's stuck for more minutes than specified.
+  timeout {
+absolute(30)
+abortBuild()
+  }
+}
+  }
+
+  // Sets the pull request build trigger.
+  static def setPullRequestBuildTrigger(def context,
+def commitStatusContext,
+def successComment = '--none--') {
+context.triggers {
+  githubPullRequest {
+admins(['asfbot'])
+useGitHubHooks()
+orgWhitelist(['apache'])
+allowMembersOfWhitelistedOrgsAsAdmin()
+permitAll()
+
+extensions {
+  commitStatus {
+// This is the name that will show up in the GitHub pull request UI
+// for this Jenkins project.
+delegate.context(commitStatusContext)
+  }
+
+  /*
+This section is disabled, because of jenkinsci/ghprb-plugin#417 
issue.
+For the time being, an equivalent configure section below is added.
+
+  // Comment messages after build completes.
+  buildStatus {
+completedStatus('SUCCESS', successComment)
+completedStatus('FAILURE', '--none--')
+completedStatus('ERROR', '--none--')
+  }
+  */
+}
+  }
+}
+
+// Comment messages after build completes.
+context.configure {
+  def messages = it / triggers / 
'org.jenkinsci.plugins.ghprb.GhprbTrigger' / extensions / 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildStatus' / messages
+  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
+message(successComment)
+result('SUCCESS')
+  }
+  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
+message('--none--')
+result('ERROR')
+  }
+  messages << 
'org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage' {
+

[jira] [Closed] (BEAM-954) FileBasedSink: remove unused code of TemporaryFileRetention

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He closed BEAM-954.
---
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

> FileBasedSink: remove unused code of TemporaryFileRetention
> ---
>
> Key: BEAM-954
> URL: https://issues.apache.org/jira/browse/BEAM-954
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core, sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.4.0-incubating
>
>
> TemporaryFileRetention option is not used.
> Removing it will simplify the redesign of IOChannelFactory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam-site pull request #85: Structure

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/incubator-beam-site/pull/85

Structure

- [x] Adds a new page for the Python SDK
- [x] Adds a link to the new Python SDK page from the 'Using the Docs' page
- [x] Renames 'releases' page to 'downloads', deletes the old 'downloads' 
page


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site structure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/85.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #85






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #84: Use jekyll site.release_latest variabl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/84


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam-site git commit: Use jekyll site.release_latest variable in direct runner sample pom.xml

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site d5b722e70 -> 3ec3a49ef


Use jekyll site.release_latest variable in direct runner sample pom.xml


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5a8e2a01
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5a8e2a01
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5a8e2a01

Branch: refs/heads/asf-site
Commit: 5a8e2a01ee3aefb85bf700d4d46d20712d5c8c7c
Parents: d5b722e
Author: melissa 
Authored: Mon Nov 14 17:03:53 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:11:33 2016 -0800

--
 src/documentation/runners/direct.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5a8e2a01/src/documentation/runners/direct.md
--
diff --git a/src/documentation/runners/direct.md 
b/src/documentation/runners/direct.md
index 1d7470d..d01608e 100644
--- a/src/documentation/runners/direct.md
+++ b/src/documentation/runners/direct.md
@@ -28,7 +28,7 @@ You must specify your dependency on the Direct Runner.
 
org.apache.beam
beam-runners-direct-java
-   0.3.0-incubating
+   {{ site.release_latest }}
runtime
 
 ```

[2/2] incubator-beam-site git commit: This closes #84

[davor]

This closes #84


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/3ec3a49e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/3ec3a49e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/3ec3a49e

Branch: refs/heads/asf-site
Commit: 3ec3a49ef9358d027764bfd73f2cfa020e667d6d
Parents: d5b722e 5a8e2a0
Author: Davor Bonaci 
Authored: Mon Nov 14 17:13:32 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:13:32 2016 -0800

--
 src/documentation/runners/direct.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[jira] [Commented] (BEAM-508) Fill in the documentation/runners/dataflow portion of the website

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665622#comment-15665622
 ] 

ASF GitHub Bot commented on BEAM-508:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/77


> Fill in the documentation/runners/dataflow portion of the website
> -
>
> Key: BEAM-508
> URL: https://issues.apache.org/jira/browse/BEAM-508
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Melissa Pashniak
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for Dataflow-runner-specific content



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/832d2abe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/832d2abe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/832d2abe

Branch: refs/heads/asf-site
Commit: 832d2abe2aa43c5c0a14366dae574f01f57f4f0d
Parents: 5fbc7b7
Author: Davor Bonaci 
Authored: Mon Nov 14 17:10:54 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:10:54 2016 -0800

--
 .../documentation/runners/dataflow/index.html   | 128 ++-
 1 file changed, 126 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/832d2abe/content/documentation/runners/dataflow/index.html
--
diff --git a/content/documentation/runners/dataflow/index.html 
b/content/documentation/runners/dataflow/index.html
index aa403ec..507be0b 100644
--- a/content/documentation/runners/dataflow/index.html
+++ b/content/documentation/runners/dataflow/index.html
@@ -140,9 +140,133 @@
 
 
   
-Using the Cloud Dataflow 
Runner
+Using the Google Cloud 
Dataflow Runner
 
-This page is under construction (https://issues.apache.org/jira/browse/BEAM-508;>BEAM-508).
+The Google Cloud Dataflow Runner uses the https://cloud.google.com/dataflow/service/dataflow-service-desc;>Cloud 
Dataflow managed service. When you run your pipeline with the Cloud 
Dataflow service, the runner uploads your executable code and dependencies to a 
Google Cloud Storage bucket and creates a Cloud Dataflow job, which executes 
your pipeline on managed resources in Google Cloud Platform.
+
+The Cloud Dataflow Runner and service are suitable for large scale, 
continuous jobs, and provide:
+
+
+  a fully managed service
+  https://cloud.google.com/dataflow/service/dataflow-service-desc#autoscaling;>autoscaling
 of the number of workers throughout the lifetime of the job
+  https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow;>dynamic
 work rebalancing
+
+
+The Beam Capability 
Matrix documents the supported capabilities of the Cloud Dataflow 
Runner.
+
+Cloud Dataflow Runner 
prerequisites and setup
+To use the Cloud Dataflow Runner, you must complete the following setup:
+
+
+  
+Select or create a Google Cloud Platform Console project.
+  
+  
+Enable billing for your project.
+  
+  
+Enable required Google Cloud APIs: Cloud Dataflow, Compute Engine, 
Stackdriver Logging, Cloud Storage, and Cloud Storage JSON. You may need to 
enable additional APIs (such as BigQuery, Cloud Pub/Sub, or Cloud Datastore) if 
you use them in your pipeline code.
+  
+  
+Install the Google Cloud SDK.
+  
+  
+Create a Cloud Storage bucket.
+
+  In the Google Cloud Platform Console, go to the Cloud Storage 
browser.
+  Click Create bucket.
+  In the Create bucket dialog, specify the following 
attributes:
+
+  Name: A unique bucket name. Do not include sensitive 
information in the bucket name, as the bucket namespace is global and publicly 
visible.
+  Storage class: Multi-Regional
+  Location:  Choose your desired location
+
+  
+  Click Create.
+
+  
+
+
+For more information, see the Before you begin section of the https://cloud.google.com/dataflow/docs/quickstarts;>Cloud Dataflow 
quickstarts.
+
+Specify your dependency
+
+You must specify your dependency on the Cloud Dataflow Runner.
+
+dependency
+  groupIdorg.apache.beam/groupId
+  artifactIdbeam-runners-google-cloud-dataflow-java/artifactId
+  version0.3.0-incubating/version
+  scoperuntime/scope
+/dependency
+
+
+
+Authentication
+
+Before running your pipeline, you must authenticate with the Google Cloud 
Platform. Run the following command to get https://developers.google.com/identity/protocols/application-default-credentials;>Application
 Default Credentials.
+
+gcloud auth 
application-default login
+
+
+
+Pipeline options for 
the Cloud Dataflow Runner
+
+When executing your pipeline with the Cloud Dataflow Runner, set these 
pipeline options.
+
+
+
+  Field
+  Description
+  Default Value
+
+
+  runner
+  The pipeline runner to use. This option allows you to determine the 
pipeline runner at runtime.
+  Set to dataflow to run on the Cloud Dataflow Service.
+
+
+  project
+  The project ID for your Google Cloud Project.
+  If not set, defaults to the default project in the current environment. 
The default project is set via gcloud.
+
+
+  streaming
+  Whether streaming mode is enabled or disabled; true if 
enabled. Set to true if running pipelines with unbounded 
PCollections.
+  

[1/3] incubator-beam-site git commit: [BEAM-508] Fill in the documentation/runners/dataflow portion of the website

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site a82a0f3bb -> d5b722e70


[BEAM-508] Fill in the documentation/runners/dataflow portion of the website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5fbc7b76
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5fbc7b76
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5fbc7b76

Branch: refs/heads/asf-site
Commit: 5fbc7b764d224b58d71ef43c52dff438fe1ddc6d
Parents: a82a0f3
Author: melissa 
Authored: Fri Nov 11 10:57:28 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:10:18 2016 -0800

--
 src/documentation/runners/dataflow.md | 113 -
 1 file changed, 111 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5fbc7b76/src/documentation/runners/dataflow.md
--
diff --git a/src/documentation/runners/dataflow.md 
b/src/documentation/runners/dataflow.md
index c49223b..57dec4c 100644
--- a/src/documentation/runners/dataflow.md
+++ b/src/documentation/runners/dataflow.md
@@ -4,6 +4,115 @@ title: "Cloud Dataflow Runner"
 permalink: /documentation/runners/dataflow/
 redirect_from: /learn/runners/dataflow/
 ---
-# Using the Cloud Dataflow Runner
+# Using the Google Cloud Dataflow Runner
 
-This page is under construction 
([BEAM-508](https://issues.apache.org/jira/browse/BEAM-508)).
+The Google Cloud Dataflow Runner uses the [Cloud Dataflow managed 
service](https://cloud.google.com/dataflow/service/dataflow-service-desc). When 
you run your pipeline with the Cloud Dataflow service, the runner uploads your 
executable code and dependencies to a Google Cloud Storage bucket and creates a 
Cloud Dataflow job, which executes your pipeline on managed resources in Google 
Cloud Platform.
+
+The Cloud Dataflow Runner and service are suitable for large scale, continuous 
jobs, and provide:
+
+* a fully managed service
+* 
[autoscaling](https://cloud.google.com/dataflow/service/dataflow-service-desc#autoscaling)
 of the number of workers throughout the lifetime of the job
+* [dynamic work 
rebalancing](https://cloud.google.com/blog/big-data/2016/05/no-shard-left-behind-dynamic-work-rebalancing-in-google-cloud-dataflow)
+
+The [Beam Capability Matrix]({{ site.baseurl 
}}/documentation/runners/capability-matrix/) documents the supported 
capabilities of the Cloud Dataflow Runner.
+
+## Cloud Dataflow Runner prerequisites and setup
+To use the Cloud Dataflow Runner, you must complete the following setup:
+
+1. Select or create a Google Cloud Platform Console project.
+
+2. Enable billing for your project.
+
+3. Enable required Google Cloud APIs: Cloud Dataflow, Compute Engine, 
Stackdriver Logging, Cloud Storage, and Cloud Storage JSON. You may need to 
enable additional APIs (such as BigQuery, Cloud Pub/Sub, or Cloud Datastore) if 
you use them in your pipeline code.
+
+4. Install the Google Cloud SDK.
+
+5. Create a Cloud Storage bucket.
+* In the Google Cloud Platform Console, go to the Cloud Storage browser.
+* Click **Create bucket**.
+* In the **Create bucket** dialog, specify the following attributes:
+  * _Name_: A unique bucket name. Do not include sensitive information in 
the bucket name, as the bucket namespace is global and publicly visible.
+  * _Storage class_: Multi-Regional
+  * _Location_:  Choose your desired location
+* Click **Create**.
+
+For more information, see the *Before you begin* section of the [Cloud 
Dataflow quickstarts](https://cloud.google.com/dataflow/docs/quickstarts).
+
+### Specify your dependency
+
+You must specify your dependency on the Cloud Dataflow Runner.
+
+```java
+
+  org.apache.beam
+  beam-runners-google-cloud-dataflow-java
+  {{ site.release_latest }}
+  runtime
+
+```
+
+### Authentication
+
+Before running your pipeline, you must authenticate with the Google Cloud 
Platform. Run the following command to get [Application Default 
Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
+
+```
+gcloud auth application-default login
+```
+
+## Pipeline options for the Cloud Dataflow Runner
+
+When executing your pipeline with the Cloud Dataflow Runner, set these 
pipeline options.
+
+
+
+  Field
+  Description
+  Default Value
+
+
+  runner
+  The pipeline runner to use. This option allows you to determine the 
pipeline runner at runtime.
+  Set to dataflow to run on the Cloud Dataflow Service.
+
+
+  project
+  The project ID for your Google Cloud Project.
+  If not set, defaults to the default project in the current environment. 
The default project is set via gcloud.
+
+
+  

[GitHub] incubator-beam-site pull request #77: [BEAM-508] Fill in the documentation/r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/77


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam-site git commit: This closes #77

[davor]

This closes #77


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/d5b722e7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/d5b722e7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/d5b722e7

Branch: refs/heads/asf-site
Commit: d5b722e70a3777d5666ddafa5618ac129370aea8
Parents: a82a0f3 832d2ab
Author: Davor Bonaci 
Authored: Mon Nov 14 17:10:54 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 17:10:54 2016 -0800

--
 .../documentation/runners/dataflow/index.html   | 128 ++-
 src/documentation/runners/dataflow.md   | 113 +++-
 2 files changed, 237 insertions(+), 4 deletions(-)
--

[GitHub] incubator-beam-site pull request #84: Use jekyll site.release_latest variabl...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/incubator-beam-site/pull/84

Use jekyll site.release_latest variable in direct runner sample pom.xml



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site directpom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/84.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #84


commit 055ec81af3539409cf747e5a67f5285757caabf2
Author: melissa 
Date:   2016-11-15T01:03:53Z

Use jekyll site.release_latest variable in direct runner sample pom.xml




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-977) Fill in Python SDK page

[Hadar Hod (JIRA)]

Hadar Hod created BEAM-977:
--

 Summary: Fill in Python SDK page
 Key: BEAM-977
 URL: https://issues.apache.org/jira/browse/BEAM-977
 Project: Beam
  Issue Type: Task
  Components: website
Reporter: Hadar Hod






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #143

[Apache Jenkins Server]

See 

[GitHub] incubator-beam-site pull request #76: [BEAM-505] Fill in the documentation/r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/76


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-505) Fill in the documentation/runners/direct portion of the website

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665535#comment-15665535
 ] 

ASF GitHub Bot commented on BEAM-505:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/76


> Fill in the documentation/runners/direct portion of the website
> ---
>
> Key: BEAM-505
> URL: https://issues.apache.org/jira/browse/BEAM-505
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Frances Perry
>Assignee: Melissa Pashniak
>
> As per 
> https://docs.google.com/document/d/1-0jMv7NnYp0Ttt4voulUMwVe_qjBYeNMLm2LusYF3gQ/edit.
> Should be a landing page for the Direct runner



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam-site git commit: [BEAM-505] Fill in the documentation/runners/direct portion of the website

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 6ab73c79a -> a82a0f3bb


[BEAM-505] Fill in the documentation/runners/direct portion of the website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/fe87fb80
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/fe87fb80
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/fe87fb80

Branch: refs/heads/asf-site
Commit: fe87fb807a310fe8e68298d4fe6fde86d7c65522
Parents: 6ab73c7
Author: melissa 
Authored: Fri Nov 11 10:44:13 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 16:34:51 2016 -0800

--
 src/documentation/runners/direct.md | 40 ++--
 1 file changed, 38 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/fe87fb80/src/documentation/runners/direct.md
--
diff --git a/src/documentation/runners/direct.md 
b/src/documentation/runners/direct.md
index 094d44e..1d7470d 100644
--- a/src/documentation/runners/direct.md
+++ b/src/documentation/runners/direct.md
@@ -1,9 +1,45 @@
 ---
 layout: default
-title: "Apache Direct Runner"
+title: "Direct Runner"
 permalink: /documentation/runners/direct/
 redirect_from: /learn/runners/direct/
 ---
 # Using the Direct Runner
 
-This page is under construction 
([BEAM-505](https://issues.apache.org/jira/browse/BEAM-505)).
+The Direct Runner executes pipelines on your machine and is designed to 
validate that pipelines adhere to the Apache Beam model as closely as possible. 
Instead of focusing on efficient pipeline execution, the Direct Runner performs 
additional checks to ensure that users do not rely on semantics that are not 
guaranteed by the model. Some of these checks include:
+
+* enforcing immutability of elements
+* enforcing encodability of elements
+* elements are processed in an arbitrary order at all points
+* serialization of user functions (`DoFn`, `CombineFn`, etc.)
+
+Using the Direct Runner for testing and development helps ensure that 
pipelines are robust across different Beam runners. In addition, debugging 
failed runs can be a non-trivial task when a pipeline executes on a remote 
cluster. Instead, it is often faster and simpler to perform local unit testing 
on your pipeline code. Unit testing your pipeline locally also allows you to 
use your preferred local debugging tools.
+
+Here are some resources with information about how to test your pipelines.
+* [Testing Unbounded Pipelines in Apache Beam]({{ site.baseurl 
}}/blog/2016/10/20/test-stream.html) talks about the use of Java classes 
[`PAssert`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html) and 
[`TestStream`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest }}/index.html?org/apache/beam/sdk/testing/TestStream.html) 
to test your pipelines.
+* The [Apache Beam WordCount Example]({{ site.baseurl 
}}/get-started/wordcount-example/) contains an example of logging and testing a 
pipeline with [`PAssert`]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest }}/index.html?org/apache/beam/sdk/testing/PAssert.html).
+
+
+## Direct Runner prerequisites and setup
+
+You must specify your dependency on the Direct Runner.
+
+```java
+
+   org.apache.beam
+   beam-runners-direct-java
+   0.3.0-incubating
+   runtime
+
+```
+
+## Pipeline options for the Direct Runner
+
+When executing your pipeline from the command-line, set `runner` to `direct`. 
The default values for the other pipeline options are generally sufficient.
+
+See the reference documentation for the  [`DirectOptions`]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/runners/direct/DirectOptions.html)[`PipelineOptions`](https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py)
 interface (and its subinterfaces) for defaults and the complete list of 
pipeline configuration options.
+
+## Additional information and caveats
+
+Local execution is limited by the memory available in your local environment. 
It is highly recommended that you run your pipeline with data sets small enough 
to fit in local memory. You can create a small in-memory data set using a [`Create`]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/transforms/Create.html)[`Create`](https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/transforms/core.py)
 transform, or you can use a [`Read`]({{ 
site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest 

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/159ff482
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/159ff482
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/159ff482

Branch: refs/heads/asf-site
Commit: 159ff4821ebfb63a93899b41000bb8d00fcfb978
Parents: fe87fb8
Author: Davor Bonaci 
Authored: Mon Nov 14 16:35:14 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 16:35:14 2016 -0800

--
 content/documentation/runners/direct/index.html | 43 +++-
 1 file changed, 41 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/159ff482/content/documentation/runners/direct/index.html
--
diff --git a/content/documentation/runners/direct/index.html 
b/content/documentation/runners/direct/index.html
index d47b51b..6cc31f9 100644
--- a/content/documentation/runners/direct/index.html
+++ b/content/documentation/runners/direct/index.html
@@ -6,7 +6,7 @@
   
   
 
-  Apache Direct Runner
+  Direct Runner
   
 
@@ -142,7 +142,46 @@
   
 Using the Direct Runner
 
-This page is under construction (https://issues.apache.org/jira/browse/BEAM-505;>BEAM-505).
+The Direct Runner executes pipelines on your machine and is designed to 
validate that pipelines adhere to the Apache Beam model as closely as possible. 
Instead of focusing on efficient pipeline execution, the Direct Runner performs 
additional checks to ensure that users do not rely on semantics that are not 
guaranteed by the model. Some of these checks include:
+
+
+  enforcing immutability of elements
+  enforcing encodability of elements
+  elements are processed in an arbitrary order at all points
+  serialization of user functions (DoFn, CombineFn, etc.)
+
+
+Using the Direct Runner for testing and development helps ensure that 
pipelines are robust across different Beam runners. In addition, debugging 
failed runs can be a non-trivial task when a pipeline executes on a remote 
cluster. Instead, it is often faster and simpler to perform local unit testing 
on your pipeline code. Unit testing your pipeline locally also allows you to 
use your preferred local debugging tools.
+
+Here are some resources with information about how to test your 
pipelines.
+
+  Testing Unbounded Pipelines 
in Apache Beam talks about the use of Java classes PAssert and TestStream to test your pipelines.
+  The Apache Beam WordCount 
Example contains an example of logging and testing a pipeline with PAssert.
+
+
+Direct Runner prerequisites and 
setup
+
+You must specify your dependency on the Direct Runner.
+
+dependency
+   groupIdorg.apache.beam/groupId
+   artifactIdbeam-runners-direct-java/artifactId
+   version0.3.0-incubating/version
+   scoperuntime/scope
+/dependency
+
+
+
+Pipeline options for the 
Direct Runner
+
+When executing your pipeline from the command-line, set runner to direct. The default values for the other 
pipeline options are generally sufficient.
+
+See the reference documentation for the  DirectOptionshttps://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/utils/options.py;>PipelineOptions interface (and its 
subinterfaces) for defaults and the complete list of pipeline configuration 
options.
+
+Additional information and 
caveats
+
+Local execution is limited by the memory available in your local 
environment. It is highly recommended that you run your pipeline with data sets 
small enough to fit in local memory. You can create a small in-memory data set 
using a Createhttps://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/transforms/core.py;>Create transform, or you can use a 
Readhttps://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/iobase.py;>Read transform to work with small local or remote 
files.
+
 
   
 

[3/3] incubator-beam-site git commit: This closes #76

[davor]

This closes #76


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/a82a0f3b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/a82a0f3b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/a82a0f3b

Branch: refs/heads/asf-site
Commit: a82a0f3bb80cbbc41eeb9fb56042fc4a33abace1
Parents: 6ab73c7 159ff48
Author: Davor Bonaci 
Authored: Mon Nov 14 16:35:14 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 16:35:14 2016 -0800

--
 content/documentation/runners/direct/index.html | 43 +++-
 src/documentation/runners/direct.md | 40 +-
 2 files changed, 79 insertions(+), 4 deletions(-)
--

[GitHub] incubator-beam pull request #1341: Simplify FileOperations interface by refa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1341


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Simplify FileOperations interface

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master ee55f6e39 -> 0117a2834


Simplify FileOperations interface

Refactor removeDirectoryAndFiles() into FileBasedSink.

FileIOChannelFactory.match() returns the empty list on an absent parent
directory instead of throwing.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1bd0f8bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1bd0f8bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1bd0f8bc

Branch: refs/heads/master
Commit: 1bd0f8bcc6213586318cb363e9ed746ae836feeb
Parents: ee55f6e
Author: Pei He 
Authored: Thu Nov 10 13:03:38 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 14 16:27:23 2016 -0800

--
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 77 +---
 .../beam/sdk/util/FileIOChannelFactory.java | 11 ++-
 .../beam/sdk/util/FileIOChannelFactoryTest.java |  8 ++
 3 files changed, 50 insertions(+), 46 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1bd0f8bc/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
index 2d058ae..f11fbee 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSink.java
@@ -22,6 +22,7 @@ import static 
com.google.common.base.Preconditions.checkNotNull;
 import static com.google.common.base.Preconditions.checkState;
 import static com.google.common.base.Strings.isNullOrEmpty;
 
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Ordering;
 
 import java.io.File;
@@ -459,7 +460,25 @@ public abstract class FileBasedSink extends Sink {
   LOG.debug("Removing temporary bundle output files in {}.", 
tempDirectory);
   FileOperations fileOperations =
   FileOperationsFactory.getFileOperations(tempDirectory, options);
-  fileOperations.removeDirectoryAndFiles(tempDirectory, knownFiles);
+  IOChannelFactory factory = IOChannelUtils.getFactory(tempDirectory);
+
+  // To partially mitigate the effects of filesystems with 
eventually-consistent
+  // directory matching APIs, we remove not only files that the filesystem 
says exist
+  // in the directory (which may be incomplete), but also files that are 
known to exist
+  // (produced by successfully completed bundles).
+  // This may still fail to remove temporary outputs of some failed 
bundles, but at least
+  // the common case (where all bundles succeed) is guaranteed to be fully 
addressed.
+  Collection matches = 
factory.match(factory.resolve(tempDirectory, "*"));
+  Set allMatches = new HashSet<>(matches);
+  allMatches.addAll(knownFiles);
+  LOG.debug(
+  "Removing {} temporary files found under {} ({} matched glob, {} 
known files)",
+  allMatches.size(),
+  tempDirectory,
+  matches.size(),
+  allMatches.size() - matches.size());
+  fileOperations.remove(allMatches);
+  fileOperations.remove(ImmutableList.of(tempDirectory));
 }
 
 /**
@@ -656,26 +675,22 @@ public abstract class FileBasedSink extends Sink {
*/
   private interface FileOperations {
 /**
- * Copy a collection of files from one location to another.
+ * Copies a collection of files from one location to another.
  *
  * The number of source filenames must equal the number of destination 
filenames.
  *
  * @param srcFilenames the source filenames.
  * @param destFilenames the destination filenames.
  */
- void copy(List srcFilenames, List destFilenames) throws 
IOException;
+void copy(List srcFilenames, List destFilenames) throws 
IOException;
 
 /**
- * Removes a directory and the files in it (but not subdirectories).
+ * Removes a collection of files or directories.
  *
- * Additionally, to partially mitigate the effects of filesystems with 
eventually-consistent
- * directory matching APIs, takes a list of files that are known to exist 
- i.e. removes the
- * union of the known files and files that the filesystem says exist in 
the directory.
- *
- * Assumes that, if directory listing had been strongly consistent, it 
would have matched
- * all of knownFiles - i.e. on a strongly consistent filesystem, 
knownFiles can be ignored.
+ * Directories are required to be empty. Non-empty directories will not 
be deleted,
+ * and this method may return silently or throw an exception.
   

[2/2] incubator-beam git commit: This closes #1341

[tgroh]

This closes #1341


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0117a283
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0117a283
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0117a283

Branch: refs/heads/master
Commit: 0117a2834c3dd6eaa202ac6a187a2634a6f33d6e
Parents: ee55f6e 1bd0f8b
Author: Thomas Groh 
Authored: Mon Nov 14 16:28:07 2016 -0800
Committer: Thomas Groh 
Committed: Mon Nov 14 16:28:07 2016 -0800

--
 .../org/apache/beam/sdk/io/FileBasedSink.java   | 77 +---
 .../beam/sdk/util/FileIOChannelFactory.java | 11 ++-
 .../beam/sdk/util/FileIOChannelFactoryTest.java |  8 ++
 3 files changed, 50 insertions(+), 46 deletions(-)
--

[1/2] incubator-beam git commit: [BEAM-852] Add validation to file based sources during create time

[robertwb]

Repository: incubator-beam
Updated Branches:
  refs/heads/python-sdk 15e78b28a -> 560fe79f8


[BEAM-852] Add validation to file based sources during create time


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/76ad2929
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/76ad2929
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/76ad2929

Branch: refs/heads/python-sdk
Commit: 76ad29296fd57e1eec97bf40d9cf3a1d54a63a3f
Parents: 15e78b2
Author: Sourabh Bajaj 
Authored: Mon Nov 14 15:40:10 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Nov 14 15:40:10 2016 -0800

--
 sdks/python/apache_beam/io/avroio.py|  8 +++-
 sdks/python/apache_beam/io/bigquery.py  |  2 +-
 sdks/python/apache_beam/io/filebasedsource.py   | 16 +++-
 .../apache_beam/io/filebasedsource_test.py  | 41 ++--
 sdks/python/apache_beam/io/textio.py| 13 +--
 5 files changed, 60 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/76ad2929/sdks/python/apache_beam/io/avroio.py
--
diff --git a/sdks/python/apache_beam/io/avroio.py 
b/sdks/python/apache_beam/io/avroio.py
index 53ed95a..e7e73dd 100644
--- a/sdks/python/apache_beam/io/avroio.py
+++ b/sdks/python/apache_beam/io/avroio.py
@@ -37,7 +37,7 @@ __all__ = ['ReadFromAvro', 'WriteToAvro']
 class ReadFromAvro(PTransform):
   """A ``PTransform`` for reading avro files."""
 
-  def __init__(self, file_pattern=None, min_bundle_size=0):
+  def __init__(self, file_pattern=None, min_bundle_size=0, validate=True):
 """Initializes ``ReadFromAvro``.
 
 Uses source '_AvroSource' to read a set of Avro files defined by a given
@@ -70,13 +70,17 @@ class ReadFromAvro(PTransform):
   file_pattern: the set of files to be read.
   min_bundle_size: the minimum size in bytes, to be considered when
splitting the input into bundles.
+  validate: flag to verify that the files exist during the pipeline
+creation time.
   **kwargs: Additional keyword arguments to be passed to the base class.
 """
 super(ReadFromAvro, self).__init__()
 self._args = (file_pattern, min_bundle_size)
+self._validate = validate
 
   def apply(self, pvalue):
-return pvalue.pipeline | Read(_AvroSource(*self._args))
+return pvalue.pipeline | Read(_AvroSource(*self._args,
+  validate=self._validate))
 
 
 class _AvroUtils(object):

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/76ad2929/sdks/python/apache_beam/io/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/bigquery.py 
b/sdks/python/apache_beam/io/bigquery.py
index f0e88a6..8d7892a 100644
--- a/sdks/python/apache_beam/io/bigquery.py
+++ b/sdks/python/apache_beam/io/bigquery.py
@@ -65,7 +65,7 @@ input entails querying the table for all its rows. The coder 
argument on
 BigQuerySource controls the reading of the lines in the export files (i.e.,
 transform a JSON object into a PCollection element). The coder is not involved
 when the same table is read as a side input since there is no intermediate
-format involved.  We get the table rows directly from the BigQuery service with
+format involved. We get the table rows directly from the BigQuery service with
 a query.
 
 Users may provide a query to read from rather than reading all of a BigQuery

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/76ad2929/sdks/python/apache_beam/io/filebasedsource.py
--
diff --git a/sdks/python/apache_beam/io/filebasedsource.py 
b/sdks/python/apache_beam/io/filebasedsource.py
index 58ad118..c7bc27e 100644
--- a/sdks/python/apache_beam/io/filebasedsource.py
+++ b/sdks/python/apache_beam/io/filebasedsource.py
@@ -50,7 +50,8 @@ class FileBasedSource(iobase.BoundedSource):
file_pattern,
min_bundle_size=0,
compression_type=fileio.CompressionTypes.AUTO,
-   splittable=True):
+   splittable=True,
+   validate=True):
 """Initializes ``FileBasedSource``.
 
 Args:
@@ -68,10 +69,13 @@ class FileBasedSource(iobase.BoundedSource):
   the file, for example, for compressed files where currently
   it is not possible to efficiently read a data range without
   decompressing the whole file.
+  validate: Boolean flag to verify that the files exist during the pipeline
+creation time.
 Raises:
   TypeError: when 

[2/2] incubator-beam git commit: Closes #1220

[robertwb]

Closes #1220


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/560fe79f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/560fe79f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/560fe79f

Branch: refs/heads/python-sdk
Commit: 560fe79f875b9d08b7256e6bf653a48b19f0ccb5
Parents: 15e78b2 76ad292
Author: Robert Bradshaw 
Authored: Mon Nov 14 15:41:00 2016 -0800
Committer: Robert Bradshaw 
Committed: Mon Nov 14 15:41:00 2016 -0800

--
 sdks/python/apache_beam/io/avroio.py|  8 +++-
 sdks/python/apache_beam/io/bigquery.py  |  2 +-
 sdks/python/apache_beam/io/filebasedsource.py   | 16 +++-
 .../apache_beam/io/filebasedsource_test.py  | 41 ++--
 sdks/python/apache_beam/io/textio.py| 13 +--
 5 files changed, 60 insertions(+), 20 deletions(-)
--

[jira] [Updated] (BEAM-210) Allow control of empty ON_TIME panes analogous to final panes

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-210:
-
Assignee: Thomas Groh  (was: Kenneth Knowles)

> Allow control of empty ON_TIME panes analogous to final panes
> -
>
> Key: BEAM-210
> URL: https://issues.apache.org/jira/browse/BEAM-210
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Mark Shields
>Assignee: Thomas Groh
>
> Today, ON_TIME panes are emitted whether or not they are empty. We had 
> decided that for final panes the user would want to control this behavior, to 
> control data volume. But for ON_TIME panes no such control exists. The 
> rationale is perhaps that the ON_TIME pane is a fundamental result that 
> should not be elided. To be considered: whether this is what we want.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-976) Update examples README.md to fix instructions to run pipelines

[Vikas Kedigehalli (JIRA)]

Vikas Kedigehalli created BEAM-976:
--

 Summary: Update examples README.md to fix instructions to run 
pipelines
 Key: BEAM-976
 URL: https://issues.apache.org/jira/browse/BEAM-976
 Project: Beam
  Issue Type: Task
  Components: examples-java
Reporter: Vikas Kedigehalli
Assignee: Kenneth Knowles
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1815

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-39) Audit trigger implementations to ensure they prefetch appropriately

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-39?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-39:

Assignee: Sam Whittle  (was: Kenneth Knowles)

> Audit trigger implementations to ensure they prefetch appropriately
> ---
>
> Key: BEAM-39
> URL: https://issues.apache.org/jira/browse/BEAM-39
> Project: Beam
>  Issue Type: Task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Sam Whittle
>  Labels: Triggers
>
> Each method of the implementation of a trigger as an 
> onElement/onFire/shouldFire state machine has a corresponding method to 
> register any state it intends to read, so that state can be read in a single 
> batch per bundle.
> This task is to read them over ensuring prefetching is adequate, and 
> potentially create a test harness to do this generically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #142

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_FlinkLocal #834

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-965) Source Transformations Don't Set Correct Output Type in Flink Streaming Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665021#comment-15665021
 ] 

ASF GitHub Bot commented on BEAM-965:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1352


> Source Transformations Don't Set Correct Output Type in Flink Streaming Runner
> --
>
> Key: BEAM-965
> URL: https://issues.apache.org/jira/browse/BEAM-965
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-965) Source Transformations Don't Set Correct Output Type in Flink Streaming Runner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-965.
-
Resolution: Fixed

> Source Transformations Don't Set Correct Output Type in Flink Streaming Runner
> --
>
> Key: BEAM-965
> URL: https://issues.apache.org/jira/browse/BEAM-965
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 0.4.0-incubating
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1352: [BEAM-965] Set Correct Output Type on Sou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1352


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-965] Set Correct Output Type on Sources in Flink Stream Runner

[aljoscha]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8bfa08519 -> ee55f6e39


[BEAM-965] Set Correct Output Type on Sources in Flink Stream Runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a61832a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a61832a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a61832a8

Branch: refs/heads/master
Commit: a61832a83c15e0aa5e3cc1985fc32aa1afb348e6
Parents: 8bfa085
Author: Aljoscha Krettek 
Authored: Fri Nov 11 23:58:16 2016 +0100
Committer: Aljoscha Krettek 
Committed: Mon Nov 14 22:07:45 2016 +0100

--
 .../FlinkStreamingTransformTranslators.java| 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a61832a8/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index a3e8a49..687e9c8 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
@@ -219,6 +219,9 @@ public class FlinkStreamingTransformTranslators {
 FlinkStreamingTranslationContext context) {
   PCollection output = context.getOutput(transform);
 
+  TypeInformation outputTypeInfo =
+  context.getTypeInfo(context.getOutput(transform));
+
   DataStream source;
   if (transform.getSource().getClass().equals(UnboundedFlinkSource.class)) 
{
 @SuppressWarnings("unchecked")
@@ -246,7 +249,7 @@ public class FlinkStreamingTransformTranslators {
 new Instant(flinkAssigner.extractTimestamp(s, -1)),
 GlobalWindow.INSTANCE,
 PaneInfo.NO_FIRING));
-  }});
+  }}).returns(outputTypeInfo);
   } else {
 try {
   UnboundedSourceWrapper sourceWrapper =
@@ -256,7 +259,7 @@ public class FlinkStreamingTransformTranslators {
   context.getExecutionEnvironment().getParallelism());
   source = context
   .getExecutionEnvironment()
-  .addSource(sourceWrapper).name(transform.getName());
+  
.addSource(sourceWrapper).name(transform.getName()).returns(outputTypeInfo);
 } catch (Exception e) {
   throw new RuntimeException(
   "Error while translating UnboundedSource: " + 
transform.getSource(), e);
@@ -276,6 +279,10 @@ public class FlinkStreamingTransformTranslators {
 FlinkStreamingTranslationContext context) {
   PCollection output = context.getOutput(transform);
 
+  TypeInformation outputTypeInfo =
+  context.getTypeInfo(context.getOutput(transform));
+
+
   DataStream source;
   try {
 BoundedSourceWrapper sourceWrapper =
@@ -285,7 +292,7 @@ public class FlinkStreamingTransformTranslators {
 context.getExecutionEnvironment().getParallelism());
 source = context
 .getExecutionEnvironment()
-.addSource(sourceWrapper).name(transform.getName());
+
.addSource(sourceWrapper).name(transform.getName()).returns(outputTypeInfo);
   } catch (Exception e) {
 throw new RuntimeException(
 "Error while translating BoundedSource: " + transform.getSource(), 
e);

[2/2] incubator-beam git commit: This closes #1352

[aljoscha]

This closes #1352


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ee55f6e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ee55f6e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ee55f6e3

Branch: refs/heads/master
Commit: ee55f6e39ff429cd406835c789ce89c4b01fb4c4
Parents: 8bfa085 a61832a
Author: Aljoscha Krettek 
Authored: Mon Nov 14 22:07:53 2016 +0100
Committer: Aljoscha Krettek 
Committed: Mon Nov 14 22:07:53 2016 +0100

--
 .../FlinkStreamingTransformTranslators.java| 13 ++---
 1 file changed, 10 insertions(+), 3 deletions(-)
--

[GitHub] incubator-beam-site pull request #83: Landing page

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/incubator-beam-site/pull/83

Landing page

- [x] Update the prose on the landing page
- [ ] Copy Blog & Twitter sections from /src/index.md
- [ ] Update content in the slider News section (perhaps with Blog & 
Twitter) OR delete this section completely 
- [ ] Add drop-downs to header sections (Get Started, Documentation, and 
Contribution)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site landing-page

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit bd797dd1810fe448bf89c6e481d60a2b11fcfd28
Author: Hadar Hod 
Date:   2016-11-14T20:50:11Z

Update content on v2 landing page

commit 0ee847c9f020e73e3bdc8e17d81381a22f4daf93
Author: Hadar Hod 
Date:   2016-11-14T21:01:59Z

Fix broken links in header




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reza Nouri updated BEAM-975:

Description: 
It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
REST service that reads data from Mongo. After a number of requests, mongoIO 
throws the following exception:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at 
com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at 
com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at 
com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
at 
com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
at 
org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I suppose there must be a problem with Mongo connection which causes this issue.

  was:
It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
REST service that reads data from Mongo. After a number of requests, mongoIO 
throws the following exception:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at 
com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at 
com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at 
com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
at 

[jira] [Created] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

Reza Nouri created BEAM-975:
---

 Summary: Issue with MongoDBIO
 Key: BEAM-975
 URL: https://issues.apache.org/jira/browse/BEAM-975
 Project: Beam
  Issue Type: Bug
Reporter: Reza Nouri


It appears that there is an issue with MongoDBIO. I am Apache Beam in a REST 
service that reads from Mongo. After a number of requests, mongoIO throws the 
following exception:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at 
com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at 
com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at 
com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
at 
com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
at 
org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I think there must be problem with Mongo connection which causes this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reza Nouri updated BEAM-975:

Description: 
It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
REST service that reads data from Mongo. After a number of requests, mongoIO 
throws the following exception:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at 
com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at 
com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at 
com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
at 
com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
at 
com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
at 
com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
at 
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
at 
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
at com.mongodb.Mongo.execute(Mongo.java:772)
at com.mongodb.Mongo$2.execute(Mongo.java:759)
at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
at 
org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
at 
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
at 
org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
at 
org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I think there must be problem with Mongo connection which causes this issue.

  was:
It appears that there is an issue with MongoDBIO. I am Apache Beam in a REST 
service that reads from Mongo. After a number of requests, mongoIO throws the 
following exception:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
at 
com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
at 
com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
at 
com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
at 
com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
at 
com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
at 
com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
at 
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
at 
com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
at 
com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
at 

[jira] [Commented] (BEAM-974) Adds attributes support to PubsubIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664877#comment-15664877
 ] 

ASF GitHub Bot commented on BEAM-974:
-

GitHub user reuvenlax opened a pull request:

https://github.com/apache/incubator-beam/pull/1358

[BEAM-974] Add PubSub attributes support to PubsubIO.

Introduces a new PubsubMessage class, that allows separate management of 
payload and attributes from PubSub. Allows the user to register functions on 
PubsubIO to parse  and format these objects in order to interact with 
attributes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam pubsub_attributes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1358


commit a0bbd6882f703dec41faac4c647ee51f680d5a4f
Author: Reuven Lax 
Date:   2016-11-14T04:29:31Z

Add PubSub attributes support to PubsubIO.




> Adds attributes support to PubsubIO
> ---
>
> Key: BEAM-974
> URL: https://issues.apache.org/jira/browse/BEAM-974
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: 0.3.0-incubating
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 0.4.0-incubating
>
>
> Adds support for Pubsub attributes by introducing a new PubsubMessage class 
> that allows manipulation of payload and attributes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1358: [BEAM-974] Add PubSub attributes support ...

[reuvenlax]

GitHub user reuvenlax opened a pull request:

https://github.com/apache/incubator-beam/pull/1358

[BEAM-974] Add PubSub attributes support to PubsubIO.

Introduces a new PubsubMessage class, that allows separate management of 
payload and attributes from PubSub. Allows the user to register functions on 
PubsubIO to parse  and format these objects in order to interact with 
attributes.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/reuvenlax/incubator-beam pubsub_attributes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1358.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1358


commit a0bbd6882f703dec41faac4c647ee51f680d5a4f
Author: Reuven Lax 
Date:   2016-11-14T04:29:31Z

Add PubSub attributes support to PubsubIO.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1814

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_RunnableOnService_SparkLocal #141

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_FlinkLocal #833

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-641) Need to test the generated archetypes projects

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664581#comment-15664581
 ] 

Davor Bonaci commented on BEAM-641:
---

We should double-check this -- I believe this is already bound properly to the 
"install" phase of the top-level build.

> Need to test the generated archetypes projects
> --
>
> Key: BEAM-641
> URL: https://issues.apache.org/jira/browse/BEAM-641
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Critical
> Fix For: 0.4.0-incubating
>
>
> Travis and Jenkins pre-submits don't test building the generated archetypes 
> projects.
> Currently, changes to archetypes have to be manually verified by:
> mvn archetype:generate \
> -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
> -DarchetypeGroupId=org.apache.beam \
> -DarchetypeVersion=0.3.0-incubating-SNAPSHOT \
> -DgroupId=com.example \
> -DartifactId=first-beam \
> -Dversion="0.3.0-incubating-SNAPSHOT" \
> -DinteractiveMode=false \
> -Dpackage=org.apache.beam.examples
> and did "mvn clean install" in first-beam project.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/3] incubator-beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/de57d1ae
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/de57d1ae
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/de57d1ae

Branch: refs/heads/asf-site
Commit: de57d1aefe42049a1ab07fc26f25632118bfddf0
Parents: 5411eed
Author: Davor Bonaci 
Authored: Mon Nov 14 09:33:04 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 09:33:04 2016 -0800

--
 content/blog/2016/10/20/test-stream.html | 2 +-
 content/feed.xml | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/de57d1ae/content/blog/2016/10/20/test-stream.html
--
diff --git a/content/blog/2016/10/20/test-stream.html 
b/content/blog/2016/10/20/test-stream.html
index 58fc9d4..abc4563 100644
--- a/content/blog/2016/10/20/test-stream.html
+++ b/content/blog/2016/10/20/test-stream.html
@@ -220,7 +220,7 @@ reactions to speculative and late panes and dropped 
data.
 
 Element Timings
 
-Elements arrive either behind, with, or after the watermark, which 
categorizes
+Elements arrive either before, with, or after the watermark, which 
categorizes
 them into the “early”, “on-time”, and “late” divisions. “Late” 
elements can be
 further subdivided into “unobservably”, “observably”, and 
“droppably” late,
 depending on the window to which they are assigned and the maximum allowed

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/de57d1ae/content/feed.xml
--
diff --git a/content/feed.xml b/content/feed.xml
index 1e64366..a91a76e 100644
--- a/content/feed.xml
+++ b/content/feed.xml
@@ -80,7 +80,7 @@ reactions to speculative and late panes and dropped 
data./p
 
 h2 id=element-timingsElement Timings/h2
 
-pElements arrive either behind, with, or after the watermark, which 
categorizes
+pElements arrive either before, with, or after the watermark, which 
categorizes
 them into the “early”, “on-time”, and “late” divisions. “Late” 
elements can be
 further subdivided into “unobservably”, “observably”, and 
“droppably” late,
 depending on the window to which they are assigned and the maximum allowed

[3/3] incubator-beam-site git commit: This closes #82

[davor]

This closes #82


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/6ab73c79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/6ab73c79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/6ab73c79

Branch: refs/heads/asf-site
Commit: 6ab73c79a9482d1b4be70bf9f8b0d93c8f6a78de
Parents: 2bd1944 de57d1a
Author: Davor Bonaci 
Authored: Mon Nov 14 09:33:04 2016 -0800
Committer: Davor Bonaci 
Committed: Mon Nov 14 09:33:04 2016 -0800

--
 content/blog/2016/10/20/test-stream.html | 2 +-
 content/feed.xml | 2 +-
 src/_posts/2016-10-20-test-stream.md | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
--

[GitHub] incubator-beam-site pull request #82: "before" instead of "behind" for "earl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/82


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam-site git commit: "before" instead of "behind" for "early" elements

[davor]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 2bd194414 -> 6ab73c79a


"before" instead of "behind" for "early" elements

Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/5411eed1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/5411eed1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/5411eed1

Branch: refs/heads/asf-site
Commit: 5411eed1658b638cf427e416fb20dd1b224bcbb5
Parents: 2bd1944
Author: ManuZhang 
Authored: Mon Nov 14 21:12:34 2016 +0800
Committer: GitHub 
Committed: Mon Nov 14 21:12:34 2016 +0800

--
 src/_posts/2016-10-20-test-stream.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/5411eed1/src/_posts/2016-10-20-test-stream.md
--
diff --git a/src/_posts/2016-10-20-test-stream.md 
b/src/_posts/2016-10-20-test-stream.md
index 797f25e..418f556 100644
--- a/src/_posts/2016-10-20-test-stream.md
+++ b/src/_posts/2016-10-20-test-stream.md
@@ -78,7 +78,7 @@ reactions to speculative and late panes and dropped data.
 
 ## Element Timings
 
-Elements arrive either behind, with, or after the watermark, which categorizes
+Elements arrive either before, with, or after the watermark, which categorizes
 them into the "early", "on-time", and "late" divisions. "Late" elements can be
 further subdivided into "unobservably", "observably", and "droppably" late,
 depending on the window to which they are assigned and the maximum allowed

[jira] [Commented] (BEAM-856) IO test should use a free port for embedded resource instance

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664286#comment-15664286
 ] 

Sergio Fernández commented on BEAM-856:
---

Cool! 
Thanks [~jbonofre].

> IO test should use a free port for embedded resource instance
> -
>
> Key: BEAM-856
> URL: https://issues.apache.org/jira/browse/BEAM-856
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 0.3.0-incubating
>Reporter: Sergio Fernández
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> {{MongoDbIOTest fails}} (addr already in use) when a MongoDB service is 
> locally running. I'd say the port should be random in the test suite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-856) IO test should use a free port for embedded resource instance

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-856.
---
   Resolution: Fixed
Fix Version/s: 0.4.0-incubating

All existing IO now use free available network port.

> IO test should use a free port for embedded resource instance
> -
>
> Key: BEAM-856
> URL: https://issues.apache.org/jira/browse/BEAM-856
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Affects Versions: 0.3.0-incubating
>Reporter: Sergio Fernández
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> {{MongoDbIOTest fails}} (addr already in use) when a MongoDB service is 
> locally running. I'd say the port should be random in the test suite.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-918) Let users set STORAGE_LEVEL via SparkPipelineOptions.

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664251#comment-15664251
 ] 

Jean-Baptiste Onofré commented on BEAM-918:
---

Waiting for merge of https://github.com/apache/incubator-beam/pull/1291 as it 
will be easier to apply.

> Let users set STORAGE_LEVEL via SparkPipelineOptions.
> -
>
> Key: BEAM-918
> URL: https://issues.apache.org/jira/browse/BEAM-918
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Jean-Baptiste Onofré
>
> Spark provides different "STORAGE_LEVEL"s for caching RDDs (disk, memory, 
> ser/de, etc.).
> The runner decides on caching when necessary, for example when a RDD is 
> accessed repeatedly.
>  
> For batch, we can let users set their preferred STORAGE_LEVEL via 
> SparkPipelineOptions.
> Note: for streaming we force a "MEMORY_ONLY", since (among other things) the 
> runner heavily relies on stateful operations, and it's natural for streaming 
> to happen in memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-927) Findbugs doesn't pass in JMS

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664200#comment-15664200
 ] 

ASF GitHub Bot commented on BEAM-927:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1357

[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1357


commit 5cd00e5bd43b47b3cb7296001344310ca6f97305
Author: Jean-Baptiste Onofré 
Date:   2016-11-14T15:27:56Z

[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO




> Findbugs doesn't pass in JMS
> 
>
> Key: BEAM-927
> URL: https://issues.apache.org/jira/browse/BEAM-927
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ 
> beam-sdks-java-io-jms ---
> [INFO] BugInstance size is 2
> [INFO] Error size is 0
> [INFO] Total bugs: 2
> [INFO] Class org.apache.beam.sdk.io.jms.JmsRecord defines non-transient 
> non-serializable instance field jmsDestination 
> [org.apache.beam.sdk.io.jms.JmsRecord] In JmsRecord.java
> [INFO] Class org.apache.beam.sdk.io.jms.JmsRecord defines non-transient 
> non-serializable instance field jmsReplyTo 
> [org.apache.beam.sdk.io.jms.JmsRecord] In JmsRecord.java
> [INFO] 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1357: [BEAM-927] Fix findbugs and re-enable Mav...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1357

[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-927

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1357.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1357


commit 5cd00e5bd43b47b3cb7296001344310ca6f97305
Author: Jean-Baptiste Onofré 
Date:   2016-11-14T15:27:56Z

[BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1356: [BEAM-930] Fix findbugs and re-enable Mav...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1356

[BEAM-930] Fix findbugs and re-enable Maven plugin in MongoDbIO

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam BEAM-930

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1356


commit d838d95d62fec12e7cd990c00bdaaedc6b7ad6ea
Author: Jean-Baptiste Onofré 
Date:   2016-11-14T15:10:53Z

[BEAM-930] Fix findbugs and re-enable Maven plugin in MongoDbIO




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1813

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_RunnableOnService_SparkLocal #140

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-930) Findbugs doesn't pass in MongoDB

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663802#comment-15663802
 ] 

Jean-Baptiste Onofré commented on BEAM-930:
---

Thanks for the report [~dhalp...@google.com].

I just launched (locally on my machine) findbugs goal and I don't see those 
errors anymore. Let me double check.

> Findbugs doesn't pass in MongoDB
> 
>
> Key: BEAM-930
> URL: https://issues.apache.org/jira/browse/BEAM-930
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> {code}
> [INFO] --- findbugs-maven-plugin:3.0.1:check (default) @ 
> beam-sdks-java-io-mongodb ---
> [INFO] BugInstance size is 2
> [INFO] Error size is 0
> [INFO] Total bugs: 2
> [INFO] Boxing/unboxing to parse a primitive 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource.getEstimatedSizeBytes(PipelineOptions)
>  [org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbSource] At 
> MongoDbIO.java:[line 227]
> [INFO] Class org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn defines 
> non-transient non-serializable instance field client 
> [org.apache.beam.sdk.io.mongodb.MongoDbIO$Write$WriteFn] In MongoDbIO.java
> [INFO]  
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_FlinkLocal #832

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[Amit Sela (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663349#comment-15663349
 ] 

Amit Sela commented on BEAM-891:


Because most tests are batch..

> Flake in Spark metrics library?
> ---
>
> Key: BEAM-891
> URL: https://issues.apache.org/jira/browse/BEAM-891
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Stas Levin
>
> [~staslev] I think you implemented this functionality originally? Want to 
> take a look? CC [~amitsela]
> Run: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_SparkLocal/org.apache.beam$beam-runners-spark/43/testReport/junit/org.apache.beam.sdk.transforms/FilterTest/testFilterGreaterThan/
> Error:
> {code}
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: 5
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:77)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:53)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
>   at 
> org.apache.beam.sdk.transforms.FilterTest.testFilterGreaterThan(FilterTest.java:122)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: 5
>   at 
> scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:43)
>   at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189)
>   at 
> scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:198)
>   at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:47)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:144)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:128)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at 
> scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:126)
>   at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:48)
>   at 
> org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:159)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.registerMetrics(SparkRuntimeContext.java:94)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.(SparkRuntimeContext.java:66)
>   at 
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:73)
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:146)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-891) Flake in Spark metrics library?

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663281#comment-15663281
 ] 

Stas Levin commented on BEAM-891:
-

Sounds good. 
Btw, why can't we just set {{options.setEnableSparkSinks(false);}} directly in 
{{TestOptionsForStreaming}}?

> Flake in Spark metrics library?
> ---
>
> Key: BEAM-891
> URL: https://issues.apache.org/jira/browse/BEAM-891
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Stas Levin
>
> [~staslev] I think you implemented this functionality originally? Want to 
> take a look? CC [~amitsela]
> Run: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_SparkLocal/org.apache.beam$beam-runners-spark/43/testReport/junit/org.apache.beam.sdk.transforms/FilterTest/testFilterGreaterThan/
> Error:
> {code}
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: 5
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:77)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:53)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
>   at 
> org.apache.beam.sdk.transforms.FilterTest.testFilterGreaterThan(FilterTest.java:122)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: 5
>   at 
> scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:43)
>   at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189)
>   at 
> scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:198)
>   at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:47)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:144)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:128)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at 
> scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:126)
>   at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:48)
>   at 
> org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:159)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.registerMetrics(SparkRuntimeContext.java:94)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.(SparkRuntimeContext.java:66)
>   at 
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:73)
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:146)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-891) Flake in Spark metrics library?

[Stas Levin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663281#comment-15663281
 ] 

Stas Levin edited comment on BEAM-891 at 11/14/16 9:29 AM:
---

Sounds good. 
Btw, why can't we just set {{options.setEnableSparkSinks(false)}} directly in 
{{TestOptionsForStreaming}}?


was (Author: staslev):
Sounds good. 
Btw, why can't we just set {{options.setEnableSparkSinks(false);}} directly in 
{{TestOptionsForStreaming}}?

> Flake in Spark metrics library?
> ---
>
> Key: BEAM-891
> URL: https://issues.apache.org/jira/browse/BEAM-891
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Daniel Halperin
>Assignee: Stas Levin
>
> [~staslev] I think you implemented this functionality originally? Want to 
> take a look? CC [~amitsela]
> Run: 
> https://builds.apache.org/job/beam_PostCommit_RunnableOnService_SparkLocal/org.apache.beam$beam-runners-spark/43/testReport/junit/org.apache.beam.sdk.transforms/FilterTest/testFilterGreaterThan/
> Error:
> {code}
> java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: 5
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:77)
>   at 
> org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:53)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:182)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:112)
>   at 
> org.apache.beam.sdk.transforms.FilterTest.testFilterGreaterThan(FilterTest.java:122)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IndexOutOfBoundsException: 5
>   at 
> scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:43)
>   at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:189)
>   at 
> scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:47)
>   at 
> scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:198)
>   at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:47)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:144)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:128)
>   at scala.collection.AbstractSeq.indexOf(Seq.scala:40)
>   at 
> scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:126)
>   at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:48)
>   at 
> org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:159)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.registerMetrics(SparkRuntimeContext.java:94)
>   at 
> org.apache.beam.runners.spark.translation.SparkRuntimeContext.(SparkRuntimeContext.java:66)
>   at 
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:73)
>   at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:146)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)