[1/2] incubator-beam git commit: Closes #1651

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4206408bf -> 5255a3381


Closes #1651


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5255a338
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5255a338
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5255a338

Branch: refs/heads/master
Commit: 5255a33812758bbb9d081962675bd0180802c82b
Parents: 4206408 5fb4f5d
Author: Dan Halperin 
Authored: Fri Dec 16 23:53:49 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 23:53:49 2016 -0800

--
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java| 23 +--
 .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 72 
 2 files changed, 63 insertions(+), 32 deletions(-)
--

[GitHub] incubator-beam pull request #1651: Forward port DataflowJavaSDK/pull/512

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1651


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-545) Pipelines and their executions naming changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756581#comment-15756581
 ] 

ASF GitHub Bot commented on BEAM-545:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1653

[BEAM-545] PipelineOptions: fix parameter name

Seems like a cut and paste error. R: @peihe

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1653


commit 65afadb15cc320acc4e1562aec0de0c82fd102bd
Author: Daniel Halperin 
Date:   2016-12-17T07:47:56Z

[BEAM-545] PipelineOptions: fix parameter name

Seems like a cut and paste error. R: @peihe




> Pipelines and their executions naming changes
> -
>
> Key: BEAM-545
> URL: https://issues.apache.org/jira/browse/BEAM-545
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> The purpose of the changes is to clarify the differences between the two, have
> consensus between runners, and unify the implementation.
> Current states:
>  * PipelineOptions.appName defaults to mainClass name
>  * DataflowPipelineOptions.jobName defaults to appName+user+datetime
>  * FlinkPipelineOptions.jobName defaults to appName+user+datetime
> Proposal:
> 1. Replace PipelineOptions.appName with PipelineOptions.pipelineName.
> *  It is the user-visible name for a specific graph.
> *  default to mainClass name.
> *  Use cases: Find all executions of a pipeline
> 2. Add jobName to top level PipelineOptions.
> *  It is the unique name for an execution
> *  defaults to pipelineName + user + datetime + random Integer
> *  Use cases:
> -- Finding all executions by USER_A between TIME_X and TIME_Y
> -- Naming resources created by the execution. for example:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1653: [BEAM-545] PipelineOptions: fix parameter...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1653

[BEAM-545] PipelineOptions: fix parameter name

Seems like a cut and paste error. R: @peihe

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1653.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1653


commit 65afadb15cc320acc4e1562aec0de0c82fd102bd
Author: Daniel Halperin 
Date:   2016-12-17T07:47:56Z

[BEAM-545] PipelineOptions: fix parameter name

Seems like a cut and paste error. R: @peihe




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1845

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1175) Separate runners-core Javadoc from SDK Javadoc, and perhaps other artifacts

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1175:
-

 Summary: Separate runners-core Javadoc from SDK Javadoc, and 
perhaps other artifacts
 Key: BEAM-1175
 URL: https://issues.apache.org/jira/browse/BEAM-1175
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Kenneth Knowles
Assignee: James Malone


Currently, the runners-core Javadoc is included with the SDK's Javadoc on the 
website.

Pipeline authors should only be viewing the SDK's Javadoc, and likely the IOs.

Generally, while I do think we can't really separate every IO Javadoc, it is 
extremely confusing to have Javadoc for separate POM dependencies included 
together, since the classes listed will not actually be available to someone 
who just depends on the SDK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1614: [BEAM-853] Force streaming execution on b...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1614


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1652: [BEAM-27, BEAM-362] Move InMemoryTimerInt...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1652

[BEAM-27, BEAM-362] Move InMemoryTimerInternals to runners-core; restore a 
deprecated copy in its place

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This is a replay of ec0bf7b4023ff75f4ec6723d2e77ed507eb57c51 with an 
additional adapter commit that should enable integration tests to succeed.

Note that this configuration of commits works will with the current 
capabilities of `git` and GitHub.  This behavior is not baked into the history, 
but is calculated dynamically by both, so it may improve over time.

Notes on what would happen if this were done in one commit:

 - `git log --follow` finds the history
 - `git diff --find-copies` finds the copy
 - `git blame --find-copies-harder` finds the copy, but `--find-copies` 
does not

I don't recall which level GitHub's UI works with, but the most important 
two are `git log` and `git blame`. To make sure it works without any of the 
above, two commits is better.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
InMemoryTimerInternals

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1652.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1652


commit 6effd6c93587d1c0a02ba9285b47d165ae5c517d
Author: Kenneth Knowles 
Date:   2016-12-16T04:45:56Z

Move InMemoryTimerInternals to runners-core

commit 557d2d724c53233ad9d34c9239ff5cf77b754d73
Author: Kenneth Knowles 
Date:   2016-12-17T04:22:59Z

Restore SDK's InMemoryTimerInternals, deprecated




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1843

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_RunnableOnService_Dataflow #1843

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #2130

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1648: [BEAM-498] Undeletes DoFnInvokers.of(OldD...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[5/5] incubator-beam git commit: This closes #1648: Undeletes DoFnInvokers.of(OldDoFn)

[kenn]

This closes #1648: Undeletes DoFnInvokers.of(OldDoFn)

  Revert "Move InMemoryTimerInternals to runners-core"
  Revert "Moves DoFnAdapters to runners-core"
  Revert "Removes ArgumentProvider.windowingInternals"
  Revert "Removes code for wrapping DoFn as an OldDoFn"


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ffe3ab3d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ffe3ab3d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ffe3ab3d

Branch: refs/heads/master
Commit: ffe3ab3d622c767871064dcdb8c5a953bbf0a1d2
Parents: 7d1976b 45ed5c7
Author: Kenneth Knowles 
Authored: Fri Dec 16 18:46:17 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 18:46:17 2016 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../apache/beam/runners/core/DoFnAdapters.java  | 344 -
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   1 +
 .../runners/core/InMemoryTimerInternals.java| 276 --
 .../beam/runners/core/SimpleDoFnRunner.java |  57 +++
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../beam/runners/core/SplittableParDo.java  |   7 +
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../core/InMemoryTimerInternalsTest.java| 155 --
 .../beam/runners/core/ReduceFnTester.java   |   1 +
 .../beam/runners/core/SplittableParDoTest.java  |  16 +-
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |   2 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 504 +++
 .../apache/beam/sdk/transforms/DoFnTester.java  |  43 ++
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 297 ++-
 .../sdk/transforms/reflect/DoFnInvoker.java |  20 +
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 +-
 .../sdk/util/state/InMemoryTimerInternals.java  | 275 ++
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |   2 +-
 .../transforms/reflect/DoFnInvokersTest.java|  42 ++
 .../util/state/InMemoryTimerInternalsTest.java  | 153 ++
 29 files changed, 1547 insertions(+), 824 deletions(-)
--

[3/5] incubator-beam git commit: Revert "Removes code for wrapping DoFn as an OldDoFn"

[kenn]

Revert "Removes code for wrapping DoFn as an OldDoFn"

This reverts commit a22de15012c51e8b7e31143021f0a298e093bf51.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a12fd8c5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a12fd8c5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a12fd8c5

Branch: refs/heads/master
Commit: a12fd8c580d3b1ea46c5be951f39046bfa0dacf3
Parents: abdbee6
Author: Eugene Kirpichov 
Authored: Fri Dec 16 15:26:28 2016 -0800
Committer: Eugene Kirpichov 
Committed: Fri Dec 16 16:39:20 2016 -0800

--
 .../apache/beam/runners/core/DoFnAdapters.java  | 150 ++
 .../org/apache/beam/sdk/transforms/OldDoFn.java | 295 ++-
 .../sdk/transforms/reflect/DoFnInvokers.java| 141 -
 .../transforms/reflect/DoFnInvokersTest.java|  36 +++
 4 files changed, 611 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a12fd8c5/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
index 0f5624f..a4002da 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
@@ -18,6 +18,8 @@
 package org.apache.beam.runners.core;
 
 import java.io.IOException;
+import java.util.Collection;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.transforms.Aggregator;
 import org.apache.beam.sdk.transforms.AggregatorRetriever;
@@ -39,6 +41,7 @@ import org.apache.beam.sdk.util.Timer;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
+import org.apache.beam.sdk.values.TypeDescriptor;
 import org.joda.time.Duration;
 import org.joda.time.Instant;
 
@@ -53,6 +56,18 @@ public class DoFnAdapters {
   /** Should not be instantiated. */
   private DoFnAdapters() {}
 
+  /**
+   * If this is an {@link OldDoFn} produced via {@link #toOldDoFn}, returns 
the class of the
+   * original {@link DoFn}, otherwise returns {@code fn.getClass()}.
+   */
+  public static Class getDoFnClass(OldDoFn fn) {
+if (fn instanceof SimpleDoFnAdapter) {
+  return ((SimpleDoFnAdapter) fn).fn.getClass();
+} else {
+  return fn.getClass();
+}
+  }
+
   /** Creates an {@link OldDoFn} that delegates to the {@link DoFn}. */
   @SuppressWarnings({"unchecked", "rawtypes"})
   public static  OldDoFn 
toOldDoFn(DoFn fn) {
@@ -64,6 +79,126 @@ public class DoFnAdapters {
 }
   }
 
+  /** Creates a {@link OldDoFn.ProcessContext} from a {@link 
DoFn.ProcessContext}. */
+  public static  OldDoFn.ProcessContext 
adaptProcessContext(
+  OldDoFn fn,
+  final DoFn.ProcessContext c,
+  final DoFnInvoker.ArgumentProvider extra) {
+return fn.new ProcessContext() {
+  @Override
+  public InputT element() {
+return c.element();
+  }
+
+  @Override
+  public  T sideInput(PCollectionView view) {
+return c.sideInput(view);
+  }
+
+  @Override
+  public Instant timestamp() {
+return c.timestamp();
+  }
+
+  @Override
+  public BoundedWindow window() {
+return extra.window();
+  }
+
+  @Override
+  public PaneInfo pane() {
+return c.pane();
+  }
+
+  @Override
+  public WindowingInternals windowingInternals() {
+return extra.windowingInternals();
+  }
+
+  @Override
+  public PipelineOptions getPipelineOptions() {
+return c.getPipelineOptions();
+  }
+
+  @Override
+  public void output(OutputT output) {
+c.output(output);
+  }
+
+  @Override
+  public void outputWithTimestamp(OutputT output, Instant timestamp) {
+c.outputWithTimestamp(output, timestamp);
+  }
+
+  @Override
+  public  void sideOutput(TupleTag tag, T output) {
+c.sideOutput(tag, output);
+  }
+
+  @Override
+  public  void sideOutputWithTimestamp(TupleTag tag, T output, 
Instant timestamp) {
+c.sideOutputWithTimestamp(tag, output, timestamp);
+  }
+
+  @Override
+  protected  Aggregator 
createAggregatorInternal(
+  String name, 

[2/5] incubator-beam git commit: Revert "Moves DoFnAdapters to runners-core"

[kenn]

Revert "Moves DoFnAdapters to runners-core"

This reverts commit 33ed3238e2b3899cff061be3056c5cc29fc60a04.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/954e57d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/954e57d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/954e57d7

Branch: refs/heads/master
Commit: 954e57d7696fd14f7d1015f4e40f025ef8538802
Parents: 4aa0ee1
Author: Eugene Kirpichov 
Authored: Fri Dec 16 15:37:02 2016 -0800
Committer: Eugene Kirpichov 
Committed: Fri Dec 16 16:39:20 2016 -0800

--
 .../apex/translation/WindowBoundTranslator.java |   2 +-
 .../operators/ApexGroupByKeyOperator.java   |   2 +-
 .../operators/ApexParDoOperator.java|   2 +-
 .../apache/beam/runners/core/DoFnAdapters.java  | 508 ---
 .../beam/runners/core/SimpleOldDoFnRunner.java  |   4 +-
 .../core/GroupAlsoByWindowsProperties.java  |   2 +-
 .../functions/FlinkDoFnFunction.java|   2 +-
 .../functions/FlinkMultiOutputDoFnFunction.java |   2 +-
 .../functions/FlinkProcessContextBase.java  |   2 +-
 .../wrappers/streaming/DoFnOperator.java|   2 +-
 .../sdk/transforms/AggregatorRetriever.java |  13 +-
 .../beam/sdk/transforms/DoFnAdapters.java   | 504 ++
 .../org/apache/beam/sdk/transforms/OldDoFn.java |   2 +-
 .../apache/beam/sdk/transforms/NoOpOldDoFn.java |   2 +-
 14 files changed, 518 insertions(+), 531 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/954e57d7/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
index ef049e1..33b9269 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/WindowBoundTranslator.java
@@ -22,8 +22,8 @@ import java.util.Collections;
 import org.apache.beam.runners.apex.ApexPipelineOptions;
 import org.apache.beam.runners.apex.translation.operators.ApexParDoOperator;
 import org.apache.beam.runners.core.AssignWindowsDoFn;
-import org.apache.beam.runners.core.DoFnAdapters;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFnAdapters;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/954e57d7/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
index 4af7ff0..48ac177 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexGroupByKeyOperator.java
@@ -413,7 +413,7 @@ public class ApexGroupByKeyOperator implements 
Operator {
 }
 
 @Override
-public  Aggregator 
createAggregatorInternal(
+protected  Aggregator 
createAggregatorInternal(
 String name, Combine.CombineFn combiner) {
   throw new UnsupportedOperationException();
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/954e57d7/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
--
diff --git 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
index 4538fb5..a3d3a97 100644
--- 
a/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
+++ 
b/runners/apex/src/main/java/org/apache/beam/runners/apex/translation/operators/ApexParDoOperator.java
@@ -38,7 +38,6 @@ import 
org.apache.beam.runners.apex.translation.utils.NoOpStepContext;
 import 

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756230#comment-15756230
 ] 

ASF GitHub Bot commented on BEAM-498:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1648


> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/5] incubator-beam git commit: Revert "Move InMemoryTimerInternals to runners-core"

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7d1976b26 -> ffe3ab3d6


Revert "Move InMemoryTimerInternals to runners-core"

This reverts commit ec0bf7b4023ff75f4ec6723d2e77ed507eb57c51.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/45ed5c70
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/45ed5c70
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/45ed5c70

Branch: refs/heads/master
Commit: 45ed5c70c18a806d0fc2e7385886285206fd18e4
Parents: 954e57d
Author: Eugene Kirpichov 
Authored: Fri Dec 16 16:33:51 2016 -0800
Committer: Eugene Kirpichov 
Committed: Fri Dec 16 16:39:20 2016 -0800

--
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   1 +
 .../runners/core/InMemoryTimerInternals.java| 276 ---
 .../core/InMemoryTimerInternalsTest.java| 155 ---
 .../beam/runners/core/ReduceFnTester.java   |   1 +
 .../beam/runners/core/SplittableParDoTest.java  |  16 +-
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |  36 +++
 .../sdk/util/state/InMemoryTimerInternals.java  | 275 ++
 .../util/state/InMemoryTimerInternalsTest.java  | 153 ++
 10 files changed, 471 insertions(+), 446 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/45ed5c70/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
index efcd771..9189191 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
@@ -27,6 +27,7 @@ import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
+import org.apache.beam.sdk.util.state.InMemoryTimerInternals;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
 import org.joda.time.Instant;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/45ed5c70/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
deleted file mode 100644
index b22fcb3..000
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++ /dev/null
@@ -1,276 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.runners.core;
-
-import static com.google.common.base.Preconditions.checkNotNull;
-import static com.google.common.base.Preconditions.checkState;
-
-import com.google.common.base.MoreObjects;
-import java.util.HashSet;
-import java.util.PriorityQueue;
-import java.util.Set;
-import javax.annotation.Nullable;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.util.TimeDomain;
-import org.apache.beam.sdk.util.TimerInternals;
-import org.apache.beam.sdk.util.WindowTracing;
-import org.apache.beam.sdk.util.state.StateNamespace;
-import org.joda.time.Instant;
-
-/**
- * Simulates the firing of timers and progression of input and output 
watermarks for a single
- * computation and key in a Windmill-like streaming environment.
- */
-public class 

[4/5] incubator-beam git commit: Revert "Removes ArgumentProvider.windowingInternals"

[kenn]

Revert "Removes ArgumentProvider.windowingInternals"

This reverts commit f3e8a0383bf9cb3f9452e0364f7deba113cadff9.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4aa0ee14
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4aa0ee14
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4aa0ee14

Branch: refs/heads/master
Commit: 4aa0ee1436a8d94f7c1c75bd0151790d14635c64
Parents: a12fd8c
Author: Eugene Kirpichov 
Authored: Fri Dec 16 15:26:32 2016 -0800
Committer: Eugene Kirpichov 
Committed: Fri Dec 16 16:39:20 2016 -0800

--
 .../apache/beam/runners/core/DoFnAdapters.java  | 14 +
 .../beam/runners/core/SimpleDoFnRunner.java | 57 
 .../beam/runners/core/SplittableParDo.java  |  7 +++
 .../apache/beam/sdk/transforms/DoFnTester.java  |  7 +++
 .../sdk/transforms/reflect/DoFnInvoker.java | 20 +++
 .../transforms/reflect/DoFnInvokersTest.java|  6 +++
 6 files changed, 111 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4aa0ee14/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
index a4002da..fc5847c 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/DoFnAdapters.java
@@ -38,6 +38,7 @@ import 
org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.Timer;
+import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.State;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
@@ -355,6 +356,14 @@ public class DoFnAdapters {
 }
 
 @Override
+public WindowingInternals windowingInternals() {
+  // The OldDoFn doesn't allow us to ask for these outside 
ProcessElements, so this
+  // should be unreachable.
+  throw new UnsupportedOperationException(
+  "Can only get WindowingInternals in processElement");
+}
+
+@Override
 public DoFn.InputProvider inputProvider() {
   throw new UnsupportedOperationException("inputProvider() exists only for 
testing");
 }
@@ -467,6 +476,11 @@ public class DoFnAdapters {
 }
 
 @Override
+public WindowingInternals windowingInternals() {
+  return context.windowingInternals();
+}
+
+@Override
 public DoFn.InputProvider inputProvider() {
   throw new UnsupportedOperationException("inputProvider() exists only for 
testing");
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/4aa0ee14/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
index d504b40..29ef3ef 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java
@@ -51,10 +51,13 @@ import 
org.apache.beam.sdk.util.ExecutionContext.StepContext;
 import org.apache.beam.sdk.util.SideInputReader;
 import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.Timer;
+import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.util.state.State;
+import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateNamespace;
 import org.apache.beam.sdk.util.state.StateNamespaces;
 import org.apache.beam.sdk.util.state.StateSpec;
@@ -416,6 +419,11 @@ public class SimpleDoFnRunner implements 
DoFnRunner windowingInternals() {
+  throw new UnsupportedOperationException("WindowingInternals are 
unsupported.");
+}
+
+@Override
 public  RestrictionTracker 
restrictionTracker() {
   throw new UnsupportedOperationException(
   "Cannot access 

[jira] [Issue Comment Deleted] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reza Nouri updated BEAM-975:

Comment: was deleted

(was: Retry cannot be a solution since this issue kills mongo process. )

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756214#comment-15756214
 ] 

Reza Nouri commented on BEAM-975:
-

Retry cannot be a solution since this issue kills mongo process. 

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-975) Issue with MongoDBIO

[Reza Nouri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756213#comment-15756213
 ] 

Reza Nouri commented on BEAM-975:
-

Retry cannot be a solution since this issue kills mongo process. 

> Issue with MongoDBIO
> 
>
> Key: BEAM-975
> URL: https://issues.apache.org/jira/browse/BEAM-975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Reza Nouri
>Assignee: Jean-Baptiste Onofré
>
> It appears that there is an issue with MongoDBIO. I am using Apache Beam in a 
> REST service that reads data from Mongo. After a number of requests, mongoIO 
> throws the following exception:
> com.mongodb.MongoSocketReadException: Prematurely reached end of stream
>   at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491)
>   at 
> com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221)
>   at 
> com.mongodb.connection.CommandHelper.receiveReply(CommandHelper.java:134)
>   at 
> com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:121)
>   at 
> com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:83)
>   at 
> com.mongodb.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:43)
>   at 
> com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115)
>   at 
> com.mongodb.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:46)
>   at 
> com.mongodb.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:381)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:96)
>   at 
> com.mongodb.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:82)
>   at 
> com.mongodb.connection.DefaultServer.getConnection(DefaultServer.java:72)
>   at 
> com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:86)
>   at 
> com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:237)
>   at 
> com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:482)
>   at com.mongodb.operation.FindOperation.execute(FindOperation.java:79)
>   at com.mongodb.Mongo.execute(Mongo.java:772)
>   at com.mongodb.Mongo$2.execute(Mongo.java:759)
>   at com.mongodb.OperationIterable.iterator(OperationIterable.java:47)
>   at com.mongodb.FindIterableImpl.iterator(FindIterableImpl.java:143)
>   at 
> org.apache.beam.sdk.io.mongodb.MongoDbIO$BoundedMongoDbReader.start(MongoDbIO.java:359)
>   at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:99)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:154)
>   at 
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:121)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> I suppose there must be a problem with Mongo connection which causes this 
> issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1842

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1842

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1651: Forward port DataflowJavaSDK/pull/512

[sammcveety]

GitHub user sammcveety opened a pull request:

https://github.com/apache/incubator-beam/pull/1651

Forward port DataflowJavaSDK/pull/512

R: @dhalperi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sammcveety/incubator-beam sgmc/fwd_512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1651.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1651


commit 0ca434f1c043635e18156a06ddef823f91290718
Author: Sam McVeety 
Date:   2016-12-17T02:10:28Z

Forward port DataflowJavaSDK/pull/512




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1841

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1841

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-475) High-quality javadoc for Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15756032#comment-15756032
 ] 

ASF GitHub Bot commented on BEAM-475:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1650


> High-quality javadoc for Beam
> -
>
> Key: BEAM-475
> URL: https://issues.apache.org/jira/browse/BEAM-475
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> We should have good Javadoc for Beam!
> Current snapshot: http://beam.incubator.apache.org/javadoc/0.1.0-incubating/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1650: [BEAM-475] View.asMap: minor javadoc fixe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1650

[dhalperi]

Closes #1650


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7d1976b2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7d1976b2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7d1976b2

Branch: refs/heads/master
Commit: 7d1976b2628e0d560df57610b8ed8a6b8443fb7b
Parents: abdbee6 6a4a699
Author: Dan Halperin 
Authored: Fri Dec 16 17:41:51 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 17:41:51 2016 -0800

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/View.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[1/2] incubator-beam git commit: View.asMap: minor javadoc fixes

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master abdbee61c -> 7d1976b26


View.asMap: minor javadoc fixes


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/6a4a6997
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/6a4a6997
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/6a4a6997

Branch: refs/heads/master
Commit: 6a4a699796fcf8a294ee0886658e6597bede0207
Parents: abdbee6
Author: Dan Halperin 
Authored: Fri Dec 16 16:26:27 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 16:49:19 2016 -0800

--
 .../core/src/main/java/org/apache/beam/sdk/transforms/View.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/6a4a6997/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
index 126679d..d18a0c6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/View.java
@@ -189,9 +189,9 @@ public class View {
 
   /**
* Returns a {@link View.AsMap} transform that takes a
-   * {@link PCollection PCollectionKVK V} as
+   * {@link PCollection PCollectionKVK, V} as
* input and produces a {@link PCollectionView} mapping each window to
-   * a {@link Map MapK, V}. It is required that each key of the input 
be
+   * a {@link Map MapK, V}. It is required that each key of the input 
be
* associated with a single value, per window. If this is not the case, 
precede this
* view with {@code Combine.perKey}, as in the example below, or 
alternatively
* use {@link View#asMultimap()}.

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #2128

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1840

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1840

[Apache Jenkins Server]

See 

[2/2] incubator-beam git commit: Closes #1649

[bchambers]

Closes #1649


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/abdbee61
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/abdbee61
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/abdbee61

Branch: refs/heads/master
Commit: abdbee61c7fa2383343f86f3e88f134163177439
Parents: 1ee191f 92dbab8
Author: bchambers 
Authored: Fri Dec 16 16:26:43 2016 -0800
Committer: bchambers 
Committed: Fri Dec 16 16:26:43 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/DistributionCell.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[GitHub] incubator-beam pull request #1649: checkstyle: missed newline in Distributio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1649


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: checkstyle: missed newline in DistributionCell

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 1ee191fb8 -> abdbee61c


checkstyle: missed newline in DistributionCell


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/92dbab8e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/92dbab8e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/92dbab8e

Branch: refs/heads/master
Commit: 92dbab8ecf2761893759c494d33aa8583e919a6f
Parents: 1ee191f
Author: Eugene Kirpichov 
Authored: Fri Dec 16 15:56:00 2016 -0800
Committer: Eugene Kirpichov 
Committed: Fri Dec 16 15:56:00 2016 -0800

--
 .../main/java/org/apache/beam/sdk/metrics/DistributionCell.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/92dbab8e/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
index e095158..7f684a8 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
@@ -66,4 +66,5 @@ public class DistributionCell implements 
MetricCell

[jira] [Commented] (BEAM-475) High-quality javadoc for Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755901#comment-15755901
 ] 

ASF GitHub Bot commented on BEAM-475:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1650

[BEAM-475] View.asMap: minor javadoc fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam javadoc-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1650


commit b0ceccb6659d60822aa9b8a84b93384c802bdefa
Author: Dan Halperin 
Date:   2016-12-17T00:26:27Z

View.asMap: minor javadoc fixes




> High-quality javadoc for Beam
> -
>
> Key: BEAM-475
> URL: https://issues.apache.org/jira/browse/BEAM-475
> Project: Beam
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
> Fix For: Not applicable
>
>
> We should have good Javadoc for Beam!
> Current snapshot: http://beam.incubator.apache.org/javadoc/0.1.0-incubating/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1650: [BEAM-475] View.asMap: minor javadoc fixe...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1650

[BEAM-475] View.asMap: minor javadoc fixes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam javadoc-fixes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1650


commit b0ceccb6659d60822aa9b8a84b93384c802bdefa
Author: Dan Halperin 
Date:   2016-12-17T00:26:27Z

View.asMap: minor javadoc fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1839

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1174) URI Java library unexpected behavior.

[Pei He (JIRA)]

Pei He created BEAM-1174:


 Summary: URI Java library unexpected behavior.
 Key: BEAM-1174
 URL: https://issues.apache.org/jira/browse/BEAM-1174
 Project: Beam
  Issue Type: Sub-task
Reporter: Pei He
Assignee: Pei He


Following code creates an invalid URI silently.
URI uri = URI.create("gs://bucket").resolve("staging")

a). uri.toString() returns gs://bucketstaging

b). "java.net.URISyntaxException: Relative path in absolute URI: 
gs://bucketstaging" thrown by
new URI(uri.getScheme(), uri.getAuthority(), uri.getPath(), uri.getQuery(), 
uri.getFragment())

Conversions between URIs and Strings have to be handled with with extreme 
caution. :(



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-498) Make DoFnWithContext the new DoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755809#comment-15755809
 ] 

ASF GitHub Bot commented on BEAM-498:
-

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/1648

[BEAM-498] Undeletes DoFnInvokers.of(OldDoFn)

In #1565 I deleted some code that's actually necessary to support the 
Dataflow worker. Bad idea. We can't delete it until Dataflow worker stops using 
DoFnInvokers.of(OldDoFn).

R: @kennknowles 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam revert-some-old-dofn

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1648


commit 22f23a70894df213a2bfcf389d3587995b9df9f7
Author: Eugene Kirpichov 
Date:   2016-12-16T23:26:28Z

Revert "Removes code for wrapping DoFn as an OldDoFn"

This reverts commit a22de15012c51e8b7e31143021f0a298e093bf51.

commit b8f91349f7c457b878b1d343ae1b20adae955baf
Author: Eugene Kirpichov 
Date:   2016-12-16T23:26:32Z

Revert "Removes ArgumentProvider.windowingInternals"

This reverts commit f3e8a0383bf9cb3f9452e0364f7deba113cadff9.

commit 485da3549a53407e8c2a5b6b5cf69740fee68a74
Author: Eugene Kirpichov 
Date:   2016-12-16T23:37:02Z

Revert "Moves DoFnAdapters to runners-core"

This reverts commit 33ed3238e2b3899cff061be3056c5cc29fc60a04.




> Make DoFnWithContext the new DoFn
> -
>
> Key: BEAM-498
> URL: https://issues.apache.org/jira/browse/BEAM-498
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1838

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #1838

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-1173) TestSparkRunner should use a PipelineVisitor to determine expected assertions.

[Amit Sela (JIRA)]

Amit Sela created BEAM-1173:
---

 Summary: TestSparkRunner should use a PipelineVisitor to determine 
expected assertions.
 Key: BEAM-1173
 URL: https://issues.apache.org/jira/browse/BEAM-1173
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Amit Sela
Priority: Minor


Instead of checking each transform on execution, the {{TestSparkRunner}} should 
use a {{PipelineVisitor}} to determine the number of assertions in the pipeline 
in order to validate the same number of {{PAssert.SUCCESS_COUNTER}}.   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2125

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1622: Make {Metric,Counter,Distribution}Cell pu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1622


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Make {Metric, Counter, Distribution}Cell public

[bchambers]

Make {Metric,Counter,Distribution}Cell public

These classes may be useful to hold on to within a runner where the step
context is known to be unchanging and there may be a desire to avoid the
extra indirection of using {Counter,Distribution} directly.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cfd1633a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cfd1633a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cfd1633a

Branch: refs/heads/master
Commit: cfd1633a441f98d100bd995bd811b6b8fa2cd62d
Parents: cc28f0c
Author: bchambers 
Authored: Wed Dec 14 15:46:50 2016 -0800
Committer: bchambers 
Committed: Fri Dec 16 14:26:35 2016 -0800

--
 .../org/apache/beam/sdk/metrics/CounterCell.java | 10 +-
 .../apache/beam/sdk/metrics/DistributionCell.java| 15 +--
 .../java/org/apache/beam/sdk/metrics/MetricCell.java |  2 +-
 3 files changed, 23 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfd1633a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
index bb65833..93700e6 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/CounterCell.java
@@ -26,13 +26,21 @@ import org.apache.beam.sdk.annotations.Experimental.Kind;
  *
  * This class generally shouldn't be used directly. The only exception is 
within a runner where
  * a counter is being reported for a specific step (rather than the counter in 
the current context).
+ * In that case retrieving the underlying cell and reporting directly to it 
avoids a step of
+ * indirection.
  */
 @Experimental(Kind.METRICS)
-class CounterCell implements MetricCell, Counter {
+public class CounterCell implements MetricCell, Counter {
 
   private final DirtyState dirty = new DirtyState();
   private final AtomicLong value = new AtomicLong();
 
+  /**
+   * Package-visibility because all {@link CounterCell CounterCells} should be 
created by
+   * {@link MetricsContainer#getCounter(MetricName)}.
+   */
+  CounterCell() {}
+
   /** Increment the counter by the given amount. */
   private void add(long n) {
 value.addAndGet(n);

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfd1633a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
index f0074a9..e095158 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/DistributionCell.java
@@ -23,14 +23,25 @@ import org.apache.beam.sdk.annotations.Experimental.Kind;
 
 /**
  * Tracks the current value (and delta) for a Distribution metric.
+ *
+ * This class generally shouldn't be used directly. The only exception is 
within a runner where
+ * a distribution is being reported for a specific step (rather than the 
distribution in the current
+ * context). In that case retrieving the underlying cell and reporting 
directly to it avoids a step
+ * of indirection.
  */
 @Experimental(Kind.METRICS)
-class DistributionCell implements MetricCell, 
Distribution {
+public class DistributionCell implements MetricCell, Distribution {
 
   private final DirtyState dirty = new DirtyState();
   private final AtomicReference value =
   new AtomicReference(DistributionData.EMPTY);
 
+  /**
+   * Package-visibility because all {@link DistributionCell DistributionCells} 
should be created by
+   * {@link MetricsContainer#getDistribution(MetricName)}.
+   */
+  DistributionCell() {}
+
   /** Increment the counter by the given amount. */
   @Override
   public void update(long n) {
@@ -55,4 +66,4 @@ class DistributionCell implements MetricCell, Di
   public Distribution getInterface() {
 return this;
   }
-}
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cfd1633a/sdks/java/core/src/main/java/org/apache/beam/sdk/metrics/MetricCell.java
--
diff --git 

[1/2] incubator-beam git commit: Closes #1622

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cc28f0cb4 -> 1ee191fb8


Closes #1622


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1ee191fb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1ee191fb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1ee191fb

Branch: refs/heads/master
Commit: 1ee191fb80ed1644b275a87fc02c11eae47904c2
Parents: cc28f0c cfd1633
Author: bchambers 
Authored: Fri Dec 16 14:26:35 2016 -0800
Committer: bchambers 
Committed: Fri Dec 16 14:26:35 2016 -0800

--
 .../org/apache/beam/sdk/metrics/CounterCell.java | 10 +-
 .../apache/beam/sdk/metrics/DistributionCell.java| 15 +--
 .../java/org/apache/beam/sdk/metrics/MetricCell.java |  2 +-
 3 files changed, 23 insertions(+), 4 deletions(-)
--

[jira] [Created] (BEAM-1172) Precommit integration tests should include some merging windows

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1172:
-

 Summary: Precommit integration tests should include some merging 
windows
 Key: BEAM-1172
 URL: https://issues.apache.org/jira/browse/BEAM-1172
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #884

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-318) AvroCoder may be affected by AVRO-607

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-318:
-
Assignee: (was: Daniel Halperin)

> AvroCoder may be affected by AVRO-607
> -
>
> Key: BEAM-318
> URL: https://issues.apache.org/jira/browse/BEAM-318
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>
> See AVRO-607
> The hypothesis is that even though AvroCoder is threadsafe according to 
> Avro's javadoc, an underlying bug in Avro library makes thread-unsafe access 
> to a Java WeakHashMap. However I have not yet succeeded at reproducing the 
> bug; with some failed attempts in the linked GitHub Pull Request history.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1547: [BEAM-646] Add PTransformOverrideFactory ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1547


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add PTransformOverrideFactory to the Core SDK

[tgroh]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 217c29bfc -> cc28f0cb4


Add PTransformOverrideFactory to the Core SDK

This migrates PTransformOverrideFactory from the DirectRunner to the
Core SDK, as part of BEAM-646.

Add getOriginalToReplacements to provide a mapping from the original
outputs to replaced outputs. This enables all replaced nodes to be
rewired to output the original output.

Migrate all DirectRunner Override Factories to the new
PTransformOverrideFactory.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3f227a0a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3f227a0a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3f227a0a

Branch: refs/heads/master
Commit: 3f227a0ad18f425767e89f88d8a1c9fdcebdd80c
Parents: 217c29b
Author: Thomas Groh 
Authored: Mon Dec 5 16:01:57 2016 -0800
Committer: Thomas Groh 
Committed: Fri Dec 16 14:21:49 2016 -0800

--
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |  7 ++--
 .../direct/DirectGroupByKeyOverrideFactory.java |  3 +-
 .../beam/runners/direct/DirectRunner.java   |  7 +++-
 .../direct/PTransformOverrideFactory.java   | 35 -
 .../direct/ParDoMultiOverrideFactory.java   |  9 +++--
 .../ParDoSingleViaMultiOverrideFactory.java | 11 +++---
 .../direct/TestStreamEvaluatorFactory.java  |  5 ++-
 .../runners/direct/ViewEvaluatorFactory.java|  4 +-
 .../direct/WriteWithShardingFactory.java|  6 ++-
 .../direct/WriteWithShardingFactoryTest.java|  4 +-
 .../beam/sdk/annotations/Experimental.java  |  5 ++-
 .../sdk/runners/PTransformOverrideFactory.java  | 41 
 12 files changed, 80 insertions(+), 57 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3f227a0a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
index 1fa059c..ab4c114 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGBKIntoKeyedWorkItemsOverrideFactory.java
@@ -18,7 +18,8 @@
 package org.apache.beam.runners.direct;
 
 import org.apache.beam.runners.core.KeyedWorkItem;
-import org.apache.beam.runners.core.SplittableParDo;
+import org.apache.beam.runners.core.SplittableParDo.GBKIntoKeyedWorkItems;
+import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
@@ -30,10 +31,10 @@ import org.apache.beam.sdk.values.PCollection;
 class DirectGBKIntoKeyedWorkItemsOverrideFactory
 implements PTransformOverrideFactory<
 PCollection>, PCollection>,
-SplittableParDo.GBKIntoKeyedWorkItems> {
+GBKIntoKeyedWorkItems> {
   @Override
   public PTransform>, 
PCollection>>
-  override(SplittableParDo.GBKIntoKeyedWorkItems transform) {
+  getReplacementTransform(GBKIntoKeyedWorkItems transform) {
 return new DirectGroupByKey.DirectGroupByKeyOnly<>();
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/3f227a0a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
index 9acf5e9..7cf3256 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/DirectGroupByKeyOverrideFactory.java
@@ -17,6 +17,7 @@
  */
 package org.apache.beam.runners.direct;
 
+import org.apache.beam.sdk.runners.PTransformOverrideFactory;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.KV;
@@ -27,7 +28,7 @@ final class DirectGroupByKeyOverrideFactory
 implements PTransformOverrideFactory<
 

[jira] [Commented] (BEAM-646) Get runners out of the apply()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755666#comment-15755666
 ] 

ASF GitHub Bot commented on BEAM-646:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1547


> Get runners out of the apply()
> --
>
> Key: BEAM-646
> URL: https://issues.apache.org/jira/browse/BEAM-646
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>
> Right now, the runner intercepts calls to apply() and replaces transforms as 
> we go. This means that there is no "original" user graph. For portability and 
> misc architectural benefits, we would like to build the original graph first, 
> and have the runner override later.
> Some runners already work in this manner, but we could integrate it more 
> smoothly, with more validation, via some handy APIs on e.g. the Pipeline 
> object.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: This closes #1547

[tgroh]

This closes #1547


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cc28f0cb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cc28f0cb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cc28f0cb

Branch: refs/heads/master
Commit: cc28f0cb4c44169f933475ae29a32599024d3a1f
Parents: 217c29b 3f227a0
Author: Thomas Groh 
Authored: Fri Dec 16 14:21:50 2016 -0800
Committer: Thomas Groh 
Committed: Fri Dec 16 14:21:50 2016 -0800

--
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |  7 ++--
 .../direct/DirectGroupByKeyOverrideFactory.java |  3 +-
 .../beam/runners/direct/DirectRunner.java   |  7 +++-
 .../direct/PTransformOverrideFactory.java   | 35 -
 .../direct/ParDoMultiOverrideFactory.java   |  9 +++--
 .../ParDoSingleViaMultiOverrideFactory.java | 11 +++---
 .../direct/TestStreamEvaluatorFactory.java  |  5 ++-
 .../runners/direct/ViewEvaluatorFactory.java|  4 +-
 .../direct/WriteWithShardingFactory.java|  6 ++-
 .../direct/WriteWithShardingFactoryTest.java|  4 +-
 .../beam/sdk/annotations/Experimental.java  |  5 ++-
 .../sdk/runners/PTransformOverrideFactory.java  | 41 
 12 files changed, 80 insertions(+), 57 deletions(-)
--

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2124

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1647: [BEAM-1089] Replace "--none--" with a mes...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1647

[BEAM-1089] Replace "--none--" with a message in Jenkins comments

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jasonkuster (I'll grab a committer later)

I believe there was some expectation that the magic value "--none--" would 
disable comments. Instead, though, Jenkins is writing that exact string to our 
pull requests. Until we figure out how to actually disable comments, we may as 
well leave informative messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam jenkins-messages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1647


commit 814b029689606d9d37dd696e4599efa0646a204b
Author: Kenneth Knowles 
Date:   2016-12-16T21:59:31Z

Replace "--none--" with a message in Jenkins comments

I believe there was some expectation that the magic value "--none--"
would disable comments. Instead, though, Jenkins is writing that
exact string to our pull requests. Until we figure out how to actually
disable comments, we may as well leave informative messages.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1139) Investigate failures in precommit - Apex & Kryo

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1139:
--
Assignee: (was: Kenneth Knowles)

> Investigate failures in precommit - Apex & Kryo
> ---
>
> Key: BEAM-1139
> URL: https://issues.apache.org/jira/browse/BEAM-1139
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Kenneth Knowles
>Priority: Minor
>
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-examples-java/5775/testReport/junit/org.apache.beam.examples/WordCountIT/testE2EWordCount/
> This is not necessarily a bug in the Apex runner, but it looks like this 
> class cannot be serialized via Kryo while the Apex runner needs it to be. 
> Probably the fix is to roll-forwards a simple change to make it Kryo 
> serializable.
> It is not clear to me the difference between this test run and others. 
> Clearly there is a coverage gap.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1089) Jenkins comments on PRs are too many & too large

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755653#comment-15755653
 ] 

ASF GitHub Bot commented on BEAM-1089:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1647

[BEAM-1089] Replace "--none--" with a message in Jenkins comments

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jasonkuster (I'll grab a committer later)

I believe there was some expectation that the magic value "--none--" would 
disable comments. Instead, though, Jenkins is writing that exact string to our 
pull requests. Until we figure out how to actually disable comments, we may as 
well leave informative messages.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam jenkins-messages

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1647


commit 814b029689606d9d37dd696e4599efa0646a204b
Author: Kenneth Knowles 
Date:   2016-12-16T21:59:31Z

Replace "--none--" with a message in Jenkins comments

I believe there was some expectation that the magic value "--none--"
would disable comments. Instead, though, Jenkins is writing that
exact string to our pull requests. Until we figure out how to actually
disable comments, we may as well leave informative messages.




> Jenkins comments on PRs are too many & too large
> 
>
> Key: BEAM-1089
> URL: https://issues.apache.org/jira/browse/BEAM-1089
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Kenneth Knowles
>Assignee: Jason Kuster
>
> Lately, I've been finding review comments somewhat drowned out by asfbot 
> copying build results onto a PR. It also generates a lot of needless email. I 
> have not yet tried to devise just the right filter, hoping we can just return 
> to the normal practice of leaving just a commit status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755649#comment-15755649
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1592


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1592: [BEAM-362] Move ActiveWindowSet and imple...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1592


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam git commit: This closes #1592

[kenn]

This closes #1592

  Move ActiveWindowSet and implementations to runners-core
  Update Dataflow worker to beam-master-20161216


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/217c29bf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/217c29bf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/217c29bf

Branch: refs/heads/master
Commit: 217c29bfc1a4758227a78defc9b35e0e41de5df1
Parents: e017ca1 bcf0298
Author: Kenneth Knowles <k...@google.com>
Authored: Fri Dec 16 14:05:57 2016 -0800
Committer: Kenneth Knowles <k...@google.com>
Committed: Fri Dec 16 14:05:57 2016 -0800

--
 .../beam/runners/core/ActiveWindowSet.java  | 173 
 .../runners/core/MergingActiveWindowSet.java| 428 +++
 .../runners/core/NonMergingActiveWindowSet.java |  91 
 .../runners/core/ReduceFnContextFactory.java|   1 -
 .../beam/runners/core/ReduceFnRunner.java   |   3 -
 .../TriggerStateMachineContextFactory.java  |   2 +-
 .../core/MergingActiveWindowSetTest.java| 262 
 .../triggers/TriggerStateMachineTester.java |   8 +-
 .../beam/runners/dataflow/dataflow.properties   |   4 +-
 .../apache/beam/sdk/util/ActiveWindowSet.java   | 173 
 .../beam/sdk/util/MergingActiveWindowSet.java   | 428 ---
 .../sdk/util/NonMergingActiveWindowSet.java |  91 
 .../sdk/util/MergingActiveWindowSetTest.java| 262 
 13 files changed, 961 insertions(+), 965 deletions(-)
--

[jira] [Updated] (BEAM-1170) Streaming watermark should be easier to read

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers updated BEAM-1170:
---
Component/s: runner-dataflow

> Streaming watermark should be easier to read
> 
>
> Key: BEAM-1170
> URL: https://issues.apache.org/jira/browse/BEAM-1170
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Joshua Litt
>Priority: Minor
>
> Currently, the only way to get at the streaming watermarks is through 
> scraping counter names. However, the watermarks are useful for determining if 
> a streaming job is 'done,' ie watermarks at infinity. We should consider 
> either exposing the watermarks through a GetWatermarks api or another 
> alternative might be a WATERMARKS_AT_INFINITY job state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755599#comment-15755599
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1643


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1643: [BEAM-362] Move InMemoryTimerInternals to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1643


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam git commit: Delete deprecated TimerCallback

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master beed6080b -> e017ca108


Delete deprecated TimerCallback


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/5587e1cc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/5587e1cc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/5587e1cc

Branch: refs/heads/master
Commit: 5587e1cccb08839ef57a7d19b5dc3cf0a3725646
Parents: d624d3b
Author: Kenneth Knowles 
Authored: Thu Dec 15 20:26:40 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 22:38:10 2016 -0800

--
 .../sdk/util/state/InMemoryTimerInternals.java  | 25 -
 .../beam/sdk/util/state/TimerCallback.java  | 38 
 2 files changed, 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5587e1cc/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
index 44f9016..44b44f0 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/InMemoryTimerInternals.java
@@ -272,29 +272,4 @@ public class InMemoryTimerInternals implements 
TimerInternals {
   return null;
 }
   }
-
-  /** Advances processing time to the given value and fires processing-time 
timers accordingly.
-   *
-   *  @deprecated Use advanceProcessingTime without callback and 
fireProcessingTimers.
-   */
-  @Deprecated
-  public void advanceProcessingTime(
-  TimerCallback timerCallback, Instant newProcessingTime) throws Exception 
{
-advanceProcessingTime(newProcessingTime);
-advanceAndFire(timerCallback, newProcessingTime, 
TimeDomain.PROCESSING_TIME);
-  }
-
-  @Deprecated
-  private void advanceAndFire(
-  TimerCallback timerCallback, Instant currentTime, TimeDomain domain)
-  throws Exception {
-checkNotNull(timerCallback);
-TimerData timer;
-while ((timer = removeNextTimer(currentTime, domain)) != null) {
-  WindowTracing.trace(
-  "{}.advanceAndFire: firing {} at {}",
-  getClass().getSimpleName(), timer, currentTime);
-  timerCallback.onTimer(timer);
-}
-  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/5587e1cc/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
deleted file mode 100644
index 83791d6..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/TimerCallback.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.util.state;
-
-import org.apache.beam.sdk.util.TimerInternals;
-
-/**
- * A callback that processes a {@link TimerInternals.TimerData TimerData}.
- *
- * @deprecated Use InMemoryTimerInternals advance and remove methods instead 
of callback.
- */
-@Deprecated
-public interface TimerCallback {
-  /** Processes the {@link TimerInternals.TimerData TimerData}. */
-  void onTimer(TimerInternals.TimerData timer) throws Exception;
-
-  TimerCallback NO_OP = new TimerCallback() {
-@Override
-public void onTimer(TimerInternals.TimerData timer) throws Exception {
-  // Nothing
-}
-  };
-}

[3/3] incubator-beam git commit: This closes #1643: [BEAM-362] Move InMemoryTimerInternals to runners-core

[kenn]

This closes #1643: [BEAM-362] Move InMemoryTimerInternals to runners-core

  Move InMemoryTimerInternals to runners-core
  Delete deprecated TimerCallback


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e017ca10
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e017ca10
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e017ca10

Branch: refs/heads/master
Commit: e017ca1082e3beb017152249fcb65de3dbc38452
Parents: beed608 ec0bf7b
Author: Kenneth Knowles 
Authored: Fri Dec 16 13:32:50 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 13:32:50 2016 -0800

--
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   1 -
 .../runners/core/InMemoryTimerInternals.java| 276 +
 .../core/InMemoryTimerInternalsTest.java| 155 ++
 .../beam/runners/core/ReduceFnTester.java   |   1 -
 .../beam/runners/core/SplittableParDoTest.java  |  16 +-
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |  36 ---
 .../sdk/util/state/InMemoryTimerInternals.java  | 300 ---
 .../beam/sdk/util/state/TimerCallback.java  |  38 ---
 .../util/state/InMemoryTimerInternalsTest.java  | 153 --
 11 files changed, 446 insertions(+), 534 deletions(-)
--

[2/3] incubator-beam git commit: Move InMemoryTimerInternals to runners-core

[kenn]

Move InMemoryTimerInternals to runners-core


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ec0bf7b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ec0bf7b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ec0bf7b4

Branch: refs/heads/master
Commit: ec0bf7b4023ff75f4ec6723d2e77ed507eb57c51
Parents: 5587e1c
Author: Kenneth Knowles 
Authored: Thu Dec 15 20:45:56 2016 -0800
Committer: Kenneth Knowles 
Committed: Thu Dec 15 22:42:24 2016 -0800

--
 .../GroupAlsoByWindowsViaOutputBufferDoFn.java  |   1 -
 .../runners/core/InMemoryTimerInternals.java| 276 +++
 .../core/InMemoryTimerInternalsTest.java| 155 +++
 .../beam/runners/core/ReduceFnTester.java   |   1 -
 .../beam/runners/core/SplittableParDoTest.java  |  16 +-
 .../triggers/TriggerStateMachineTester.java |   2 +-
 .../translation/SparkGroupAlsoByWindowFn.java   |   2 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  |  36 ---
 .../sdk/util/state/InMemoryTimerInternals.java  | 275 --
 .../util/state/InMemoryTimerInternalsTest.java  | 153 --
 10 files changed, 446 insertions(+), 471 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ec0bf7b4/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
index 9189191..efcd771 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java
@@ -27,7 +27,6 @@ import org.apache.beam.sdk.util.SystemDoFnInternal;
 import org.apache.beam.sdk.util.TimerInternals;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
-import org.apache.beam.sdk.util.state.InMemoryTimerInternals;
 import org.apache.beam.sdk.util.state.StateInternals;
 import org.apache.beam.sdk.util.state.StateInternalsFactory;
 import org.joda.time.Instant;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ec0bf7b4/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
new file mode 100644
index 000..b22fcb3
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core;
+
+import static com.google.common.base.Preconditions.checkNotNull;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.common.base.MoreObjects;
+import java.util.HashSet;
+import java.util.PriorityQueue;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.apache.beam.sdk.util.TimerInternals;
+import org.apache.beam.sdk.util.WindowTracing;
+import org.apache.beam.sdk.util.state.StateNamespace;
+import org.joda.time.Instant;
+
+/**
+ * Simulates the firing of timers and progression of input and output 
watermarks for a single
+ * computation and key in a Windmill-like streaming environment.
+ */
+public class InMemoryTimerInternals implements TimerInternals {
+
+  /** At most one timer per timestamp is kept. */
+  private Set existingTimers = new HashSet<>();
+
+  /** Pending input 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Gearpump #155

[Apache Jenkins Server]

See 


Changes:

[owenzhang1990] Upgrade Gearpump version

[owenzhang1990] Add Window.Bound translator

[owenzhang1990] Skip window assignment when windows don't change

[owenzhang1990] Remove print to stdout

[owenzhang1990] Fix NoOpAggregatorFactory

--
[...truncated 26557 lines...]
  CombineTest.testAccumulatingCombine:477->runTestAccumulatingCombine:236 » 
Runtime
  CombineTest.testAccumulatingCombineEmpty:483->runTestAccumulatingCombine:236 
» Runtime
  CombineTest.testBasicCombine:221->runTestBasicCombine:200 » Runtime Failed to 
...
  CombineTest.testBasicCombineEmpty:229->runTestBasicCombine:200 » Runtime 
Faile...
  CombineTest.testCombineGloballyAsSingletonView:635 » Runtime Failed to 
constru...
  CombineTest.testCombinePerKeyPrimitiveDisplayData:712 » Runtime Failed to 
cons...
  CombineTest.testCombinePerKeyWithHotKeyFanoutPrimitiveDisplayData:729 » 
Runtime
  CombineTest.testFixedWindowsCombine:256 » Runtime Failed to construct 
instance...
  CombineTest.testFixedWindowsCombineWithContext:282 » Runtime Failed to 
constru...
  CombineTest.testGlobalCombineWithDefaultsAndTriggers:368 » Runtime Failed to 
c...
  CombineTest.testHotKeyCombining:520 » Runtime Failed to construct instance 
fro...
  CombineTest.testHotKeyCombiningWithAccumulationMode:555 » Runtime Failed to 
co...
  CombineTest.testSessionsCombine:395 » Runtime Failed to construct instance 
fro...
  CombineTest.testSessionsCombineWithContext:420 » Runtime Failed to construct 
i...
  CombineTest.testSimpleCombine:171->runTestSimpleCombine:120 » Runtime Failed 
t...
  CombineTest.testSimpleCombineEmpty:193->runTestSimpleCombine:120 » Runtime 
Fai...
  
CombineTest.testSimpleCombineWithContext:178->runTestSimpleCombineWithContext:141
 » Runtime
  
CombineTest.testSimpleCombineWithContextEmpty:186->runTestSimpleCombineWithContext:141
 » Runtime
  CombineTest.testSlidingWindowsCombineWithContext:319 » Runtime Failed to 
const...
  CombineTest.testWindowedCombineEmpty:462 » Runtime Failed to construct 
instanc...
  CountTest.testCountGloballyBasic:88 » Runtime Failed to construct instance 
fro...
  CountTest.testCountGloballyEmpty:103 » Runtime Failed to construct instance 
fr...
  CountTest.testCountPerElementBasic:52 » Runtime Failed to construct instance 
f...
  CountTest.testCountPerElementEmpty:74 » Runtime Failed to construct instance 
f...
  CreateTest.testCreate:83 » Runtime Failed to construct instance from factory 
m...
  CreateTest.testCreateEmpty:96 » Runtime Failed to construct instance from 
fact...
  CreateTest.testCreateTimestamped:238 » Runtime Failed to construct instance 
fr...
  CreateTest.testCreateTimestampedEmpty:257 » Runtime Failed to construct 
instan...
  CreateTest.testCreateWithKVVoidType:307 » Runtime Failed to construct 
instance...
  CreateTest.testCreateWithNullsAndValues:140 » Runtime Failed to construct 
inst...
  CreateTest.testCreateWithUnserializableElements:219 » Runtime Failed to 
constr...
  CreateTest.testCreateWithVoidType:298 » Runtime Failed to construct instance 
f...
  DistinctTest.testDistinct:56 » Runtime Failed to construct instance from 
facto...
  DistinctTest.testDistinctEmpty:75 » Runtime Failed to construct instance from 
...
  DistinctTest.testDistinctWithRepresentativeValue:118 » Runtime Failed to 
const...
  FilterTest.testFilterByPredicate:89 » Runtime Failed to construct instance 
fro...
  FilterTest.testFilterGreaterThan:115 » Runtime Failed to construct instance 
fr...
  FilterTest.testFilterGreaterThanEq:141 » Runtime Failed to construct instance 
...
  FilterTest.testFilterLessThan:102 » Runtime Failed to construct instance from 
...
  FilterTest.testFilterLessThanEq:128 » Runtime Failed to construct instance 
fro...
  FilterTest.testIdentityFilterByPredicate:63 » Runtime Failed to construct 
inst...
  FilterTest.testNoFilterByPredicate:76 » Runtime Failed to construct instance 
f...
  FlattenTest.testEmptyFlattenAsSideInput:157 » Runtime Failed to construct 
inst...
  FlattenTest.testFlattenInputMultipleCopies:123 » Runtime Failed to construct 
i...
  FlattenTest.testFlattenIterables:214 » Runtime Failed to construct instance 
fr...
  FlattenTest.testFlattenIterablesCollections:265 » Runtime Failed to construct 
...
  FlattenTest.testFlattenIterablesEmpty:283 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenIterablesLists:232 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenIterablesSets:247 » Runtime Failed to construct 
instanc...
  FlattenTest.testFlattenPCollectionList:77 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenPCollectionListEmpty:110 » Runtime Failed to construct 
...
  FlattenTest.testFlattenPCollectionListEmptyThenParDo:183 » Runtime Failed to 
c...
  FlattenTest.testFlattenPCollectionListThenParDo:93 » Runtime Failed to 
constru...
  GroupByKeyTest.testGroupByKey:98 » Runtime 

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Gearpump #155

[Apache Jenkins Server]

See 


Changes:

[owenzhang1990] Upgrade Gearpump version

[owenzhang1990] Add Window.Bound translator

[owenzhang1990] Skip window assignment when windows don't change

[owenzhang1990] Remove print to stdout

[owenzhang1990] Fix NoOpAggregatorFactory

--
[...truncated 26556 lines...]
  CombineFnsTest.testComposedCombineWithContext:181 » Runtime Failed to 
construc...
  CombineTest.testAccumulatingCombine:477->runTestAccumulatingCombine:236 » 
Runtime
  CombineTest.testAccumulatingCombineEmpty:483->runTestAccumulatingCombine:236 
» Runtime
  CombineTest.testBasicCombine:221->runTestBasicCombine:200 » Runtime Failed to 
...
  CombineTest.testBasicCombineEmpty:229->runTestBasicCombine:200 » Runtime 
Faile...
  CombineTest.testCombineGloballyAsSingletonView:635 » Runtime Failed to 
constru...
  CombineTest.testCombinePerKeyPrimitiveDisplayData:712 » Runtime Failed to 
cons...
  CombineTest.testCombinePerKeyWithHotKeyFanoutPrimitiveDisplayData:729 » 
Runtime
  CombineTest.testFixedWindowsCombine:256 » Runtime Failed to construct 
instance...
  CombineTest.testFixedWindowsCombineWithContext:282 » Runtime Failed to 
constru...
  CombineTest.testGlobalCombineWithDefaultsAndTriggers:368 » Runtime Failed to 
c...
  CombineTest.testHotKeyCombining:520 » Runtime Failed to construct instance 
fro...
  CombineTest.testHotKeyCombiningWithAccumulationMode:555 » Runtime Failed to 
co...
  CombineTest.testSessionsCombine:395 » Runtime Failed to construct instance 
fro...
  CombineTest.testSessionsCombineWithContext:420 » Runtime Failed to construct 
i...
  CombineTest.testSimpleCombine:171->runTestSimpleCombine:120 » Runtime Failed 
t...
  CombineTest.testSimpleCombineEmpty:193->runTestSimpleCombine:120 » Runtime 
Fai...
  
CombineTest.testSimpleCombineWithContext:178->runTestSimpleCombineWithContext:141
 » Runtime
  
CombineTest.testSimpleCombineWithContextEmpty:186->runTestSimpleCombineWithContext:141
 » Runtime
  CombineTest.testSlidingWindowsCombineWithContext:319 » Runtime Failed to 
const...
  CombineTest.testWindowedCombineEmpty:462 » Runtime Failed to construct 
instanc...
  CountTest.testCountGloballyBasic:88 » Runtime Failed to construct instance 
fro...
  CountTest.testCountGloballyEmpty:103 » Runtime Failed to construct instance 
fr...
  CountTest.testCountPerElementBasic:52 » Runtime Failed to construct instance 
f...
  CountTest.testCountPerElementEmpty:74 » Runtime Failed to construct instance 
f...
  CreateTest.testCreate:83 » Runtime Failed to construct instance from factory 
m...
  CreateTest.testCreateEmpty:96 » Runtime Failed to construct instance from 
fact...
  CreateTest.testCreateTimestamped:238 » Runtime Failed to construct instance 
fr...
  CreateTest.testCreateTimestampedEmpty:257 » Runtime Failed to construct 
instan...
  CreateTest.testCreateWithKVVoidType:307 » Runtime Failed to construct 
instance...
  CreateTest.testCreateWithNullsAndValues:140 » Runtime Failed to construct 
inst...
  CreateTest.testCreateWithUnserializableElements:219 » Runtime Failed to 
constr...
  CreateTest.testCreateWithVoidType:298 » Runtime Failed to construct instance 
f...
  DistinctTest.testDistinct:56 » Runtime Failed to construct instance from 
facto...
  DistinctTest.testDistinctEmpty:75 » Runtime Failed to construct instance from 
...
  DistinctTest.testDistinctWithRepresentativeValue:118 » Runtime Failed to 
const...
  FilterTest.testFilterByPredicate:89 » Runtime Failed to construct instance 
fro...
  FilterTest.testFilterGreaterThan:115 » Runtime Failed to construct instance 
fr...
  FilterTest.testFilterGreaterThanEq:141 » Runtime Failed to construct instance 
...
  FilterTest.testFilterLessThan:102 » Runtime Failed to construct instance from 
...
  FilterTest.testFilterLessThanEq:128 » Runtime Failed to construct instance 
fro...
  FilterTest.testIdentityFilterByPredicate:63 » Runtime Failed to construct 
inst...
  FilterTest.testNoFilterByPredicate:76 » Runtime Failed to construct instance 
f...
  FlattenTest.testEmptyFlattenAsSideInput:157 » Runtime Failed to construct 
inst...
  FlattenTest.testFlattenInputMultipleCopies:123 » Runtime Failed to construct 
i...
  FlattenTest.testFlattenIterables:214 » Runtime Failed to construct instance 
fr...
  FlattenTest.testFlattenIterablesCollections:265 » Runtime Failed to construct 
...
  FlattenTest.testFlattenIterablesEmpty:283 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenIterablesLists:232 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenIterablesSets:247 » Runtime Failed to construct 
instanc...
  FlattenTest.testFlattenPCollectionList:77 » Runtime Failed to construct 
instan...
  FlattenTest.testFlattenPCollectionListEmpty:110 » Runtime Failed to construct 
...
  FlattenTest.testFlattenPCollectionListEmptyThenParDo:183 » Runtime Failed to 
c...
  

[2/6] incubator-beam git commit: Add Window.Bound translator

[kenn]

Add Window.Bound translator


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/85d54ab2
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/85d54ab2
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/85d54ab2

Branch: refs/heads/gearpump-runner
Commit: 85d54ab20f21297da25059ed7b4c8ed02e93bb74
Parents: 46d3563
Author: manuzhang 
Authored: Fri Dec 16 16:49:06 2016 +0800
Committer: manuzhang 
Committed: Fri Dec 16 16:49:06 2016 +0800

--
 .../gearpump/GearpumpPipelineTranslator.java|  3 +
 .../translators/WindowBoundTranslator.java  | 97 
 2 files changed, 100 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/85d54ab2/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineTranslator.java
index 84dfeec..20624ed 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpPipelineTranslator.java
@@ -29,6 +29,7 @@ import 
org.apache.beam.runners.gearpump.translators.ReadBoundedTranslator;
 import org.apache.beam.runners.gearpump.translators.ReadUnboundedTranslator;
 import org.apache.beam.runners.gearpump.translators.TransformTranslator;
 import org.apache.beam.runners.gearpump.translators.TranslationContext;
+import org.apache.beam.runners.gearpump.translators.WindowBoundTranslator;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.io.Read;
 import org.apache.beam.sdk.runners.TransformHierarchy;
@@ -37,6 +38,7 @@ import org.apache.beam.sdk.transforms.Flatten;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.values.PValue;
 
 import org.apache.gearpump.util.Graph;
@@ -71,6 +73,7 @@ public class GearpumpPipelineTranslator extends 
Pipeline.PipelineVisitor.Default
 registerTransformTranslator(Flatten.FlattenPCollectionList.class,
 new FlattenPCollectionTranslator());
 registerTransformTranslator(ParDo.BoundMulti.class, new 
ParDoBoundMultiTranslator());
+registerTransformTranslator(Window.Bound.class, new 
WindowBoundTranslator());
 registerTransformTranslator(Create.Values.class, new 
CreateValuesTranslator());
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/85d54ab2/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowBoundTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowBoundTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowBoundTranslator.java
new file mode 100644
index 000..11f30fc
--- /dev/null
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/WindowBoundTranslator.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.beam.runners.gearpump.translators;
+
+import com.google.common.collect.Iterables;
+
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.Window;
+import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.WindowedValue;
+import org.apache.beam.sdk.util.WindowingStrategy;
+import org.apache.beam.sdk.values.PCollection;
+import 

[3/6] incubator-beam git commit: Skip window assignment when windows don't change

[kenn]

Skip window assignment when windows don't change


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c37de002
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c37de002
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c37de002

Branch: refs/heads/gearpump-runner
Commit: c37de00266b35ca6f04ddbe7b564f9054b26b622
Parents: 85d54ab
Author: manuzhang 
Authored: Fri Dec 16 16:49:38 2016 +0800
Committer: manuzhang 
Committed: Fri Dec 16 16:49:38 2016 +0800

--
 .../beam/runners/gearpump/GearpumpRunner.java   | 60 
 1 file changed, 11 insertions(+), 49 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c37de002/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
index ed0813d..8477870 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/GearpumpRunner.java
@@ -23,20 +23,19 @@ import com.typesafe.config.ConfigValueFactory;
 import java.util.HashMap;
 import java.util.Map;
 
-import org.apache.beam.runners.core.AssignWindows;
 import org.apache.beam.runners.gearpump.translators.TranslationContext;
 import org.apache.beam.sdk.Pipeline;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsValidator;
 import org.apache.beam.sdk.runners.PipelineRunner;
 import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.Flatten;
-import org.apache.beam.sdk.transforms.OldDoFn;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.transforms.ParDo;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.Window;
 import org.apache.beam.sdk.transforms.windowing.WindowFn;
+import org.apache.beam.sdk.util.IdentityWindowFn;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionList;
@@ -75,9 +74,10 @@ public class GearpumpRunner extends 
PipelineRunner {
 
   public  OutputT apply(
   PTransform transform, InputT input) {
-if (Window.Bound.class.equals(transform.getClass())) {
+if (Window.Bound.class.equals(transform.getClass())
+&& isNullOrIdentityWindowFn(((Window.Bound) transform).getWindowFn())) 
{
   return (OutputT) super.apply(
-  new AssignWindowsAndSetStrategy((Window.Bound) transform), 
input);
+  ParDo.of(new IdentityFn()), input);
 } else if 
(Flatten.FlattenPCollectionList.class.equals(transform.getClass())
 && ((PCollectionList) input).size() == 0) {
   return (OutputT) Pipeline.applyTransform(input.getPipeline().begin(), 
Create.of());
@@ -139,53 +139,15 @@ public class GearpumpRunner extends 
PipelineRunner {
 return config.withValue(GEARPUMP_SERIALIZERS, 
ConfigValueFactory.fromMap(serializers));
   }
 
+  private static class IdentityFn extends DoFn {
 
-  /**
-   * copied from DirectPipelineRunner.
-   * used to replace Window.Bound till window function is added to Gearpump 
Stream DSL
-   */
-  private static class AssignWindowsAndSetStrategy
-  extends PTransform {
-
-private final Window.Bound wrapped;
-
-AssignWindowsAndSetStrategy(Window.Bound wrapped) {
-  this.wrapped = wrapped;
-}
-
-@Override
-public PCollection apply(PCollection input) {
-  WindowingStrategy outputStrategy =
-  wrapped.getOutputStrategyInternal(input.getWindowingStrategy());
-
-  WindowFn windowFn =
-  (WindowFn) outputStrategy.getWindowFn();
-
-  if (!windowFn.isNonMerging()) {
-throw new UnsupportedOperationException(
-"merging window is not supported in Gearpump pipeline");
-  }
-
-  // If the Window.Bound transform only changed parts other than the 
WindowFn, then
-  // we skip AssignWindows even though it should be harmless in a perfect 
world.
-  // The world is not perfect, and a GBK may have set it to InvalidWindows 
to forcibly
-  // crash if another GBK is performed without explicitly setting the 
WindowFn. So we skip
-  // AssignWindows in this case.
-  if (wrapped.getWindowFn() == null) {
-return 

[4/6] incubator-beam git commit: Remove print to stdout

[kenn]

Remove print to stdout


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cb8c5e5e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cb8c5e5e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cb8c5e5e

Branch: refs/heads/gearpump-runner
Commit: cb8c5e5e59348897e0bc04d5ffb885c15a18c6af
Parents: c37de00
Author: manuzhang 
Authored: Fri Dec 16 16:50:08 2016 +0800
Committer: manuzhang 
Committed: Fri Dec 16 16:50:08 2016 +0800

--
 .../runners/gearpump/translators/FlattenPCollectionTranslator.java  | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/cb8c5e5e/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionTranslator.java
index b06d5a8..b740ab5 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/FlattenPCollectionTranslator.java
@@ -33,7 +33,6 @@ public class FlattenPCollectionTranslator implements
   @Override
   public void translate(Flatten.FlattenPCollectionList transform, 
TranslationContext context) {
 JavaStream merged = null;
-System.out.println("PCollectionList size " + 
context.getInput(transform).size());
 for (PCollection collection : context.getInput(transform).getAll()) {
   JavaStream inputStream = context.getInputStream(collection);
   if (null == merged) {

[GitHub] incubator-beam pull request #1623: [BEAM-1086] Upgrade to latest Gearpump sn...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1623


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/6] incubator-beam git commit: Upgrade Gearpump version

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/gearpump-runner 88de0cb23 -> b6e7bb659


Upgrade Gearpump version


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/46d3563e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/46d3563e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/46d3563e

Branch: refs/heads/gearpump-runner
Commit: 46d3563ec37e7f5b39ae564ac73c7ca2b0185bb5
Parents: 88de0cb
Author: manuzhang 
Authored: Fri Dec 16 16:48:16 2016 +0800
Committer: manuzhang 
Committed: Fri Dec 16 16:48:16 2016 +0800

--
 runners/gearpump/pom.xml|  39 +--
 .../translators/GroupByKeyTranslator.java   | 109 ++-
 .../gearpump/translators/io/GearpumpSource.java |  28 +++--
 .../gearpump/translators/io/ValuesSource.java   |  21 ++--
 .../translators/utils/TranslatorUtils.java  |  35 ++
 5 files changed, 148 insertions(+), 84 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/46d3563e/runners/gearpump/pom.xml
--
diff --git a/runners/gearpump/pom.xml b/runners/gearpump/pom.xml
index 04bd724..9320561 100644
--- a/runners/gearpump/pom.xml
+++ b/runners/gearpump/pom.xml
@@ -43,7 +43,7 @@
   
 UTF-8
 UTF-8
-0.8.1
+0.8.3-SNAPSHOT
   
 
   
@@ -121,29 +121,12 @@
   gearpump-core_2.11
   ${gearpump.version}
   provided
-  
-
-  com.google.code.findbugs
-  jsr305
-
-  
-
-
-  org.apache.gearpump
-  gearpump-daemon_2.11
-  ${gearpump.version}
-  
-
-  org.apache.gearpump
-  gearpump-experimental-cgroup_2.11
-
-  
 
 
   com.typesafe
   config
-  provided
   1.3.0
+  provided
 
 
   org.scala-lang
@@ -170,10 +153,6 @@
   beam-runners-core-java
 
 
-  org.slf4j
-  slf4j-api
-
-
   joda-time
   joda-time
 
@@ -198,20 +177,6 @@
   jsr305
 
 
-  org.apache.gearpump
-  gearpump-shaded-metrics-graphite_2.11
-  ${gearpump.version}
-  assembly
-  test
-
-
-  org.apache.gearpump
-  gearpump-shaded-guava_2.11
-  ${gearpump.version}
-  assembly
-  test
-
-
   junit
   junit
   test

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/46d3563e/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
index 43e3336..d64f1bf 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/GroupByKeyTranslator.java
@@ -19,23 +19,33 @@
 package org.apache.beam.runners.gearpump.translators;
 
 import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
 
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Iterator;
+import java.io.Serializable;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.LinkedList;
 import java.util.List;
 
+import org.apache.beam.runners.gearpump.translators.utils.TranslatorUtils;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.GlobalWindow;
+import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.values.KV;
 
+import org.apache.beam.sdk.values.PCollection;
 import org.apache.gearpump.streaming.dsl.javaapi.JavaStream;
-import org.apache.gearpump.streaming.javaapi.dsl.functions.FlatMapFunction;
+import org.apache.gearpump.streaming.dsl.window.api.Accumulating$;
+import org.apache.gearpump.streaming.dsl.window.api.EventTimeTrigger$;
+import org.apache.gearpump.streaming.dsl.window.api.Window;
+import org.apache.gearpump.streaming.dsl.window.api.WindowFn;
+import org.apache.gearpump.streaming.dsl.window.impl.Bucket;
 import org.apache.gearpump.streaming.javaapi.dsl.functions.GroupByFunction;
 import org.apache.gearpump.streaming.javaapi.dsl.functions.MapFunction;
 import org.apache.gearpump.streaming.javaapi.dsl.functions.ReduceFunction;
-
+import scala.collection.JavaConversions;
 
 
 /**
@@ -44,56 +54,97 @@ import 

[5/6] incubator-beam git commit: Fix NoOpAggregatorFactory

[kenn]

Fix NoOpAggregatorFactory


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8e0e8199
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8e0e8199
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8e0e8199

Branch: refs/heads/gearpump-runner
Commit: 8e0e81999c118b7d24402088a0a63601c967b3c0
Parents: cb8c5e5
Author: manuzhang 
Authored: Fri Dec 16 16:50:30 2016 +0800
Committer: manuzhang 
Committed: Fri Dec 16 16:50:30 2016 +0800

--
 .../utils/NoOpAggregatorFactory.java| 24 +++-
 1 file changed, 23 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8e0e8199/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/NoOpAggregatorFactory.java
--
diff --git 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/NoOpAggregatorFactory.java
 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/NoOpAggregatorFactory.java
index cd404a5..bfc73bf 100644
--- 
a/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/NoOpAggregatorFactory.java
+++ 
b/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils/NoOpAggregatorFactory.java
@@ -36,6 +36,28 @@ public class NoOpAggregatorFactory implements 
AggregatorFactory, Serializable {
   ExecutionContext.StepContext stepContext,
   String aggregatorName,
   Combine.CombineFn combine) {
-return null;
+return new NoOpAggregator<>();
   }
+
+  private static class NoOpAggregator implements 
Aggregator,
+  java.io.Serializable {
+private static final long serialVersionUID = 1L;
+
+@Override
+public void addValue(InputT value) {
+}
+
+@Override
+public String getName() {
+  // TODO Auto-generated method stub
+  return null;
+}
+
+@Override
+public Combine.CombineFn getCombineFn() {
+  // TODO Auto-generated method stub
+  return null;
+}
+
+  };
 }

[6/6] incubator-beam git commit: This closes #1623: [BEAM-1086] Upgrade to latest Gearpump snapshot

[kenn]

This closes #1623: [BEAM-1086] Upgrade to latest Gearpump snapshot

  Fix NoOpAggregatorFactory
  Remove print to stdout
  Skip window assignment when windows don't change
  Add Window.Bound translator
  Upgrade Gearpump version


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b6e7bb65
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b6e7bb65
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b6e7bb65

Branch: refs/heads/gearpump-runner
Commit: b6e7bb659f33e346c00e66ca96e3c54dd7ef07da
Parents: 88de0cb 8e0e819
Author: Kenneth Knowles 
Authored: Fri Dec 16 13:17:55 2016 -0800
Committer: Kenneth Knowles 
Committed: Fri Dec 16 13:17:55 2016 -0800

--
 runners/gearpump/pom.xml|  39 +--
 .../gearpump/GearpumpPipelineTranslator.java|   3 +
 .../beam/runners/gearpump/GearpumpRunner.java   |  60 ++
 .../FlattenPCollectionTranslator.java   |   1 -
 .../translators/GroupByKeyTranslator.java   | 109 ++-
 .../translators/WindowBoundTranslator.java  |  97 +
 .../gearpump/translators/io/GearpumpSource.java |  28 +++--
 .../gearpump/translators/io/ValuesSource.java   |  21 ++--
 .../utils/NoOpAggregatorFactory.java|  24 +++-
 .../translators/utils/TranslatorUtils.java  |  35 ++
 10 files changed, 282 insertions(+), 135 deletions(-)
--

[jira] [Commented] (BEAM-1040) Hadoop InputFormat - IO Transform for reads

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755524#comment-15755524
 ] 

Stephen Sisk commented on BEAM-1040:


Work is in progress on this. Hoping to have another update mid-january.

> Hadoop InputFormat - IO Transform for reads
> ---
>
> Key: BEAM-1040
> URL: https://issues.apache.org/jira/browse/BEAM-1040
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> We should build a IO read transform that will read data from services 
> supporting the Hadoop InputFormat  interface [1]
> This will make it easy to connect to a variety of databases while still 
> providing some aspects of scalability since the InputFormat interface 
> provides for parallel reading. 
> [1] 
> https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/mapred/InputFormat.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1171) Source.getDefaultOutputCoder() - add a version that allows access to a CoderRegistry

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1171:
--

 Summary: Source.getDefaultOutputCoder() - add a version that 
allows access to a CoderRegistry
 Key: BEAM-1171
 URL: https://issues.apache.org/jira/browse/BEAM-1171
 Project: Beam
  Issue Type: Bug
  Components: beam-model, sdk-java-core
Reporter: Stephen Sisk
Assignee: Stephen Sisk


The Source API's getDefaultOutputCoder currently takes no parameters. This 
means that you can't access the coder registry to ask for coders based on the 
type.

The PTransform API's version of getDefaultOutputCoder has 3 different form, 
which do allow access to the coder registry if necessary.

However, while working on BEAM-1040 (Hadoop IO connector), we realized that a 
source does not always know it's output type, and thus will need to calculate 
the coder at run time. To do this, it would be helpful to have a reference to 
the CoderRegistry - we should look at what we're doing on PTransform and decide 
if we should port that over onto Source.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1170) Streaming watermark should be easier to read

[Joshua Litt (JIRA)]

Joshua Litt created BEAM-1170:
-

 Summary: Streaming watermark should be easier to read
 Key: BEAM-1170
 URL: https://issues.apache.org/jira/browse/BEAM-1170
 Project: Beam
  Issue Type: Improvement
Reporter: Joshua Litt
Priority: Minor


Currently, the only way to get at the streaming watermarks is through scraping 
counter names. However, the watermarks are useful for determining if a 
streaming job is 'done,' ie watermarks at infinity. We should consider either 
exposing the watermarks through a GetWatermarks api or another alternative 
might be a WATERMARKS_AT_INFINITY job state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1108) Remove deprecated Dataflow Runner options and update documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755477#comment-15755477
 ] 

ASF GitHub Bot commented on BEAM-1108:
--

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1646


> Remove deprecated Dataflow Runner options and update documentation
> --
>
> Key: BEAM-1108
> URL: https://issues.apache.org/jira/browse/BEAM-1108
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> Umbrella bug for removing deprecated {{DataflowPipelineXOptions}} 
> configurations, plus improving documentation. Will update bug description as 
> more tasks arise.
> 1. Remove the {{TEARDOWN_POLICY}} option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1646: [BEAM-1108] Remove outdated language abou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1646


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: [BEAM-1108] Remove outdated language about experimental autoscaling

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 33b7ca792 -> beed6080b


[BEAM-1108] Remove outdated language about experimental autoscaling


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b723
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b723
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b723

Branch: refs/heads/master
Commit: b7231fd4d79063b2feae1ac59d5c54f2b337
Parents: 33b7ca7
Author: Dan Halperin 
Authored: Fri Dec 16 08:23:22 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 12:47:52 2016 -0800

--
 .../apache/beam/examples/complete/TopWikipediaSessions.java | 9 -
 1 file changed, 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b723/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
--
diff --git 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
index df7f81e..8e0b815 100644
--- 
a/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
+++ 
b/examples/java/src/main/java/org/apache/beam/examples/complete/TopWikipediaSessions.java
@@ -62,15 +62,6 @@ import org.joda.time.Instant;
  *
  * The default input is {@code 
gs://apache-beam-samples/wikipedia_edits/*.json} and can be
  * overridden with {@code --input}.
- *
- * The input for this example is large enough that it's a good place to 
enable (experimental)
- * autoscaling:
- * {@code
- *   --autoscalingAlgorithm=BASIC
- *   --maxNumWorkers=20
- * }
- * 
- * This will automatically scale the number of workers up over time until the 
job completes.
  */
 public class TopWikipediaSessions {
   private static final String EXPORTED_WIKI_TABLE =

[2/2] incubator-beam git commit: Closes #1646

[dhalperi]

Closes #1646


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/beed6080
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/beed6080
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/beed6080

Branch: refs/heads/master
Commit: beed6080b157ed6ff6157386927c3455c9aa347b
Parents: 33b7ca7 b72
Author: Dan Halperin 
Authored: Fri Dec 16 12:47:53 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 12:47:53 2016 -0800

--
 .../apache/beam/examples/complete/TopWikipediaSessions.java | 9 -
 1 file changed, 9 deletions(-)
--

[jira] [Commented] (BEAM-1169) MetricsTest matchers should loosen expectations on physical values

[Ben Chambers (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755377#comment-15755377
 ] 

Ben Chambers commented on BEAM-1169:


Alternatively, we may need the Runner to define how Metrics are matched, and 
the test just sets up the "correct" values. That would allow different 
runners/tests to impose reasonable restrictions.

> MetricsTest matchers should loosen expectations on physical values
> --
>
> Key: BEAM-1169
> URL: https://issues.apache.org/jira/browse/BEAM-1169
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Ben Chambers
>
> We could use `atLeast(N)` rather than `equals(N)` for the attempted values, 
> but even that may be false without violating the behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1169) MetricsTest matchers should loosen expectations on physical values

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-1169:
--

 Summary: MetricsTest matchers should loosen expectations on 
physical values
 Key: BEAM-1169
 URL: https://issues.apache.org/jira/browse/BEAM-1169
 Project: Beam
  Issue Type: Sub-task
Reporter: Ben Chambers


We could use `atLeast(N)` rather than `equals(N)` for the attempted values, but 
even that may be false without violating the behavior.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1112) Python E2E Integration Test Framework - Batch Only

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755143#comment-15755143
 ] 

ASF GitHub Bot commented on BEAM-1112:
--

GitHub user markflyhigh reopened a pull request:

https://github.com/apache/incubator-beam/pull/1639

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - E2e test framework that supports TestRunner start and verify pipeline 
job.
   - add `TestOptions` which defined `on_success_matcher` that is used to 
verify state/output of pipeline job.
   - validate `on_success_matcher` before pipeline execution to make sure 
it's unpicklable to a subclass of BaseMatcher.
   - create a `TestDataflowRunner` which provide functionalities of 
`DataflowRunner` plus result verification.
   - provide a test verifier `PipelineStateMatcher` that can verify 
pipeline job finished in DONE or not.
 - Add wordcount_it (it = integration test) that build e2e test based on 
existing wordcount pipeline.
   - include wordcount_it to nose collector, so that wordcount_it can be 
collected and run by nose.
   - skip ITs when running unit tests from tox in precommit and postcommit.

Current changes will not change behavior of existing pre/postcommit.
Test is done by running `tox -e py27 -c sdks/python/tox.ini` for unit test 
and running wordcount_it with `TestDataflowRunner` on service 
([link](https://pantheon.corp.google.com/dataflow/job/2016-12-15_17_36_16-3857167705491723621?project=google.com:clouddfe)).

TODO:
 - Output data verifier that verify pipeline output that stores in 
filesystem.
 - Add wordcount_it to precommit and replace existing wordcount execution 
command in postcommit with a better structured nose command.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam e2e-testrunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639


commit e1e1fa3a60e1fe234829432d144d6689e240b6f0
Author: Mark Liu 
Date:   2016-12-16T01:41:20Z

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

commit 0e7007879ee082e3afe5db36107f51c03274f3f5
Author: Mark Liu 
Date:   2016-12-16T02:55:53Z

fixup! Fix Code Style




> Python E2E Integration Test Framework - Batch Only
> --
>
> Key: BEAM-1112
> URL: https://issues.apache.org/jira/browse/BEAM-1112
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Parity with Java. 
> Build e2e integration test framework that can configure and run batch 
> pipeline with specified test runner, wait for pipeline execution and verify 
> results with given verifiers in the end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1112) Python E2E Integration Test Framework - Batch Only

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15755142#comment-15755142
 ] 

ASF GitHub Bot commented on BEAM-1112:
--

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1639


> Python E2E Integration Test Framework - Batch Only
> --
>
> Key: BEAM-1112
> URL: https://issues.apache.org/jira/browse/BEAM-1112
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Parity with Java. 
> Build e2e integration test framework that can configure and run batch 
> pipeline with specified test runner, wait for pipeline execution and verify 
> results with given verifiers in the end.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1639: [BEAM-1112] Python E2E Test Framework And...

[markflyhigh]

GitHub user markflyhigh reopened a pull request:

https://github.com/apache/incubator-beam/pull/1639

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - E2e test framework that supports TestRunner start and verify pipeline 
job.
   - add `TestOptions` which defined `on_success_matcher` that is used to 
verify state/output of pipeline job.
   - validate `on_success_matcher` before pipeline execution to make sure 
it's unpicklable to a subclass of BaseMatcher.
   - create a `TestDataflowRunner` which provide functionalities of 
`DataflowRunner` plus result verification.
   - provide a test verifier `PipelineStateMatcher` that can verify 
pipeline job finished in DONE or not.
 - Add wordcount_it (it = integration test) that build e2e test based on 
existing wordcount pipeline.
   - include wordcount_it to nose collector, so that wordcount_it can be 
collected and run by nose.
   - skip ITs when running unit tests from tox in precommit and postcommit.

Current changes will not change behavior of existing pre/postcommit.
Test is done by running `tox -e py27 -c sdks/python/tox.ini` for unit test 
and running wordcount_it with `TestDataflowRunner` on service 
([link](https://pantheon.corp.google.com/dataflow/job/2016-12-15_17_36_16-3857167705491723621?project=google.com:clouddfe)).

TODO:
 - Output data verifier that verify pipeline output that stores in 
filesystem.
 - Add wordcount_it to precommit and replace existing wordcount execution 
command in postcommit with a better structured nose command.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam e2e-testrunner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639


commit e1e1fa3a60e1fe234829432d144d6689e240b6f0
Author: Mark Liu 
Date:   2016-12-16T01:41:20Z

[BEAM-1112] Python E2E Test Framework And Wordcount E2E Test

commit 0e7007879ee082e3afe5db36107f51c03274f3f5
Author: Mark Liu 
Date:   2016-12-16T02:55:53Z

fixup! Fix Code Style




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-450) Modules are shaded to the same path

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754870#comment-15754870
 ] 

ASF GitHub Bot commented on BEAM-450:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1633


> Modules are shaded to the same path
> ---
>
> Key: BEAM-450
> URL: https://issues.apache.org/jira/browse/BEAM-450
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>  Labels: newbie, starter
>
> Right now multiple modules are using the same repackaged path. We should be 
> using per-artifact paths so that they don't conflict.
> One proposal was simply to adopt 
> {{${project.groupId}.${project.artifactId}.repackaged}} as the shading 
> location. If it works.
> This is a good starter issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1633: [BEAM-450] Shade modules to separate path...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1633


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-450] Shade modules to separate paths

[dhalperi]

[BEAM-450] Shade modules to separate paths


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/235027b9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/235027b9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/235027b9

Branch: refs/heads/master
Commit: 235027b9fc6e322c469b099d168e60bf72a567db
Parents: 5ebbd50
Author: Dan Halperin 
Authored: Thu Dec 15 13:50:39 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 08:28:40 2016 -0800

--
 runners/core-java/pom.xml  | 4 ++--
 runners/google-cloud-dataflow-java/pom.xml | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/235027b9/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index b5c610b..704aeaf 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -90,11 +90,11 @@
  the second relocation. -->
 
   com.google.common
-  
org.apache.beam.sdk.repackaged.com.google.common
+  
org.apache.beam.runners.core.repackaged.com.google.common
 
 
   com.google.thirdparty
-  
org.apache.beam.sdk.repackaged.com.google.thirdparty
+  
org.apache.beam.runners.core.repackaged.com.google.thirdparty
 
   
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/235027b9/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 77187d6..46ac7ef 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -133,15 +133,15 @@
 
 com.google.common.**.testing.*
   
-  
org.apache.beam.sdk.repackaged.com.google.common
+  
org.apache.beam.runners.dataflow.repackaged.com.google.common
 
 
   com.google.thirdparty
-  
org.apache.beam.sdk.repackaged.com.google.thirdparty
+  
org.apache.beam.runners.dataflow.repackaged.com.google.thirdparty
 
 
   com.google.cloud.bigtable
-  
org.apache.beam.sdk.repackaged.com.google.cloud.bigtable
+  
org.apache.beam.runners.dataflow.repackaged.com.google.cloud.bigtable
   
 
com.google.cloud.bigtable.config.BigtableOptions*
 
com.google.cloud.bigtable.config.CredentialOptions*

[1/2] incubator-beam git commit: Closes #1633

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5ebbd500c -> 33b7ca792


Closes #1633


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/33b7ca79
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/33b7ca79
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/33b7ca79

Branch: refs/heads/master
Commit: 33b7ca7924e6f3ac7e5a9380e6330de3c316c138
Parents: 5ebbd50 235027b
Author: Dan Halperin 
Authored: Fri Dec 16 08:28:40 2016 -0800
Committer: Dan Halperin 
Committed: Fri Dec 16 08:28:40 2016 -0800

--
 runners/core-java/pom.xml  | 4 ++--
 runners/google-cloud-dataflow-java/pom.xml | 6 +++---
 2 files changed, 5 insertions(+), 5 deletions(-)
--

[jira] [Commented] (BEAM-1108) Remove deprecated Dataflow Runner options and update documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754857#comment-15754857
 ] 

ASF GitHub Bot commented on BEAM-1108:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1646

[BEAM-1108] Remove outdated language about experimental autoscaling

R: @lukecwik or @kennknowles or @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam autoscaling-language

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1646


commit 5d33aa79663a3f30dbd11ae9e8733181edde1a2c
Author: Dan Halperin 
Date:   2016-12-16T16:23:22Z

[BEAM-1108] Remove outdated language about experimental autoscaling




> Remove deprecated Dataflow Runner options and update documentation
> --
>
> Key: BEAM-1108
> URL: https://issues.apache.org/jira/browse/BEAM-1108
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> Umbrella bug for removing deprecated {{DataflowPipelineXOptions}} 
> configurations, plus improving documentation. Will update bug description as 
> more tasks arise.
> 1. Remove the {{TEARDOWN_POLICY}} option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: beam_PostCommit_Python_Verify #883

[Apache Jenkins Server]

See 

--
[...truncated 2878 lines...]
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/singleton.out"
  }
], 
"user_name": "assert_that/singleton"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s6", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.WindowInto", 
"shortValue": "WindowIntoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.WindowIntoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/WindowInto.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s4"
}, 
"serialized_fn": "", 
"user_name": "assert_that/WindowInto"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "s7", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": "WindowedValueCoder$", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": 
"TimestampCoder$eJxrYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlwhmbmpxSWJuQXOID5XIYNmYyFjbSFTkh4ANWETWg==",
 
  "component_encodings": []
}, 
{
  "@type": "SingletonCoder$", 
  "component_encodings": []
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"assert_that/ViewAsList(assert_that|WindowInto.None)/CreatePCollectionView.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s6"
}, 
"user_name": 
"assert_that/ViewAsList(assert_that|WindowInto.None)/CreatePCollectionView"
  }
}, 
{
  "kind": 

[GitHub] incubator-beam pull request #1646: [BEAM-1108] Remove outdated language abou...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1646

[BEAM-1108] Remove outdated language about experimental autoscaling

R: @lukecwik or @kennknowles or @tgroh 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam autoscaling-language

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1646


commit 5d33aa79663a3f30dbd11ae9e8733181edde1a2c
Author: Dan Halperin 
Date:   2016-12-16T16:23:22Z

[BEAM-1108] Remove outdated language about experimental autoscaling




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1041) IOChannelFactory: supports for recursively remove().

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15754636#comment-15754636
 ] 

Pei He commented on BEAM-1041:
--

Yes
I think this jira could be obsolete given the IOChannelFactory re-design will 
use URI, and an utility can delete directories recursively based on the URI 
hierarchy.

https://docs.google.com/document/d/11TdPyZ9_zmjokhNWM3Id-XJsVG3qel2lhdKTknmZ_7M/edit

> IOChannelFactory: supports for recursively remove().
> 
>
> Key: BEAM-1041
> URL: https://issues.apache.org/jira/browse/BEAM-1041
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Pei He
>Assignee: Tenghuan He
>
> 1. FileIOChannelFactory:
> Maybe using deleteDirectory() as suggested in here:
> http://stackoverflow.com/questions/779519/delete-directories-recursively-in-java
> 2. GcsIOChannelFactory:
> Since gcs doesn't have directories, remove recursively means remove all files 
> whose prefixes match.
> However, there are two issues to consider:
> First. remove("gs://mybucket/directory-to-remove")
> a. it should remove "gs://mybucket/directory-to-remove/my-file".
> b. it should not remove "gs://mybucket/directory-to-remove-my-file".
> Second, there should be a way to skip the prefix match for 
> remove(collection-of-a-thousand-files), perhaps by adding a recursive boolean 
> parameter to remove().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1168) FileBaseSink: Handle output to locations such as ".", "~", or maybe ".."

[Pei He (JIRA)]

Pei He created BEAM-1168:


 Summary: FileBaseSink: Handle output to locations such as ".", 
"~", or maybe ".."
 Key: BEAM-1168
 URL: https://issues.apache.org/jira/browse/BEAM-1168
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-java-core
Reporter: Pei He
Priority: Minor


Currently, WordCount will write to hidden files if output is set to "." or "..".
.-2-of-5
..-2-of-5

Or ~-2-of-5, if the output is set to "~".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-362) Move shared runner functionality out of SDK and into runners/core-java

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15753990#comment-15753990
 ] 

ASF GitHub Bot commented on BEAM-362:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1644


> Move shared runner functionality out of SDK and into runners/core-java
> --
>
> Key: BEAM-362
> URL: https://issues.apache.org/jira/browse/BEAM-362
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1644: [BEAM-362] Port runners to runners-core A...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1644


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---