[GitHub] incubator-beam pull request #1124: Update Dataflow container image for curre...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1124

Update Dataflow container image for current version of Apache Beam.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Tested with WordCount Dataflow job 2016-10-17_22_16_00-3640376692187174806 
in integration project.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam update_container

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1124.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1124


commit 3c47539c5954abdb00431bcfdf0ee223329a1a60
Author: Luke Cwik 
Date:   2016-10-18T04:58:26Z

Update Dataflow container image for current version of Apache Beam.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1357

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1559

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1559

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1559

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-742) Move trigger state machines to runners-core, convert triggers to AST

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584321#comment-15584321
 ] 

ASF GitHub Bot commented on BEAM-742:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1101


> Move trigger state machines to runners-core, convert triggers to AST
> 
>
> Key: BEAM-742
> URL: https://issues.apache.org/jira/browse/BEAM-742
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1101: [BEAM-742] Translation from Trigger to Tr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1101


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[12/18] incubator-beam git commit: Make return types of trigger static factory methods precise

[kenn]

Make return types of trigger static factory methods precise

This is helpful in testing and loses no abstraction - the code
locations calling these methods already know the more-specific
type that will be returned.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e1c5bfbc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e1c5bfbc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e1c5bfbc

Branch: refs/heads/master
Commit: e1c5bfbc76c0ef3766d5b1bf2dbd47e13f0ed97c
Parents: e0c5766
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:15:32 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:36 2016 -0700

--
 .../java/org/apache/beam/sdk/transforms/windowing/AfterAll.java  | 2 +-
 .../java/org/apache/beam/sdk/transforms/windowing/AfterEach.java | 2 +-
 .../org/apache/beam/sdk/transforms/windowing/AfterFirst.java | 2 +-
 .../org/apache/beam/sdk/transforms/windowing/AfterWatermark.java | 4 ++--
 .../java/org/apache/beam/sdk/transforms/windowing/Never.java | 2 +-
 .../java/org/apache/beam/sdk/transforms/windowing/Trigger.java   | 2 +-
 6 files changed, 7 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1c5bfbc/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
index cc8c97f..0e37d33 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterAll.java
@@ -41,7 +41,7 @@ public class AfterAll extends OnceTrigger {
   /**
* Returns an {@code AfterAll} {@code Trigger} with the given subtriggers.
*/
-  public static OnceTrigger of(OnceTrigger... triggers) {
+  public static AfterAll of(OnceTrigger... triggers) {
 return new AfterAll(Arrays.asList(triggers));
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1c5bfbc/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
index 629c640..961d97f 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterEach.java
@@ -54,7 +54,7 @@ public class AfterEach extends Trigger {
* Returns an {@code AfterEach} {@code Trigger} with the given subtriggers.
*/
   @SafeVarargs
-  public static Trigger inOrder(Trigger... triggers) {
+  public static AfterEach inOrder(Trigger... triggers) {
 return new AfterEach(Arrays.asList(triggers));
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1c5bfbc/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
index 6b06cfa..7840fc4 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterFirst.java
@@ -42,7 +42,7 @@ public class AfterFirst extends OnceTrigger {
   /**
* Returns an {@code AfterFirst} {@code Trigger} with the given subtriggers.
*/
-  public static OnceTrigger of(OnceTrigger... triggers) {
+  public static AfterFirst of(OnceTrigger... triggers) {
 return new AfterFirst(Arrays.asList(triggers));
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e1c5bfbc/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index da96de3..89c1ba9 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -102,11 +102,11 @@ public class AfterWatermark {
   

[09/18] incubator-beam git commit: Make Trigger#subTriggers public and non-null

[kenn]

Make Trigger#subTriggers public and non-null


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/87c7811a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/87c7811a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/87c7811a

Branch: refs/heads/master
Commit: 87c7811a3bab6bf4bc0b8b9181127fe074579898
Parents: 3c73170
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:08:53 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../beam/sdk/transforms/windowing/DefaultTrigger.java   | 2 +-
 .../org/apache/beam/sdk/transforms/windowing/Trigger.java   | 9 ++---
 .../java/org/apache/beam/sdk/util/ReshuffleTrigger.java | 2 +-
 3 files changed, 8 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87c7811a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
index d6b72ef..fee7cdf 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/DefaultTrigger.java
@@ -30,7 +30,7 @@ import org.joda.time.Instant;
 public class DefaultTrigger extends Trigger{
 
   private DefaultTrigger() {
-super(null);
+super();
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87c7811a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
index a960aa4..cfabb8b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Trigger.java
@@ -20,6 +20,7 @@ package org.apache.beam.sdk.transforms.windowing;
 import com.google.common.base.Joiner;
 import java.io.Serializable;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.Objects;
 import javax.annotation.Nullable;
@@ -263,13 +264,15 @@ public abstract class Trigger implements Serializable {
 public abstract MergingTriggerInfo trigger();
   }
 
-  @Nullable
   protected final List subTriggers;
 
-  protected Trigger(@Nullable List subTriggers) {
+  protected Trigger(List subTriggers) {
 this.subTriggers = subTriggers;
   }
 
+  protected Trigger() {
+this(Collections.EMPTY_LIST);
+  }
 
   /**
* Called every time an element is incorporated into a window.
@@ -370,7 +373,7 @@ public abstract class Trigger implements Serializable {
 }
   }
 
-  public Iterable subTriggers() {
+  public List subTriggers() {
 return subTriggers;
   }
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/87c7811a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReshuffleTrigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReshuffleTrigger.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReshuffleTrigger.java
index 9e2c27d..437f14a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReshuffleTrigger.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/ReshuffleTrigger.java
@@ -31,7 +31,7 @@ import org.joda.time.Instant;
 public class ReshuffleTrigger extends Trigger {
 
   public ReshuffleTrigger() {
-super(null);
+super();
   }
 
   @Override

[02/18] incubator-beam git commit: Add accessors to Repeatedly trigger

[kenn]

Add accessors to Repeatedly trigger


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/303a42ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/303a42ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/303a42ab

Branch: refs/heads/master
Commit: 303a42ab993aee70527272dd908c3568a29f1e27
Parents: f46ce0d
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:57:48 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../apache/beam/sdk/transforms/windowing/Repeatedly.java  | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/303a42ab/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
index 8858798..45bc6c1 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Repeatedly.java
@@ -49,10 +49,16 @@ public class Repeatedly extends Trigger {
 return new Repeatedly(repeated);
   }
 
-  private Repeatedly(Trigger repeated) {
-super(Arrays.asList(repeated));
+  private Trigger repeatedTrigger;
+
+  private Repeatedly(Trigger repeatedTrigger) {
+super(Arrays.asList(repeatedTrigger));
+this.repeatedTrigger = repeatedTrigger;
   }
 
+  public Trigger getRepeatedTrigger() {
+return repeatedTrigger;
+  }
 
   @Override
   public void onElement(OnElementContext c) throws Exception {

[07/18] incubator-beam git commit: Make OrFinallyTrigger public so it can be examined

[kenn]

Make OrFinallyTrigger public so it can be examined


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2e565172
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2e565172
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2e565172

Branch: refs/heads/master
Commit: 2e565172d74bc906bdbdb1c4ee986c9e3d65089f
Parents: 303a42a
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:58:12 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2e565172/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
index 1a03450..9bef45a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
@@ -26,7 +26,7 @@ import org.joda.time.Instant;
 /**
  * Executes the {@code actual} trigger until it finishes or until the {@code 
until} trigger fires.
  */
-class OrFinallyTrigger extends Trigger {
+public class OrFinallyTrigger extends Trigger {
 
   private static final int ACTUAL = 0;
   private static final int UNTIL = 1;

[17/18] incubator-beam git commit: Add TriggerStateMachines with conversion from Trigger

[kenn]

Add TriggerStateMachines with conversion from Trigger


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/00672961
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/00672961
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/00672961

Branch: refs/heads/master
Commit: 00672961b5a3115c298c457dfe43f543947298a0
Parents: 2107f79
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:02:52 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:49 2016 -0700

--
 .../core/triggers/TriggerStateMachines.java | 210 +++
 .../core/triggers/TriggerStateMachinesTest.java | 199 ++
 2 files changed, 409 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/00672961/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
new file mode 100644
index 000..317e3b9
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/TriggerStateMachines.java
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.triggers;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.Lists;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.List;
+import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.windowing.AfterAll;
+import org.apache.beam.sdk.transforms.windowing.AfterDelayFromFirstElement;
+import org.apache.beam.sdk.transforms.windowing.AfterEach;
+import org.apache.beam.sdk.transforms.windowing.AfterFirst;
+import org.apache.beam.sdk.transforms.windowing.AfterPane;
+import org.apache.beam.sdk.transforms.windowing.AfterProcessingTime;
+import 
org.apache.beam.sdk.transforms.windowing.AfterSynchronizedProcessingTime;
+import org.apache.beam.sdk.transforms.windowing.AfterWatermark;
+import org.apache.beam.sdk.transforms.windowing.DefaultTrigger;
+import org.apache.beam.sdk.transforms.windowing.Never.NeverTrigger;
+import org.apache.beam.sdk.transforms.windowing.OrFinallyTrigger;
+import org.apache.beam.sdk.transforms.windowing.Repeatedly;
+import org.apache.beam.sdk.transforms.windowing.Trigger;
+import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
+import org.apache.beam.sdk.util.TimeDomain;
+import org.joda.time.Instant;
+
+/** Translates a {@link Trigger} to a {@link TriggerStateMachine}. */
+public class TriggerStateMachines {
+
+  private TriggerStateMachines() {}
+
+  @VisibleForTesting static final StateMachineConverter CONVERTER = new 
StateMachineConverter();
+
+  public static TriggerStateMachine stateMachineForTrigger(Trigger trigger) {
+return CONVERTER.evaluateTrigger(trigger);
+  }
+
+  public static OnceTriggerStateMachine stateMachineForOnceTrigger(OnceTrigger 
trigger) {
+return CONVERTER.evaluateOnceTrigger(trigger);
+  }
+
+  @VisibleForTesting
+  static class StateMachineConverter {
+
+public TriggerStateMachine evaluateTrigger(Trigger trigger) {
+  Method evaluationMethod = getEvaluationMethod(trigger.getClass());
+  return tryEvaluate(evaluationMethod, trigger);
+}
+
+public OnceTriggerStateMachine evaluateOnceTrigger(OnceTrigger trigger) {
+  Method evaluationMethod = getEvaluationMethod(trigger.getClass());
+  return (OnceTriggerStateMachine) tryEvaluate(evaluationMethod, trigger);
+}
+
+private TriggerStateMachine tryEvaluate(Method evaluationMethod, Trigger 
trigger) {
+  try {
+return (TriggerStateMachine) 

[13/18] incubator-beam git commit: Accessors for AfterDelay

[kenn]

Accessors for AfterDelay


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fa9b3812
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fa9b3812
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fa9b3812

Branch: refs/heads/master
Commit: fa9b3812e7262b1e0368f613d3f667b71f5de59e
Parents: e1c5bfb
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:17:14 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:36 2016 -0700

--
 .../windowing/AfterDelayFromFirstElement.java| 15 +++
 1 file changed, 15 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/fa9b3812/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterDelayFromFirstElement.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterDelayFromFirstElement.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterDelayFromFirstElement.java
index c4bc946..6078b34 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterDelayFromFirstElement.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterDelayFromFirstElement.java
@@ -97,6 +97,21 @@ public abstract class AfterDelayFromFirstElement extends 
OnceTrigger {
   }
 
   /**
+   * The time domain according for which this trigger sets timers.
+   */
+  public TimeDomain getTimeDomain() {
+return timeDomain;
+  }
+
+  /**
+   * The mapping functions applied to the arrival time of an element to 
determine when to
+   * set a wake-up timer for triggering.
+   */
+  public List> getTimestampMappers() {
+return timestampMappers;
+  }
+
+  /**
* Aligns timestamps to the smallest multiple of {@code size} since the 
{@code offset} greater
* than the timestamp.
*

[11/18] incubator-beam git commit: Add accessors for AfterPane(StateMachine) parameters

[kenn]

Add accessors for AfterPane(StateMachine) parameters


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e0c57664
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e0c57664
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e0c57664

Branch: refs/heads/master
Commit: e0c576649382ea4d1d70ee9e54ff25018210dcfb
Parents: b4445ac
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:14:32 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:36 2016 -0700

--
 .../beam/runners/core/triggers/AfterPaneStateMachine.java | 7 +++
 .../org/apache/beam/sdk/transforms/windowing/AfterPane.java   | 7 +++
 2 files changed, 14 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/e0c57664/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterPaneStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterPaneStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterPaneStateMachine.java
index 723aba6..288643d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterPaneStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterPaneStateMachine.java
@@ -49,6 +49,13 @@ private static final StateTag

[01/18] incubator-beam git commit: Add access to values from AfterDelay triggers

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7ac725583 -> c5329f9b4


Add access to values from AfterDelay triggers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b4445ac4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b4445ac4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b4445ac4

Branch: refs/heads/master
Commit: b4445ac43530441463ee11c395bcf631b66ef2e9
Parents: 0df929f
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:13:43 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../AfterDelayFromFirstElementStateMachine.java| 17 -
 1 file changed, 16 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b4445ac4/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
index a6616fa..02b156b 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterDelayFromFirstElementStateMachine.java
@@ -82,7 +82,7 @@ public abstract class AfterDelayFromFirstElementStateMachine 
extends OnceTrigger
*/
   protected final List> 
timestampMappers;
 
-  private final TimeDomain timeDomain;
+  protected final TimeDomain timeDomain;
 
   public AfterDelayFromFirstElementStateMachine(
   TimeDomain timeDomain,
@@ -97,6 +97,21 @@ public abstract class AfterDelayFromFirstElementStateMachine 
extends OnceTrigger
   }
 
   /**
+   * The time domain according for which this trigger sets timers.
+   */
+  public TimeDomain getTimeDomain() {
+return timeDomain;
+  }
+
+  /**
+   * The mapping functions applied to the arrival time of an element to 
determine when to
+   * set a wake-up timer for triggering.
+   */
+  public List> getTimestampMappers() {
+return timestampMappers;
+  }
+
+  /**
* Aligns timestamps to the smallest multiple of {@code size} since the 
{@code offset} greater
* than the timestamp.
*

[18/18] incubator-beam git commit: This closes #1101

[kenn]

This closes #1101


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c5329f9b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c5329f9b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c5329f9b

Branch: refs/heads/master
Commit: c5329f9b43c51cf1933d28f88ad44adce96ec7b6
Parents: 7ac7255 0067296
Author: Kenneth Knowles 
Authored: Mon Oct 17 20:37:24 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 20:37:24 2016 -0700

--
 .../core/triggers/AfterAllStateMachine.java |   5 +
 .../AfterDelayFromFirstElementStateMachine.java |  17 +-
 .../core/triggers/AfterEachStateMachine.java|   5 +
 .../core/triggers/AfterFirstStateMachine.java   |   6 +
 .../core/triggers/AfterPaneStateMachine.java|   7 +
 .../triggers/AfterWatermarkStateMachine.java|   4 +-
 .../core/triggers/TriggerStateMachines.java | 210 +++
 .../core/triggers/TriggerStateMachinesTest.java | 199 ++
 .../beam/sdk/transforms/windowing/AfterAll.java |   2 +-
 .../windowing/AfterDelayFromFirstElement.java   |  15 ++
 .../sdk/transforms/windowing/AfterEach.java |   2 +-
 .../sdk/transforms/windowing/AfterFirst.java|   2 +-
 .../sdk/transforms/windowing/AfterPane.java |   7 +
 .../AfterSynchronizedProcessingTime.java|   6 +-
 .../transforms/windowing/AfterWatermark.java|  12 +-
 .../transforms/windowing/DefaultTrigger.java|   2 +-
 .../beam/sdk/transforms/windowing/Never.java|   8 +-
 .../transforms/windowing/OrFinallyTrigger.java  |  17 +-
 .../sdk/transforms/windowing/Repeatedly.java|  10 +-
 .../beam/sdk/transforms/windowing/Trigger.java  |  11 +-
 .../apache/beam/sdk/util/ReshuffleTrigger.java  |   2 +-
 21 files changed, 528 insertions(+), 21 deletions(-)
--

[16/18] incubator-beam git commit: Make return types more precise for AfterWatermarkTriggerStateMachine

[kenn]

Make return types more precise for AfterWatermarkTriggerStateMachine


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/476dcd74
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/476dcd74
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/476dcd74

Branch: refs/heads/master
Commit: 476dcd740c97831a5d55953014fc5d99135addce
Parents: 703c84e
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:46:08 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:36 2016 -0700

--
 .../beam/runners/core/triggers/AfterWatermarkStateMachine.java   | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/476dcd74/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
index 5ad6214..524c057 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterWatermarkStateMachine.java
@@ -93,11 +93,11 @@ public class AfterWatermarkStateMachine {
   this.lateTrigger = lateTrigger;
 }
 
-public TriggerStateMachine withEarlyFirings(OnceTriggerStateMachine 
earlyTrigger) {
+public AfterWatermarkEarlyAndLate withEarlyFirings(OnceTriggerStateMachine 
earlyTrigger) {
   return new AfterWatermarkEarlyAndLate(earlyTrigger, lateTrigger);
 }
 
-public TriggerStateMachine withLateFirings(OnceTriggerStateMachine 
lateTrigger) {
+public AfterWatermarkEarlyAndLate withLateFirings(OnceTriggerStateMachine 
lateTrigger) {
   return new AfterWatermarkEarlyAndLate(earlyTrigger, lateTrigger);
 }
 

[08/18] incubator-beam git commit: Construct AfterFirstStateMachine with a list of subtriggers

[kenn]

Construct AfterFirstStateMachine with a list of subtriggers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/77332f1e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/77332f1e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/77332f1e

Branch: refs/heads/master
Commit: 77332f1e612caf9090e148e1493c11ca8e753076
Parents: b2bb7c0
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:43:08 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../beam/runners/core/triggers/AfterFirstStateMachine.java | 6 ++
 1 file changed, 6 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/77332f1e/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
index 272c278..f4b305e 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterFirstStateMachine.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.core.triggers;
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
 import java.util.Arrays;
 import java.util.List;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
@@ -46,6 +47,11 @@ public class AfterFirstStateMachine extends 
OnceTriggerStateMachine {
 return new 
AfterFirstStateMachine(Arrays.asList(triggers));
   }
 
+  public static OnceTriggerStateMachine of(
+  Iterable triggers) {
+return new AfterFirstStateMachine(ImmutableList.copyOf(triggers));
+  }
+
   @Override
   public void onElement(OnElementContext c) throws Exception {
 for (ExecutableTriggerStateMachine subTrigger : c.trigger().subTriggers()) 
{

[03/18] incubator-beam git commit: Add direct accessors for the components of OrFinallyTrigger

[kenn]

Add direct accessors for the components of OrFinallyTrigger


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8afb80e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8afb80e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8afb80e1

Branch: refs/heads/master
Commit: 8afb80e18f80a9d5a4ed18623a770dbf15ff5e65
Parents: b19918d
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:44:35 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../sdk/transforms/windowing/OrFinallyTrigger.java   | 15 +++
 1 file changed, 15 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8afb80e1/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
index 25b7b34..1a03450 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/OrFinallyTrigger.java
@@ -35,6 +35,21 @@ class OrFinallyTrigger extends Trigger {
 super(Arrays.asList(actual, until));
   }
 
+  /**
+   * The main trigger, which will continue firing until the "until" trigger 
fires. See
+   * {@link #getUntilTrigger()}
+   */
+  public Trigger getMainTrigger() {
+return subTriggers().get(ACTUAL);
+  }
+
+  /**
+   * The trigger that signals termination of this trigger.
+   */
+  public OnceTrigger getUntilTrigger() {
+return (OnceTrigger) subTriggers().get(UNTIL);
+  }
+
   @Override
   public void onElement(OnElementContext c) throws Exception {
 c.trigger().subTrigger(ACTUAL).invokeOnElement(c);

[04/18] incubator-beam git commit: Make AfterSynchronizedProcessingTime public

[kenn]

Make AfterSynchronizedProcessingTime public

We need to be able to access this class to reason about it when
converting a trigger to a state machine.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b19918df
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b19918df
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b19918df

Branch: refs/heads/master
Commit: b19918df1992d445ad8c13a63722c690ddca3899
Parents: 77332f1
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:43:33 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../transforms/windowing/AfterSynchronizedProcessingTime.java  | 6 +-
 1 file changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b19918df/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
index 59ece10..b96b293 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterSynchronizedProcessingTime.java
@@ -25,7 +25,11 @@ import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.util.TimeDomain;
 import org.joda.time.Instant;
 
-class AfterSynchronizedProcessingTime extends AfterDelayFromFirstElement {
+/**
+ * A trigger that fires after synchronized processing time has reached a shared
+ * threshold between upstream workers.
+ */
+public class AfterSynchronizedProcessingTime extends 
AfterDelayFromFirstElement {
 
   @Override
   @Nullable

[15/18] incubator-beam git commit: Make NeverTrigger public for translation

[kenn]

Make NeverTrigger public for translation


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/703c84ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/703c84ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/703c84ef

Branch: refs/heads/master
Commit: 703c84efd9c8219b2a8a205f0b6f1f9a999e927c
Parents: fa9b381
Author: Kenneth Knowles 
Authored: Thu Oct 13 22:17:26 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:36 2016 -0700

--
 .../java/org/apache/beam/sdk/transforms/windowing/Never.java   | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/703c84ef/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Never.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Never.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Never.java
index 353258b..07b70f4 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Never.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Never.java
@@ -39,8 +39,10 @@ public final class Never {
 return new NeverTrigger();
   }
 
-  // package-private in order to check identity for string formatting.
-  static class NeverTrigger extends OnceTrigger {
+  /**
+   * The actual trigger class for {@link Never} triggers.
+   */
+  public static class NeverTrigger extends OnceTrigger {
 protected NeverTrigger() {
   super(null);
 }

[06/18] incubator-beam git commit: Add accessors to AfterWatermark trigger

[kenn]

Add accessors to AfterWatermark trigger


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/0df929f9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/0df929f9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/0df929f9

Branch: refs/heads/master
Commit: 0df929f984c2f5af3d19a4c54635c345ccd5b410
Parents: 2e56517
Author: Kenneth Knowles 
Authored: Thu Oct 13 21:13:27 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../apache/beam/sdk/transforms/windowing/AfterWatermark.java | 8 
 1 file changed, 8 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/0df929f9/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
index e2463d8..da96de3 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/AfterWatermark.java
@@ -85,6 +85,14 @@ public class AfterWatermark {
 @Nullable
 private final OnceTrigger lateTrigger;
 
+public OnceTrigger getEarlyTrigger() {
+  return earlyTrigger;
+}
+
+public OnceTrigger getLateTrigger() {
+  return lateTrigger;
+}
+
 @SuppressWarnings("unchecked")
 public AfterWatermarkEarlyAndLate(OnceTrigger earlyTrigger, OnceTrigger 
lateTrigger) {
   super(lateTrigger == null

[05/18] incubator-beam git commit: Construct AfterEachStateMachine from list of subtriggers

[kenn]

Construct AfterEachStateMachine from list of subtriggers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/f46ce0db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/f46ce0db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/f46ce0db

Branch: refs/heads/master
Commit: f46ce0db372ffb66915eda22092bd760e474b9c0
Parents: 8afb80e
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:56:54 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../beam/runners/core/triggers/AfterEachStateMachine.java   | 5 +
 1 file changed, 5 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/f46ce0db/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
index 140ac75..38357d4 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterEachStateMachine.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.core.triggers;
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
 import java.util.Arrays;
 import java.util.List;
 
@@ -54,6 +55,10 @@ public class AfterEachStateMachine extends 
TriggerStateMachine {
 return new 
AfterEachStateMachine(Arrays.asList(triggers));
   }
 
+  public static TriggerStateMachine inOrder(Iterable triggers) {
+return new AfterEachStateMachine(ImmutableList.copyOf(triggers));
+  }
+
   @Override
   public void onElement(OnElementContext c) throws Exception {
 if (!c.trigger().isMerging()) {

[10/18] incubator-beam git commit: Construct AfterAllStateMachine with a list of subtriggers

[kenn]

Construct AfterAllStateMachine with a list of subtriggers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2bb7c04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2bb7c04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2bb7c04

Branch: refs/heads/master
Commit: b2bb7c048086a3e5eee7d2652d4bb971bc0694e7
Parents: 87c7811
Author: Kenneth Knowles 
Authored: Thu Oct 13 20:42:38 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 19:56:35 2016 -0700

--
 .../apache/beam/runners/core/triggers/AfterAllStateMachine.java | 5 +
 1 file changed, 5 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2bb7c04/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
index 2f4ad63..12cbc3d 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/triggers/AfterAllStateMachine.java
@@ -20,6 +20,7 @@ package org.apache.beam.runners.core.triggers;
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
 import java.util.Arrays;
 import java.util.List;
 import 
org.apache.beam.runners.core.triggers.TriggerStateMachine.OnceTriggerStateMachine;
@@ -45,6 +46,10 @@ public class AfterAllStateMachine extends 
OnceTriggerStateMachine {
 return new 
AfterAllStateMachine(Arrays.asList(triggers));
   }
 
+  public static OnceTriggerStateMachine of(Iterable triggers) {
+return new AfterAllStateMachine(ImmutableList.copyOf(triggers));
+  }
+
   @Override
   public void onElement(OnElementContext c) throws Exception {
 for (ExecutableTriggerStateMachine subTrigger : 
c.trigger().unfinishedSubTriggers()) {

[jira] [Created] (BEAM-766) Trigger AST should use @AutoValue

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-766:


 Summary: Trigger AST should use @AutoValue
 Key: BEAM-766
 URL: https://issues.apache.org/jira/browse/BEAM-766
 Project: Beam
  Issue Type: Improvement
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Trivial


Triggers are already enriched POJOs that could use {{@AutoValue}}. Certainly 
once they are converted to a simple AST this is the way to go.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-765) TriggerStateMachine does not need to be separate from OnceTriggerStateMachine

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-765:


 Summary: TriggerStateMachine does not need to be separate from 
OnceTriggerStateMachine
 Key: BEAM-765
 URL: https://issues.apache.org/jira/browse/BEAM-765
 Project: Beam
  Issue Type: Improvement
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Priority: Trivial


In runners-core, the state machine implementation of triggers does not need to 
have the fine-grained type-like enforcement of whether a trigger is a 
{{OnceTrigger}} or not. It may simplify the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1356

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1123: Add support for experiments flag

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/incubator-beam/pull/1123

Add support for experiments flag

Adds a new --experiment flag, to allow runners to optionally enable
experimental features.

This is similar to the same flag in the java sdk.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam exp

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1123.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1123


commit a347743f21d35081cc16c0862922f97861bb7e3f
Author: Ahmet Altay 
Date:   2016-10-18T01:17:24Z

Add support for experiments

Adds a new --experiment flag, to allow runners to optionally enable
experimental features.

This is similar to the same flag in the java sdk.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1355

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-269) Create BigDecimal Coder

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-269:
-
Issue Type: New Feature  (was: Improvement)

> Create BigDecimal Coder
> ---
>
> Key: BEAM-269
> URL: https://issues.apache.org/jira/browse/BEAM-269
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Jesse Anderson
>Assignee: Jesse Anderson
> Fix For: 0.2.0-incubating
>
>
> There isn't a coder for BigDecimal. This class is especially important for 
> financial companies to represent money.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1557

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1557

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1557

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1558

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1558

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1558

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1558

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_RunnableOnService_SparkLocal #30

[Apache Jenkins Server]

See 


Changes:

[dhalperi] [BEAM-697] TextIO.Write: Preserve validation in withHeader/withFooter

--
[...truncated 3091 lines...]
  EmptyStreamAssertionTest.testFixedWindows:67 » Runtime 
java.net.BindException:...
  FlattenStreamingTest.testFlattenBoundedUnbounded:99 » Runtime 
java.net.BindExc...
  FlattenStreamingTest.testFlattenUnbounded:79 » Runtime 
java.net.BindException:...
  KafkaStreamingTest.testRun:109 » Runtime java.net.BindException: Cannot 
assign...
  ResumeFromCheckpointStreamingTest.testRun:120->run:160 » Runtime 
java.net.Bind...
  SimpleStreamingWordCountTest.testFixedWindows:84 » Runtime 
java.net.BindExcept...

Tests run: 40, Failures: 0, Errors: 26, Skipped: 0

[ERROR] There are test failures.

Please refer to 

 for the individual test results.
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ beam-runners-spark ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
beam-runners-spark ---
[INFO] 
[INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @ 
beam-runners-spark ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ beam-runners-spark ---
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.5 in the shaded jar.
[INFO] Including commons-collections:commons-collections:jar:3.2.1 in the 
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.8.8 in the shaded 
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 in the 
shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0 in the shaded jar.
[INFO] Including io.netty:netty:jar:3.6.2.Final in the shaded jar.
[INFO] Including com.google.guava:guava:jar:19.0 in the shaded jar.
[INFO] Including com.google.auto.service:auto-service:jar:1.0-rc2 in the shaded 
jar.
[INFO] Including com.google.auto:auto-common:jar:0.3 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.7.2 in the 
shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2 in 
the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 in the 
shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.8.1 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.14 in the shaded jar.
[INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
[INFO] Including 
org.apache.beam:beam-sdks-java-core:jar:0.3.0-incubating-SNAPSHOT in the shaded 
jar.
[INFO] Including io.grpc:grpc-auth:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.auth:google-auth-library-credentials:jar:0.4.0 in 
the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.0.1 in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.3.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.3.Final in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-all:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.0.0 in the shaded 
jar.
[INFO] Including com.google.code.gson:gson:jar:2.3 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-nano:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
in the shaded jar.
[INFO] Including io.grpc:grpc-okhttp:jar:1.0.1 in the shaded jar.
[INFO] Including com.squareup.okio:okio:jar:1.6.0 in the shaded jar.
[INFO] Including com.squareup.okhttp:okhttp:jar:2.5.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.0.1 in the shaded jar.
[INFO] Including 

[jira] [Resolved] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-763.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid 
> RunnableOnService test
> --
>
> Key: BEAM-763
> URL: https://issues.apache.org/jira/browse/BEAM-763
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> TestPipeline.create(options) is not compatible with how TestPipeline 
> functions. This overrides the properties provided by the maven 
> surefire/failsafe profiles setup by the various runners for integration 
> testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-697) TextIO withHeader() and withFooter() overwrite validation setting to false

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-697:
-
Summary: TextIO withHeader() and withFooter() overwrite validation setting 
to false  (was: TextIO.Write.Bound.withHeader() and withFooter() do not use the 
current value of validate for the returned Bound instance)

> TextIO withHeader() and withFooter() overwrite validation setting to false
> --
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-697.
--
Resolution: Fixed
  Assignee: Jeffrey Payne  (was: Frances Perry)

> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Jeffrey Payne
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-697] TextIO.Write: Preserve validation in withHeader/withFooter

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 129c9bc17 -> 7ac725583


[BEAM-697] TextIO.Write: Preserve validation in withHeader/withFooter

now use the current value of validate for the returned Bound instance,
added supporting tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/b2025483
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/b2025483
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/b2025483

Branch: refs/heads/master
Commit: b202548323b4d59b11bbdf06c99d0f99e6a947ef
Parents: 129c9bc
Author: Jeffrey Scott Keone Payne 
Authored: Fri Oct 14 13:33:16 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 17:01:57 2016 -0700

--
 .../java/org/apache/beam/sdk/io/TextIO.java |  4 ++--
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 25 
 2 files changed, 27 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2025483/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
index 3ae2a0c..fbff3ba 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
@@ -667,7 +667,7 @@ public class TextIO {
*/
   public Bound withHeader(@Nullable String header) {
 return new Bound<>(name, filenamePrefix, filenameSuffix, header, 
footer, coder, numShards,
-shardTemplate, false, writableByteChannelFactory);
+shardTemplate, validate, writableByteChannelFactory);
   }
 
   /**
@@ -682,7 +682,7 @@ public class TextIO {
*/
   public Bound withFooter(@Nullable String footer) {
 return new Bound<>(name, filenamePrefix, filenameSuffix, header, 
footer, coder, numShards,
-shardTemplate, false, writableByteChannelFactory);
+shardTemplate, validate, writableByteChannelFactory);
   }
 
   /**

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/b2025483/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java
index 2131ece..dc71693 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextIOTest.java
@@ -35,6 +35,7 @@ import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.greaterThan;
 import static org.hamcrest.Matchers.hasItem;
 import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.not;
 import static org.hamcrest.Matchers.startsWith;
 import static org.junit.Assert.assertEquals;
 import static org.junit.Assert.assertFalse;
@@ -557,6 +558,30 @@ public class TextIOTest {
   }
 
   @Test
+  public void testWriteDisplayDataValidateThenHeader() {
+TextIO.Write.Bound write = TextIO.Write
+.to("foo")
+.withHeader("myHeader");
+
+DisplayData displayData = DisplayData.from(write);
+
+assertThat(displayData, hasDisplayItem("fileHeader", "myHeader"));
+assertThat(displayData, not(hasDisplayItem("validation", false)));
+  }
+
+  @Test
+  public void testWriteDisplayDataValidateThenFooter() {
+TextIO.Write.Bound write = TextIO.Write
+.to("foo")
+.withFooter("myFooter");
+
+DisplayData displayData = DisplayData.from(write);
+
+assertThat(displayData, hasDisplayItem("fileFooter", "myFooter"));
+assertThat(displayData, not(hasDisplayItem("validation", false)));
+  }
+
+  @Test
   @Category(RunnableOnService.class)
   @Ignore("[BEAM-436] DirectRunner RunnableOnService tempLocation 
configuration insufficient")
   public void testPrimitiveWriteDisplayData() throws IOException {

[2/2] incubator-beam git commit: Closes #1111

[dhalperi]

Closes #


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7ac72558
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7ac72558
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7ac72558

Branch: refs/heads/master
Commit: 7ac725583a96c0db492dfc1de69067a3bab7c7d3
Parents: 129c9bc b202548
Author: Dan Halperin 
Authored: Mon Oct 17 17:02:46 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 17:02:46 2016 -0700

--
 .../java/org/apache/beam/sdk/io/TextIO.java |  4 ++--
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 25 
 2 files changed, 27 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-697) TextIO.Write.Bound.withHeader() and withFooter() do not use the current value of validate for the returned Bound instance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583904#comment-15583904
 ] 

ASF GitHub Bot commented on BEAM-697:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/


> TextIO.Write.Bound.withHeader() and withFooter() do not use the current value 
> of validate for the returned Bound instance
> -
>
> Key: BEAM-697
> URL: https://issues.apache.org/jira/browse/BEAM-697
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Affects Versions: 0.2.0-incubating, 0.3.0-incubating
>Reporter: Jeffrey Payne
>Assignee: Frances Perry
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> {{TextIO.Write.Bound.withHeader()}} and {{TextIO.Write.Bound.withFooter()}} 
> both pass a literal value of {{false}} to the constructor's {{validate}} 
> parameter for the returned {{Bound}} instance.  This could, obviously, lead 
> to unintended behavior wrt pipeline validation.  Looks like a simple copy and 
> paste error from {{TextIO.Write.Bound.withoutValidation()}} :)
> I'm about to push another iteration for a PR for BEAM-55 and have included 
> the *trivial* fix for this bug it in that set of changes.  Creating this in 
> case my PR is not accepted right away.  Please close if this issue is wholly 
> unnecessary.  Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1111: [BEAM-697] TextIO.Write.Bound.withHeader(...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1354

[Apache Jenkins Server]

See 

[GitHub] incubator-beam-site pull request #46: Add Blog Post on TestStream

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam-site/pull/46

Add Blog Post on TestStream

The post goes into details on the previous and expanded testing
infrastructure provided by the Beam SDK and provides examples of the
times at which elements arrive, and how that impacts the outputs
produced by a PipelineRunner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam-site 
test_stream_blog_post

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam-site/pull/46.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #46


commit ec8f638568f59ac0348c44c763404aa689f7a05f
Author: Thomas Groh 
Date:   2016-10-14T00:26:30Z

Add Blog Post on TestStream

The post goes into details on the previous and expanded testing
infrastructure provided by the Beam SDK and provides examples of the
times at which elements arrive, and how that impacts the outputs
produced by a PipelineRunner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583797#comment-15583797
 ] 

ASF GitHub Bot commented on BEAM-755:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1122

[BEAM-755] SplittableParDoTest does not require RunnableOnService

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 
CC: @jkff @tgroh


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam SplittableParDoTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1122


commit 14f549f7518655aa9eb4443f0885303a6fc57ebf
Author: Kenneth Knowles 
Date:   2016-10-17T22:51:11Z

SplittableParDoTest does not require RunnableOnService




> beam-runners-core-java NeedsRunner tests not executing
> --
>
> Key: BEAM-755
> URL: https://issues.apache.org/jira/browse/BEAM-755
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Kenneth Knowles
>
> org.apache.beam:beam-runners-core-java is not specified as an integration 
> test dependency to scan within runners/pom.xml
> There is also in runners/direct-java/pom.xml where its 
> org.apache.beam:beam-runners-java-core and should be 
> org.apache.beam:beam-runners-core-java
> Finally, even if these dependencies are added and the typo fixed. When 
> running the runnable on service integration tests, SplittableParDoTest which 
> contains @RunnableOnService tests (part of runners/core-java) doesn't execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1122: [BEAM-755] SplittableParDoTest does not r...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1122

[BEAM-755] SplittableParDoTest does not require RunnableOnService

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 
CC: @jkff @tgroh


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam SplittableParDoTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1122.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1122


commit 14f549f7518655aa9eb4443f0885303a6fc57ebf
Author: Kenneth Knowles 
Date:   2016-10-17T22:51:11Z

SplittableParDoTest does not require RunnableOnService




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1556

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1353

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1556

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1556

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1555

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583650#comment-15583650
 ] 

Pei He commented on BEAM-763:
-

i don't have permission to resolve this issue. Could someone help me to mark it 
as fixed?

> BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid 
> RunnableOnService test
> --
>
> Key: BEAM-763
> URL: https://issues.apache.org/jira/browse/BEAM-763
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Pei He
>Priority: Minor
>
> TestPipeline.create(options) is not compatible with how TestPipeline 
> functions. This overrides the properties provided by the maven 
> surefire/failsafe profiles setup by the various runners for integration 
> testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is unstable: beam_PostCommit_MavenVerify #1555

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1555

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #1119: [BEAM-763] Update BigQueryIOTest.testBuil...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1119


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583643#comment-15583643
 ] 

ASF GitHub Bot commented on BEAM-763:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1119


> BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid 
> RunnableOnService test
> --
>
> Key: BEAM-763
> URL: https://issues.apache.org/jira/browse/BEAM-763
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Pei He
>Priority: Minor
>
> TestPipeline.create(options) is not compatible with how TestPipeline 
> functions. This overrides the properties provided by the maven 
> surefire/failsafe profiles setup by the various runners for integration 
> testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] incubator-beam git commit: [BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to NeedsRunner tests.

[dhalperi]

[BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to 
NeedsRunner tests.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c4530142
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c4530142
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c4530142

Branch: refs/heads/master
Commit: c4530142ebd95c2dc5b793b0892ba42f31e216b4
Parents: cc3e53f
Author: Pei He 
Authored: Mon Oct 17 13:13:52 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 15:06:04 2016 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c4530142/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
index 9d63611..74c35a6 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java
@@ -802,7 +802,7 @@ public class BigQueryIOTest implements Serializable {
   }
 
   @Test
-  @Category(RunnableOnService.class)
+  @Category(NeedsRunner.class)
   public void testBuildSourceWithTableAndSqlDialect() {
 BigQueryOptions bqOptions = 
PipelineOptionsFactory.as(BigQueryOptions.class);
 bqOptions.setProject("defaultProject");
@@ -1658,7 +1658,7 @@ public class BigQueryIOTest implements Serializable {
   }
 
   @Test
-  @Category(RunnableOnService.class)
+  @Category(NeedsRunner.class)
   public void testPassThroughThenCleanup() throws Exception {
 Pipeline p = TestPipeline.create();
 

[1/2] incubator-beam git commit: Closes #1119

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master cc3e53f32 -> 129c9bc17


Closes #1119


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/129c9bc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/129c9bc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/129c9bc1

Branch: refs/heads/master
Commit: 129c9bc176b1fb36197a79a1b4f7c77b9611b35b
Parents: cc3e53f c453014
Author: Dan Halperin 
Authored: Mon Oct 17 15:06:04 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 15:06:04 2016 -0700

--
 .../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTest.java | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

Re: Build failed in Jenkins: beam_PostCommit_RunnableOnService_SparkLocal #27

[Dan Halperin]

Sorry folks for the spam -- this is me playing around with the Spark
postcommit seeing if it somehow magically passes after 2 months of idleness.

I'll remove the email hook until I get it configured correctly.

On Mon, Oct 17, 2016 at 2:49 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  RunnableOnService_SparkLocal/27/changes>
>
> Changes:
>
> [lcwik] Move the step output ids to use a flat namespace. Also add a
> logical
>
> [dhalperi] Remove Remaining Nested Contexts from NullableCoder
>
> [bchambers] Fix SplittableParDoTest
>
> [dhalperi] AvroIO.Write: minor doc and test fixes
>
> --
> [...truncated 5861 lines...]
>   EmptyStreamAssertionTest.testFixedWindows:67 » Runtime
> java.net.BindException:...
>   FlattenStreamingTest.testFlattenBoundedUnbounded:99 » Runtime
> java.net.BindExc...
>   FlattenStreamingTest.testFlattenUnbounded:79 » Runtime
> java.net.BindException:...
>   KafkaStreamingTest.testRun:109 » Runtime java.net.BindException: Cannot
> assign...
>   ResumeFromCheckpointStreamingTest.testRun:120->run:160 » Runtime
> java.net.Bind...
>   SimpleStreamingWordCountTest.testFixedWindows:84 » Runtime
> java.net.BindExcept...
>
> Tests run: 40, Failures: 0, Errors: 26, Skipped: 0
>
> [ERROR] There are test failures.
>
> Please refer to  RunnableOnService_SparkLocal/ws/runners/spark/target/surefire-reports>
> for the individual test results.
> [JENKINS] Recording test results
> [INFO]
> [INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ beam-runners-spark ---
> [INFO] Building jar:  RunnableOnService_SparkLocal/ws/runners/spark/target/beam-
> runners-spark-0.3.0-incubating-SNAPSHOT.jar>
> [INFO]
> [INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @
> beam-runners-spark ---
> [INFO]
> [INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @
> beam-runners-spark ---
> [INFO] Building jar:  RunnableOnService_SparkLocal/ws/runners/spark/target/beam-
> runners-spark-0.3.0-incubating-SNAPSHOT-tests.jar>
> [INFO]
> [INFO] --- maven-shade-plugin:2.4.1:shade (default) @ beam-runners-spark
> ---
> [INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded
> jar.
> [INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.1 in the shaded
> jar.
> [INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
> [INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
> [INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded
> jar.
> [INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded
> jar.
> [INFO] Including commons-lang:commons-lang:jar:2.5 in the shaded jar.
> [INFO] Including commons-collections:commons-collections:jar:3.2.1 in the
> shaded jar.
> [INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.8.8 in the
> shaded jar.
> [INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 in the
> shaded jar.
> [INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0 in the
> shaded jar.
> [INFO] Including io.netty:netty:jar:3.6.2.Final in the shaded jar.
> [INFO] Including com.google.guava:guava:jar:19.0 in the shaded jar.
> [INFO] Including com.google.auto.service:auto-service:jar:1.0-rc2 in the
> shaded jar.
> [INFO] Including com.google.auto:auto-common:jar:0.3 in the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.7.2 in the
> shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2
> in the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 in
> the shaded jar.
> [INFO] Including org.apache.avro:avro:jar:1.8.1 in the shaded jar.
> [INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the
> shaded jar.
> [INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
> [INFO] Including org.slf4j:slf4j-api:jar:1.7.14 in the shaded jar.
> [INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
> [INFO] Including 
> org.apache.beam:beam-sdks-java-core:jar:0.3.0-incubating-SNAPSHOT
> in the shaded jar.
> [INFO] Including io.grpc:grpc-auth:jar:1.0.1 in the shaded jar.
> [INFO] Including com.google.auth:google-auth-library-credentials:jar:0.4.0
> in the shaded jar.
> [INFO] Including io.grpc:grpc-core:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-context:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-netty:jar:1.0.1 in the shaded jar.
> [INFO] Including io.netty:netty-codec-http2:jar:4.1.3.Final in the shaded
> jar.
> [INFO] Including io.netty:netty-codec-http:jar:4.1.3.Final in the shaded
> jar.
> [INFO] Including io.grpc:grpc-stub:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-all:jar:1.0.1 in the shaded jar.
> [INFO] Including io.grpc:grpc-protobuf:jar:1.0.1 in the shaded jar.
> [INFO] 

Build failed in Jenkins: beam_PostCommit_RunnableOnService_SparkLocal #27

[Apache Jenkins Server]

See 


Changes:

[lcwik] Move the step output ids to use a flat namespace. Also add a logical

[dhalperi] Remove Remaining Nested Contexts from NullableCoder

[bchambers] Fix SplittableParDoTest

[dhalperi] AvroIO.Write: minor doc and test fixes

--
[...truncated 5861 lines...]
  EmptyStreamAssertionTest.testFixedWindows:67 » Runtime 
java.net.BindException:...
  FlattenStreamingTest.testFlattenBoundedUnbounded:99 » Runtime 
java.net.BindExc...
  FlattenStreamingTest.testFlattenUnbounded:79 » Runtime 
java.net.BindException:...
  KafkaStreamingTest.testRun:109 » Runtime java.net.BindException: Cannot 
assign...
  ResumeFromCheckpointStreamingTest.testRun:120->run:160 » Runtime 
java.net.Bind...
  SimpleStreamingWordCountTest.testFixedWindows:84 » Runtime 
java.net.BindExcept...

Tests run: 40, Failures: 0, Errors: 26, Skipped: 0

[ERROR] There are test failures.

Please refer to 

 for the individual test results.
[JENKINS] Recording test results
[INFO] 
[INFO] --- maven-jar-plugin:2.5:jar (default-jar) @ beam-runners-spark ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
beam-runners-spark ---
[INFO] 
[INFO] --- maven-jar-plugin:2.5:test-jar (default-test-jar) @ 
beam-runners-spark ---
[INFO] Building jar: 

[INFO] 
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ beam-runners-spark ---
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[INFO] Including org.xerial.snappy:snappy-java:jar:1.1.2.1 in the shaded jar.
[INFO] Including commons-codec:commons-codec:jar:1.4 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty:jar:6.1.26 in the shaded jar.
[INFO] Including org.mortbay.jetty:jetty-util:jar:6.1.26 in the shaded jar.
[INFO] Including commons-logging:commons-logging:jar:1.1.1 in the shaded jar.
[INFO] Including commons-lang:commons-lang:jar:2.5 in the shaded jar.
[INFO] Including commons-collections:commons-collections:jar:3.2.1 in the 
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.8.8 in the shaded 
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8 in the 
shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.0.0 in the shaded jar.
[INFO] Including io.netty:netty:jar:3.6.2.Final in the shaded jar.
[INFO] Including com.google.guava:guava:jar:19.0 in the shaded jar.
[INFO] Including com.google.auto.service:auto-service:jar:1.0-rc2 in the shaded 
jar.
[INFO] Including com.google.auto:auto-common:jar:0.3 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.7.2 in the 
shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.7.2 in 
the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.7.2 in the 
shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.8.1 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.7 in the shaded jar.
[INFO] Including org.tukaani:xz:jar:1.5 in the shaded jar.
[INFO] Including org.slf4j:slf4j-api:jar:1.7.14 in the shaded jar.
[INFO] Including joda-time:joda-time:jar:2.4 in the shaded jar.
[INFO] Including 
org.apache.beam:beam-sdks-java-core:jar:0.3.0-incubating-SNAPSHOT in the shaded 
jar.
[INFO] Including io.grpc:grpc-auth:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.auth:google-auth-library-credentials:jar:0.4.0 in 
the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.0.1 in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.3.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.3.Final in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-all:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.0.0 in the shaded 
jar.
[INFO] Including com.google.code.gson:gson:jar:2.3 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-nano:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
in the shaded jar.
[INFO] Including io.grpc:grpc-okhttp:jar:1.0.1 in the shaded jar.
[INFO] Including com.squareup.okio:okio:jar:1.6.0 in the shaded jar.
[INFO] Including 

[GitHub] incubator-beam pull request #1121: Remove unnecessary rawtype cast

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1121

Remove unnecessary rawtype cast

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @jkff 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam fix-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1121


commit ce2a6502009f2cecd9b214b29124ca72b8cb9362
Author: bchambers 
Date:   2016-10-17T21:41:54Z

Remove unnecessary rawtype cast




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-757) The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's own.

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583565#comment-15583565
 ] 

Kenneth Knowles commented on BEAM-757:
--

Just a note here that I'm going to have to refactor them a bit to run {{DoFn}} 
directly instead of via wrapping to {{OldDoFn}}. So if there are things you see 
in {{DoFnRunner}} that seem like leftovers from before Beam, or other concerns, 
let me know.

> The SparkRunner should utilize the SDK's DoFnRunner instead of writing it's 
> own.
> 
>
> Key: BEAM-757
> URL: https://issues.apache.org/jira/browse/BEAM-757
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>
> The SDK now provides DoFnRunner implementations, and so to avoid maintaining 
> against the SDK, the runner should leverage the runner API instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1092: [BEAM-701] AvroIO.Write: minor doc and te...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1092


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-701) Support Avro Compression for writes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583567#comment-15583567
 ] 

ASF GitHub Bot commented on BEAM-701:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1092


> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: features
> Fix For: 0.3.0-incubating
>
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: AvroIO.Write: minor doc and test fixes

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e849d95d1 -> cc3e53f32


AvroIO.Write: minor doc and test fixes

* Document the defualt codec.
* Minor docfix in the SerializableAvroCodecFactory.
* Swap the order of assertEquals in tests to match the expected pattern 
(expected, actual).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/95ce636f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/95ce636f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/95ce636f

Branch: refs/heads/master
Commit: 95ce636f985e471a9803762d2c4236165a51f6db
Parents: e849d95
Author: Dan Halperin 
Authored: Wed Oct 12 18:10:25 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 14:38:32 2016 -0700

--
 .../src/main/java/org/apache/beam/sdk/io/AvroIO.java|  4 
 .../beam/sdk/io/SerializableAvroCodecFactory.java   |  2 +-
 .../test/java/org/apache/beam/sdk/io/AvroIOTest.java| 12 ++--
 .../beam/sdk/io/SerializableAvroCodecFactoryTest.java   |  8 
 4 files changed, 15 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95ce636f/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index eeb4bb7..d912ff7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -110,6 +110,10 @@ import org.apache.beam.sdk.values.PDone;
  * .withSuffix(".avro"));
  * } 
  *
+ * By default, {@link AvroIO.Write} produces output files that are 
compressed using the
+ * {@link org.apache.avro.file.DeflateCodec CodecFactory.deflateCodec(6)}. 
This default can
+ * be changed or overridden using {@link AvroIO.Write#withCodec}.
+ *
  * Permissions
  * Permission requirements depend on the {@link PipelineRunner} that is used 
to execute the
  * Dataflow job. Please refer to the documentation of corresponding {@link 
PipelineRunner}s for

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95ce636f/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
index ce52e99..92c2a64 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/SerializableAvroCodecFactory.java
@@ -37,7 +37,7 @@ import java.util.regex.Pattern;
 import org.apache.avro.file.CodecFactory;
 
 /**
- * A wrapper allows  {@link org.apache.avro.file.CodecFactory}s to be 
serialized using Java's
+ * A wrapper that allows {@link org.apache.avro.file.CodecFactory}s to be 
serialized using Java's
  * standard serialization mechanisms.
  */
 class SerializableAvroCodecFactory implements Externalizable {

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/95ce636f/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
index 1b1b1fa..4825875 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/io/AvroIOTest.java
@@ -178,7 +178,7 @@ public class AvroIOTest {
 p.run();
 DataFileStream dataFileStream = new DataFileStream(new 
FileInputStream(outputFile),
 new GenericDatumReader());
-assertEquals(dataFileStream.getMetaString("avro.codec"), "deflate");
+assertEquals("deflate", dataFileStream.getMetaString("avro.codec"));
   }
 
   @Test
@@ -207,7 +207,7 @@ public class AvroIOTest {
 p.run();
 DataFileStream dataFileStream = new DataFileStream(new 
FileInputStream(outputFile),
 new GenericDatumReader());
-assertEquals(dataFileStream.getMetaString("avro.codec"), "null");
+assertEquals("null", dataFileStream.getMetaString("avro.codec"));
   }
 
   @DefaultCoder(AvroCoder.class)
@@ -280,7 +280,7 @@ public class AvroIOTest {
   public void testWriteWithDefaultCodec() throws Exception {
 AvroIO.Write.Bound write = AvroIO.Write
 .to("gs://bucket/foo/baz");
-assertEquals(write.getCodec().toString(), 
CodecFactory.deflateCodec(6).toString());

[2/2] incubator-beam git commit: Closes #1092

[dhalperi]

Closes #1092


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/cc3e53f3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/cc3e53f3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/cc3e53f3

Branch: refs/heads/master
Commit: cc3e53f32881234f6f10033f873686009b2e0cfc
Parents: e849d95 95ce636
Author: Dan Halperin 
Authored: Mon Oct 17 14:38:33 2016 -0700
Committer: Dan Halperin 
Committed: Mon Oct 17 14:38:33 2016 -0700

--
 .../src/main/java/org/apache/beam/sdk/io/AvroIO.java|  4 
 .../beam/sdk/io/SerializableAvroCodecFactory.java   |  2 +-
 .../test/java/org/apache/beam/sdk/io/AvroIOTest.java| 12 ++--
 .../beam/sdk/io/SerializableAvroCodecFactoryTest.java   |  8 
 4 files changed, 15 insertions(+), 11 deletions(-)
--

[GitHub] incubator-beam pull request #1120: [BEAM-764] Remove cloneAs from PipelineOp...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1120

[BEAM-764] Remove cloneAs from PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam rm-clone-as

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1120


commit 044b47f0a9c91c92785f86d9879dbfba834c0276
Author: Pei He 
Date:   2016-10-17T21:13:42Z

[BEAM-764] Remove cloneAs from PipelineOptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-764) Remove cloneAs from PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583500#comment-15583500
 ] 

ASF GitHub Bot commented on BEAM-764:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1120

[BEAM-764] Remove cloneAs from PipelineOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam rm-clone-as

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1120


commit 044b47f0a9c91c92785f86d9879dbfba834c0276
Author: Pei He 
Date:   2016-10-17T21:13:42Z

[BEAM-764] Remove cloneAs from PipelineOptions




> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>  Labels: codehealth
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-764) Remove cloneAs from PipelineOptions

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-764:

Labels: codehealth  (was: )

> Remove cloneAs from PipelineOptions
> ---
>
> Key: BEAM-764
> URL: https://issues.apache.org/jira/browse/BEAM-764
> Project: Beam
>  Issue Type: Task
>Reporter: Pei He
>Assignee: Pei He
>  Labels: codehealth
>
> PipelineOptions.cloneAs was a workaround to support running multiple 
> pipelines in Dataflow examples for a streaming pipeline and its injector.
> After the Beam examples refactoring, cloneAs is no longer needed.
> cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
> requires users to manually set them. So, I am deleting it. 
> However, we should figure out a better API and implementation to support 
> running multiple pipelines with the same configurations (whether through 
> PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-764) Remove cloneAs from PipelineOptions

[Pei He (JIRA)]

Pei He created BEAM-764:
---

 Summary: Remove cloneAs from PipelineOptions
 Key: BEAM-764
 URL: https://issues.apache.org/jira/browse/BEAM-764
 Project: Beam
  Issue Type: Task
Reporter: Pei He
Assignee: Pei He


PipelineOptions.cloneAs was a workaround to support running multiple pipelines 
in Dataflow examples for a streaming pipeline and its injector.

After the Beam examples refactoring, cloneAs is no longer needed.
cloneAs also has known issue, such as: JsonIgnore fields are not cloned, and 
requires users to manually set them. So, I am deleting it. 

However, we should figure out a better API and implementation to support 
running multiple pipelines with the same configurations (whether through 
PipelineOptions or not).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-750) Remove StateTag

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583376#comment-15583376
 ] 

Kenneth Knowles commented on BEAM-750:
--

To be quite explicit - state kinds do not exist in the Beam model. Today all 
state tags are "system" tags - meaning that they are used in triggers and 
{{ReduceFnRunner}}. The concept of a state kind was to be sure that these are 
distinct from the state that the user uses. But there is, in fact, no 
model-level requirement that they exist in the same data store at all. So state 
kind is actually a runner-specific technique to ensure distinctness.

> Remove StateTag
> ---
>
> Key: BEAM-750
> URL: https://issues.apache.org/jira/browse/BEAM-750
> Project: Beam
>  Issue Type: Improvement
>Reporter: Eugene Kirpichov
>Assignee: Kenneth Knowles
>
> https://github.com/apache/incubator-beam/pull/1044 made StateTag in some 
> cases ignore the state kind (user/system). This is currently harmless, 
> because there is no user-facing API for defining state anyway, so conflicts 
> between user/system state can't happen.
> [~kenn] says this should be solved by removing StateTag entirely (along with 
> state kinds).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583350#comment-15583350
 ] 

Kenneth Knowles commented on BEAM-755:
--

Taking yet another look, these should actually be {{NeedsRunner}} tests for 
_both_ of {{ParDo}} and {{SplittableParDo}}.

> beam-runners-core-java NeedsRunner tests not executing
> --
>
> Key: BEAM-755
> URL: https://issues.apache.org/jira/browse/BEAM-755
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Kenneth Knowles
>
> org.apache.beam:beam-runners-core-java is not specified as an integration 
> test dependency to scan within runners/pom.xml
> There is also in runners/direct-java/pom.xml where its 
> org.apache.beam:beam-runners-java-core and should be 
> org.apache.beam:beam-runners-core-java
> Finally, even if these dependencies are added and the typo fixed. When 
> running the runnable on service integration tests, SplittableParDoTest which 
> contains @RunnableOnService tests (part of runners/core-java) doesn't execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[Pei He (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583339#comment-15583339
 ] 

Pei He commented on BEAM-763:
-

BigQueryIOTest doesn't work with RunnableOnService:
https://issues.apache.org/jira/browse/BEAM-436

However, we didn't change it to NeedsRunner when backport PR from Dataflow.

Sent PR to update.

> BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid 
> RunnableOnService test
> --
>
> Key: BEAM-763
> URL: https://issues.apache.org/jira/browse/BEAM-763
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Pei He
>Priority: Minor
>
> TestPipeline.create(options) is not compatible with how TestPipeline 
> functions. This overrides the properties provided by the maven 
> surefire/failsafe profiles setup by the various runners for integration 
> testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-755) beam-runners-core-java NeedsRunner tests not executing

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-755:
-
Summary: beam-runners-core-java NeedsRunner tests not executing  (was: 
beam-runners-core-java RunnableOnService tests not executing)

> beam-runners-core-java NeedsRunner tests not executing
> --
>
> Key: BEAM-755
> URL: https://issues.apache.org/jira/browse/BEAM-755
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Luke Cwik
>Assignee: Kenneth Knowles
>
> org.apache.beam:beam-runners-core-java is not specified as an integration 
> test dependency to scan within runners/pom.xml
> There is also in runners/direct-java/pom.xml where its 
> org.apache.beam:beam-runners-java-core and should be 
> org.apache.beam:beam-runners-core-java
> Finally, even if these dependencies are added and the typo fixed. When 
> running the runnable on service integration tests, SplittableParDoTest which 
> contains @RunnableOnService tests (part of runners/core-java) doesn't execute.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is still unstable: beam_PostCommit_MavenVerify #1553

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1553

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Runners :: Spark #1553

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1553

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583315#comment-15583315
 ] 

ASF GitHub Bot commented on BEAM-763:
-

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1119

[BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to 
NeedsRunner tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
change-bq-tests-to-needsRunners

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1119


commit 3965dbdb184075333086a87448b2f153e66654e2
Author: Pei He 
Date:   2016-10-17T20:13:52Z

[BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to 
NeedsRunner tests.




> BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid 
> RunnableOnService test
> --
>
> Key: BEAM-763
> URL: https://issues.apache.org/jira/browse/BEAM-763
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Pei He
>Priority: Minor
>
> TestPipeline.create(options) is not compatible with how TestPipeline 
> functions. This overrides the properties provided by the maven 
> surefire/failsafe profiles setup by the various runners for integration 
> testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1119: [BEAM-763] Update BigQueryIOTest.testBuil...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/1119

[BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to 
NeedsRunner tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam 
change-bq-tests-to-needsRunners

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1119.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1119


commit 3965dbdb184075333086a87448b2f153e66654e2
Author: Pei He 
Date:   2016-10-17T20:13:52Z

[BEAM-763] Update BigQueryIOTest.testBuildSourceWithTableAndSqlDialect to 
NeedsRunner tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-763) BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is not a valid RunnableOnService test

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-763:
--

 Summary: BigQueryIOTest.testBuildSourceWithTableAndSqlDialect is 
not a valid RunnableOnService test
 Key: BEAM-763
 URL: https://issues.apache.org/jira/browse/BEAM-763
 Project: Beam
  Issue Type: Test
Reporter: Luke Cwik
Assignee: Pei He
Priority: Minor


TestPipeline.create(options) is not compatible with how TestPipeline functions. 
This overrides the properties provided by the maven surefire/failsafe profiles 
setup by the various runners for integration testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-762) Use composition over inheritance in spark StreamingEvaluationContext if two contexts are necessary.

[Pei He (JIRA)]

Pei He created BEAM-762:
---

 Summary: Use composition over inheritance in spark 
StreamingEvaluationContext if two contexts are necessary.
 Key: BEAM-762
 URL: https://issues.apache.org/jira/browse/BEAM-762
 Project: Beam
  Issue Type: Improvement
  Components: runner-spark
Reporter: Pei He
Priority: Minor


After working on PR: https://github.com/apache/incubator-beam/pull/1096 ,
I felt it is easy to forget updating spark streaming context with current 
inheritance.

And, having a single EvaluationContext to support both streaming and batch is 
even better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-762) Use composition over inheritance in spark StreamingEvaluationContext if two contexts are necessary.

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He updated BEAM-762:

Labels: starter  (was: )

> Use composition over inheritance in spark StreamingEvaluationContext if two 
> contexts are necessary.
> ---
>
> Key: BEAM-762
> URL: https://issues.apache.org/jira/browse/BEAM-762
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Reporter: Pei He
>Priority: Minor
>  Labels: starter
>
> After working on PR: https://github.com/apache/incubator-beam/pull/1096 ,
> I felt it is easy to forget updating spark streaming context with current 
> inheritance.
> And, having a single EvaluationContext to support both streaming and batch is 
> even better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-761) SplittableParDoTest fails

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-761.

   Resolution: Fixed
 Assignee: Ben Chambers
Fix Version/s: 0.3.0-incubating

> SplittableParDoTest fails
> -
>
> Key: BEAM-761
> URL: https://issues.apache.org/jira/browse/BEAM-761
> Project: Beam
>  Issue Type: Test
>Reporter: Luke Cwik
>Assignee: Ben Chambers
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Coder propagation was missing in GBKIntoKeyedWorkItems
> Fixed with https://github.com/apache/incubator-beam/pull/1117/files
> Filed for completeness.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-761) SplittableParDoTest fails

[Luke Cwik (JIRA)]

Luke Cwik created BEAM-761:
--

 Summary: SplittableParDoTest fails
 Key: BEAM-761
 URL: https://issues.apache.org/jira/browse/BEAM-761
 Project: Beam
  Issue Type: Test
Reporter: Luke Cwik
Priority: Minor


Coder propagation was missing in GBKIntoKeyedWorkItems

Fixed with https://github.com/apache/incubator-beam/pull/1117/files

Filed for completeness.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1118: Restore ReshuffleTriggerTest to SDK

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1118

Restore ReshuffleTriggerTest to SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @lukecwik 

This restore was missed in a64acb2f from #1083, where all prior trigger 
files were restored for compatibility. We have some internal dependencies 
lingering.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
ReshuffleTriggerTest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1118


commit 45d3c1ad2122ba335b1f8279d95e79cd7df2661d
Author: Kenneth Knowles 
Date:   2016-10-17T19:44:29Z

Restore ReshuffleTriggerTest to SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1117

[kenn]

This closes #1117


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e849d95d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e849d95d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e849d95d

Branch: refs/heads/master
Commit: e849d95d1b5175c4e758345be278644a38769910
Parents: 5747951 577d04a
Author: Kenneth Knowles 
Authored: Mon Oct 17 12:40:11 2016 -0700
Committer: Kenneth Knowles 
Committed: Mon Oct 17 12:40:11 2016 -0700

--
 .../beam/runners/core/GBKIntoKeyedWorkItems.java   | 17 -
 .../beam/runners/core/SplittableParDoTest.java | 10 +++---
 2 files changed, 23 insertions(+), 4 deletions(-)
--

[GitHub] incubator-beam pull request #1117: Fix SplittableParDoTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1117


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Fix SplittableParDoTest

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 5747951ff -> e849d95d1


Fix SplittableParDoTest

This required fixing GBKIntoKeyedWorkItems to properly set the coder on
the primitive, and updating the assertions to match the actual (and
correct) behavior.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/577d04ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/577d04ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/577d04ab

Branch: refs/heads/master
Commit: 577d04ab0799b18d5c2c88e2250859678f589968
Parents: b8e6eea
Author: bchambers 
Authored: Mon Oct 17 12:35:03 2016 -0700
Committer: bchambers 
Committed: Mon Oct 17 12:35:03 2016 -0700

--
 .../beam/runners/core/GBKIntoKeyedWorkItems.java   | 17 -
 .../beam/runners/core/SplittableParDoTest.java | 10 +++---
 2 files changed, 23 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/577d04ab/runners/core-java/src/main/java/org/apache/beam/runners/core/GBKIntoKeyedWorkItems.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GBKIntoKeyedWorkItems.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GBKIntoKeyedWorkItems.java
index ca4d681..304e349 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/GBKIntoKeyedWorkItems.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/GBKIntoKeyedWorkItems.java
@@ -17,10 +17,15 @@
  */
 package org.apache.beam.runners.core;
 
+import static com.google.common.base.Preconditions.checkArgument;
+
 import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.KvCoder;
 import org.apache.beam.sdk.transforms.GroupByKey;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.KeyedWorkItem;
+import org.apache.beam.sdk.util.KeyedWorkItemCoder;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
@@ -34,7 +39,17 @@ public class GBKIntoKeyedWorkItems
 extends PTransform>, 
PCollection>> {
   @Override
   public PCollection> apply(PCollection> input) {
-return PCollection.createPrimitiveOutputInternal(
+checkArgument(input.getCoder() instanceof KvCoder,
+"Expected input coder to be KvCoder, but was %s",
+input.getCoder().getClass().getSimpleName());
+
+KvCoder kvCoder = (KvCoder) input.getCoder();
+Coder> coder = KeyedWorkItemCoder.of(
+kvCoder.getKeyCoder(), kvCoder.getValueCoder(),
+input.getWindowingStrategy().getWindowFn().windowCoder());
+PCollection> collection = 
PCollection.createPrimitiveOutputInternal(
 input.getPipeline(), input.getWindowingStrategy(), input.isBounded());
+collection.setCoder((Coder) coder);
+return collection;
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/577d04ab/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
--
diff --git 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
index a76c4da..b7cdc64 100644
--- 
a/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
+++ 
b/runners/core-java/src/test/java/org/apache/beam/runners/core/SplittableParDoTest.java
@@ -128,12 +128,14 @@ public class SplittableParDoTest {
 Pipeline pipeline = TestPipeline.create();
 DoFn boundedFn = new BoundedFakeFn();
 assertEquals(
+"Applying a bounded SDF to a bounded collection produces a bounded 
collection",
 PCollection.IsBounded.BOUNDED,
 makeBoundedCollection(pipeline)
 .apply("bounded to bounded", new SplittableParDo<>(boundedFn))
 .isBounded());
 assertEquals(
-PCollection.IsBounded.BOUNDED,
+"Applying a bounded SDF to an unbounded collection produces an 
unbounded collection",
+PCollection.IsBounded.UNBOUNDED,
 makeUnboundedCollection(pipeline)
 .apply("bounded to unbounded", new SplittableParDo<>(boundedFn))
 .isBounded());
@@ -145,12 +147,14 @@ public class SplittableParDoTest {
 Pipeline pipeline = TestPipeline.create();
 DoFn unboundedFn = 

[GitHub] incubator-beam pull request #1117: Fix SplittableParDoTest

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/incubator-beam/pull/1117

Fix SplittableParDoTest

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This required fixing GBKIntoKeyedWorkItems to properly set the coder on
the primitive, and updating the assertions to match the actual (and
correct) behavior.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/incubator-beam fix-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1117


commit 577d04ab0799b18d5c2c88e2250859678f589968
Author: bchambers 
Date:   2016-10-17T19:35:03Z

Fix SplittableParDoTest

This required fixing GBKIntoKeyedWorkItems to properly set the coder on
the primitive, and updating the assertions to match the actual (and
correct) behavior.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-469.
--
Resolution: Fixed

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-469:
-
Fix Version/s: (was: Not applicable)
   0.3.0-incubating

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583205#comment-15583205
 ] 

Daniel Halperin commented on BEAM-469:
--

Backwards compatibility: this is a change to the encoding of data, which will 
only affect Cloud Dataflow streaming pipelines across update. This is not yet 
part of the Beam model, and no one in Beam depends on it.

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: 0.3.0-incubating
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-469:
-
Labels: backward-incompatible  (was: )

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
>  Labels: backward-incompatible
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-469) NullableCoder optimized encoding via passthrough context

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-469:
-
Summary: NullableCoder optimized encoding via passthrough context  (was: 
NullableCoder should encode using the context given and not always use the 
nested context)

> NullableCoder optimized encoding via passthrough context
> 
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-469) NullableCoder should encode using the context given and not always use the nested context

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583194#comment-15583194
 ] 

ASF GitHub Bot commented on BEAM-469:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1104


> NullableCoder should encode using the context given and not always use the 
> nested context
> -
>
> Key: BEAM-469
> URL: https://issues.apache.org/jira/browse/BEAM-469
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Thomas Groh
>Priority: Trivial
> Fix For: Not applicable
>
>
> NullableCoder should encode using the context given and not always use the 
> nested context. For coders which can efficiently encode in the outer context 
> such as StringUtf8Coder or ByteArrayCoder, we are forcing them to prefix 
> themselves with their length.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1104: [BEAM-469] Remove Remaining Nested Contex...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---