Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1517

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_MavenVerify #1517

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-701) Support Avro Compression for writes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570442#comment-15570442
 ] 

ASF GitHub Bot commented on BEAM-701:
-

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1092

[BEAM-701] AvroIO.Write: minor doc and test fixes

* Document the default codec.
* Swap the order of assertEquals in tests to match the expected pattern 
(expected, actual).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
tweak-avroio-compressed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1092.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1092


commit 2f593dea39cf818992b6285641ba601fe14425e1
Author: Dan Halperin 
Date:   2016-10-13T01:10:25Z

AvroIO.Write: minor doc and test fixes

* Document the defualt codec.
* Swap the order of assertEquals in tests to match the expected pattern 
(expected, actual).




> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: features
> Fix For: 0.3.0-incubating
>
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1092: [BEAM-701] AvroIO.Write: minor doc and te...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/incubator-beam/pull/1092

[BEAM-701] AvroIO.Write: minor doc and test fixes

* Document the default codec.
* Swap the order of assertEquals in tests to match the expected pattern 
(expected, actual).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/incubator-beam 
tweak-avroio-compressed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1092.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1092


commit 2f593dea39cf818992b6285641ba601fe14425e1
Author: Dan Halperin 
Date:   2016-10-13T01:10:25Z

AvroIO.Write: minor doc and test fixes

* Document the defualt codec.
* Swap the order of assertEquals in tests to match the expected pattern 
(expected, actual).




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-746) Stop using the Streaming Flag in BigQueryIO.Write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570420#comment-15570420
 ] 

ASF GitHub Bot commented on BEAM-746:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1091

[BEAM-746] Choose BigQuery Write implementation based on Input PCollection

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Stop using PipelineOptions, and instead use the boundedness of the input
to choose how to write to BigQuery. This means that runners that don't
use the streaming flag still function appropriately.

Fixes BEAM-746

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam bqwrite_unbounded

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1091


commit 48dceca7cb7affaade60147f20e1913bbdee
Author: Thomas Groh 
Date:   2016-10-13T00:58:57Z

Choose BigQuery Write implementation based on Input PCollection

Stop using PipelineOptions, and instead use the boundedness of the input
to choose how to write to BigQuery. This means that runners that don't
use the streaming flag still function appropriately.

Fixes BEAM-746




> Stop using the Streaming Flag in BigQueryIO.Write
> -
>
> Key: BEAM-746
> URL: https://issues.apache.org/jira/browse/BEAM-746
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Minor
>
> The choice of write PTransform should be based on the boundedness of the 
> input PCollection, not a PipelineOption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1091: [BEAM-746] Choose BigQuery Write implemen...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1091

[BEAM-746] Choose BigQuery Write implementation based on Input PCollection

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Stop using PipelineOptions, and instead use the boundedness of the input
to choose how to write to BigQuery. This means that runners that don't
use the streaming flag still function appropriately.

Fixes BEAM-746

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam bqwrite_unbounded

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1091


commit 48dceca7cb7affaade60147f20e1913bbdee
Author: Thomas Groh 
Date:   2016-10-13T00:58:57Z

Choose BigQuery Write implementation based on Input PCollection

Stop using PipelineOptions, and instead use the boundedness of the input
to choose how to write to BigQuery. This means that runners that don't
use the streaming flag still function appropriately.

Fixes BEAM-746




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-746) Stop using the Streaming Flag in BigQueryIO.Write

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-746:


 Summary: Stop using the Streaming Flag in BigQueryIO.Write
 Key: BEAM-746
 URL: https://issues.apache.org/jira/browse/BEAM-746
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Thomas Groh
Assignee: Thomas Groh
Priority: Minor


The choice of write PTransform should be based on the boundedness of the input 
PCollection, not a PipelineOption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build became unstable: beam_PostCommit_MavenVerify #1513

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1513

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-65) SplittableDoFn

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570384#comment-15570384
 ] 

ASF GitHub Bot commented on BEAM-65:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/896


> SplittableDoFn
> --
>
> Key: BEAM-65
> URL: https://issues.apache.org/jira/browse/BEAM-65
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Daniel Halperin
>Assignee: Eugene Kirpichov
>Priority: Minor
>
> SplittableDoFn is a proposed enhancement for "dynamically splittable work" to 
> the Beam model.
> Among other things, it would allow a unified implementation of 
> bounded/unbounded sources with dynamic work rebalancing and the ability to 
> express multiple scalable steps (e.g., global expansion -> file sizing & 
> parsing -> splitting files into independently-processable blocks) via 
> composition rather than inheritance.
> This would make it much easier to implement many types of sources, to modify 
> and reuse existing sources. Also, it would improve scalability of the Beam 
> model by moving things like splitting a source from the control plane (where 
> it is today -- glob -> List sent over service APIs) into the 
> data plane (PCollection -> PCollection -> ...).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/4] incubator-beam git commit: [BEAM-65] SplittableDoFn prototype.

[bchambers]

[BEAM-65] SplittableDoFn prototype.

Work in progress. Currently only runs in direct runner,
and not ready for any use by real users.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a0a24883
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a0a24883
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a0a24883

Branch: refs/heads/master
Commit: a0a24883737850052f54290255780e868c0b63dc
Parents: a5d1293
Author: Eugene Kirpichov 
Authored: Thu Aug 11 17:13:53 2016 -0700
Committer: bchambers 
Committed: Wed Oct 12 17:29:20 2016 -0700

--
 runners/core-java/pom.xml   |   6 +
 .../runners/core/ElementAndRestriction.java |  42 ++
 .../core/ElementAndRestrictionCoder.java|  67 +++
 .../runners/core/GBKIntoKeyedWorkItems.java |  40 ++
 .../beam/runners/core/SplittableParDo.java  | 469 
 .../core/ElementAndRestrictionCoderTest.java| 127 +
 .../beam/runners/core/SplittableParDoTest.java  | 467 
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |  66 +++
 .../beam/runners/direct/DirectRunner.java   |   5 +
 .../runners/direct/ParDoOverrideFactory.java|  55 ++
 .../beam/runners/direct/SplittableDoFnTest.java | 225 
 .../beam/sdk/annotations/Experimental.java  |   8 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 218 +++-
 .../beam/sdk/transforms/DoFnAdapters.java   |  27 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  | 117 ++--
 .../org/apache/beam/sdk/transforms/ParDo.java   |  19 +
 .../sdk/transforms/reflect/DoFnInvoker.java |  26 +-
 .../sdk/transforms/reflect/DoFnInvokers.java| 179 +-
 .../sdk/transforms/reflect/DoFnSignature.java   | 114 +++-
 .../sdk/transforms/reflect/DoFnSignatures.java  | 366 -
 .../splittabledofn/RestrictionTracker.java  |  42 ++
 .../transforms/splittabledofn/package-info.java |  22 +
 .../org/apache/beam/sdk/coders/KvCoderTest.java |  99 ++--
 .../apache/beam/sdk/transforms/ParDoTest.java   |  46 ++
 .../transforms/reflect/DoFnInvokersTest.java| 261 -
 .../DoFnSignaturesProcessElementTest.java   |   4 +-
 .../DoFnSignaturesSplittableDoFnTest.java   | 543 +++
 27 files changed, 3495 insertions(+), 165 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a0a24883/runners/core-java/pom.xml
--
diff --git a/runners/core-java/pom.xml b/runners/core-java/pom.xml
index d958dd2..d84c420 100644
--- a/runners/core-java/pom.xml
+++ b/runners/core-java/pom.xml
@@ -190,6 +190,12 @@
 
 
 
+  com.google.auto.value
+  auto-value
+  provided
+
+
+
   com.google.code.findbugs
   annotations
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a0a24883/runners/core-java/src/main/java/org/apache/beam/runners/core/ElementAndRestriction.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/ElementAndRestriction.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ElementAndRestriction.java
new file mode 100644
index 000..4a5d0c4
--- /dev/null
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/ElementAndRestriction.java
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.transforms.DoFn;
+
+/**
+ * A tuple of an element and a restriction applied to processing it with a
+ * https://s.apache.org/splittable-do-fn";>splittable {@link DoFn}.
+ */
+@Experimental(Experimental.Kind.SPLITTABLE_DO_FN)
+@AutoValue
+public abstract class ElementAndRestriction {
+  /** The element to process. */
+  public abstract ElementT element();
+
+  /** The restriction applied to processing the element. *

[GitHub] incubator-beam pull request #896: [BEAM-65] SplittableDoFn prototype.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/896


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/4] incubator-beam git commit: [BEAM-65] SplittableDoFn prototype.

[bchambers]

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a0a24883/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
index 3eee74a..f671a67 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnAdapters.java
@@ -25,6 +25,7 @@ import org.apache.beam.sdk.transforms.reflect.DoFnInvoker;
 import org.apache.beam.sdk.transforms.reflect.DoFnInvokers;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignature;
 import org.apache.beam.sdk.transforms.reflect.DoFnSignatures;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.transforms.windowing.PaneInfo;
 import org.apache.beam.sdk.util.WindowingInternals;
@@ -37,8 +38,8 @@ import org.joda.time.Instant;
 /**
  * Utility class containing adapters to/from {@link DoFn} and {@link OldDoFn}.
  *
- * @deprecated This class will go away when we start running {@link DoFn}'s 
directly (using
- * {@link DoFnInvoker}) rather than via {@link OldDoFn}.
+ * @deprecated This class will go away when we start running {@link DoFn}'s 
directly (using {@link
+ * DoFnInvoker}) rather than via {@link OldDoFn}.
  */
 @Deprecated
 public class DoFnAdapters {
@@ -176,6 +177,18 @@ public class DoFnAdapters {
   }
 
   /**
+   * If the fn was created using {@link #toOldDoFn}, returns the original 
{@link DoFn}. Otherwise,
+   * returns {@code null}.
+   */
+  public static  DoFn 
getDoFn(OldDoFn fn) {
+if (fn instanceof SimpleDoFnAdapter) {
+  return ((SimpleDoFnAdapter) fn).fn;
+} else {
+  return null;
+}
+  }
+
+  /**
* Wraps a {@link DoFn} that doesn't require access to {@link BoundedWindow} 
as an {@link
* OldDoFn}.
*/
@@ -324,6 +337,11 @@ public class DoFnAdapters {
 public DoFn.OutputReceiver outputReceiver() {
   throw new UnsupportedOperationException("outputReceiver() exists only 
for testing");
 }
+
+@Override
+public  RestrictionTracker 
restrictionTracker() {
+  throw new UnsupportedOperationException("This is a non-splittable DoFn");
+}
   }
 
   /**
@@ -412,5 +430,10 @@ public class DoFnAdapters {
 public DoFn.OutputReceiver outputReceiver() {
   throw new UnsupportedOperationException("outputReceiver() exists only 
for testing");
 }
+
+@Override
+public  RestrictionTracker 
restrictionTracker() {
+  throw new UnsupportedOperationException("This is a non-splittable DoFn");
+}
   }
 }

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a0a24883/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
index 11a4cbd..302bb02 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFnTester.java
@@ -46,7 +46,9 @@ import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingInternals;
 import org.apache.beam.sdk.util.state.InMemoryStateInternals;
+import org.apache.beam.sdk.util.state.InMemoryTimerInternals;
 import org.apache.beam.sdk.util.state.StateInternals;
+import org.apache.beam.sdk.util.state.TimerCallback;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TimestampedValue;
 import org.apache.beam.sdk.values.TupleTag;
@@ -222,8 +224,11 @@ public class DoFnTester implements 
AutoCloseable {
 if (state == State.UNINITIALIZED) {
   initializeState();
 }
-TestContext context = createContext(fn);
+TestContext context = createContext(fn);
 context.setupDelegateAggregators();
+// State and timer internals are per-bundle.
+stateInternals = InMemoryStateInternals.forKey(new Object());
+timerInternals = new InMemoryTimerInternals();
 try {
   fn.startBundle(context);
 } catch (UserCodeException e) {
@@ -460,6 +465,35 @@ public class DoFnTester implements 
AutoCloseable {
 return extractAggregatorValue(agg.getName(), agg.getCombineFn());
   }
 
+  private static TimerCallback collectInto(final 
List firedTimers) {
+return new TimerCallback() {
+  @Override
+  public void onTimer(TimerInternals.TimerData timer) throws Exception {
+firedTimers.add(timer);
+  }
+};
+  }
+
+  public List advanceInputWatermark(Instant 
newWatermark) {
+try {
+  fin

[1/4] incubator-beam git commit: [BEAM-65] SplittableDoFn prototype.

[bchambers]

Repository: incubator-beam
Updated Branches:
  refs/heads/master a5d129361 -> 13b45895e


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/a0a24883/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java
index d057765..0bfe2be 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnInvokersTest.java
@@ -17,14 +17,32 @@
  */
 package org.apache.beam.sdk.transforms.reflect;
 
+import static org.hamcrest.CoreMatchers.instanceOf;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Matchers.same;
 import static org.mockito.Mockito.any;
+import static org.mockito.Mockito.doAnswer;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
-
+import static org.mockito.Mockito.when;
+
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.coders.CustomCoder;
 import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.DoFn.GetInitialRestriction;
+import org.apache.beam.sdk.transforms.DoFn.ProcessContinuation;
 import org.apache.beam.sdk.transforms.OldDoFn;
 import 
org.apache.beam.sdk.transforms.reflect.testhelper.DoFnInvokersTestHelper;
+import org.apache.beam.sdk.transforms.splittabledofn.RestrictionTracker;
 import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
 import org.apache.beam.sdk.util.UserCodeException;
 import org.apache.beam.sdk.util.WindowingInternals;
@@ -34,7 +52,9 @@ import org.junit.Test;
 import org.junit.rules.ExpectedException;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
+import org.mockito.AdditionalAnswers;
 import org.mockito.Mock;
+import org.mockito.Mockito;
 import org.mockito.MockitoAnnotations;
 
 /** Tests for {@link DoFnInvokers}. */
@@ -76,11 +96,16 @@ public class DoFnInvokersTest {
   public WindowingInternals windowingInternals() {
 return mockWindowingInternals;
   }
+
+  @Override
+  public  RestrictionTracker 
restrictionTracker() {
+return null;
+  }
 };
   }
 
-  private void invokeProcessElement(DoFn fn) {
-DoFnInvokers.INSTANCE
+  private ProcessContinuation invokeProcessElement(DoFn fn) {
+return DoFnInvokers.INSTANCE
 .newByteBuddyInvoker(fn)
 .invokeProcessElement(mockContext, extraContextFactory);
   }
@@ -106,9 +131,9 @@ public class DoFnInvokersTest {
   @ProcessElement
   public void processElement(ProcessContext c) throws Exception {}
 }
-MockFn fn = mock(MockFn.class);
-invokeProcessElement(fn);
-verify(fn).processElement(mockContext);
+MockFn mockFn = mock(MockFn.class);
+assertEquals(ProcessContinuation.stop(), invokeProcessElement(mockFn));
+verify(mockFn).processElement(mockContext);
   }
 
   interface InterfaceWithProcessElement {
@@ -128,7 +153,7 @@ public class DoFnInvokersTest {
   public void testDoFnWithProcessElementInterface() throws Exception {
 IdentityUsingInterfaceWithProcessElement fn =
 mock(IdentityUsingInterfaceWithProcessElement.class);
-invokeProcessElement(fn);
+assertEquals(ProcessContinuation.stop(), invokeProcessElement(fn));
 verify(fn).processElement(mockContext);
   }
 
@@ -149,14 +174,14 @@ public class DoFnInvokersTest {
   @Test
   public void testDoFnWithMethodInSuperclass() throws Exception {
 IdentityChildWithoutOverride fn = mock(IdentityChildWithoutOverride.class);
-invokeProcessElement(fn);
+assertEquals(ProcessContinuation.stop(), invokeProcessElement(fn));
 verify(fn).process(mockContext);
   }
 
   @Test
   public void testDoFnWithMethodInSubclass() throws Exception {
 IdentityChildWithOverride fn = mock(IdentityChildWithOverride.class);
-invokeProcessElement(fn);
+assertEquals(ProcessContinuation.stop(), invokeProcessElement(fn));
 verify(fn).process(mockContext);
   }
 
@@ -167,7 +192,7 @@ public class DoFnInvokersTest {
   public void processElement(ProcessContext c, BoundedWindow w) throws 
Exception {}
 }
 MockFn fn = mock(MockFn.class);
-invokeProcessElement(fn);
+assertEquals(ProcessContinuation.stop(), invokeProcessElement(fn));
 verify(fn).processElement(mockContext, mockWindow);
   }
 
@@ -178,7 +203,7 @@ public class DoFnInvokersTest {
   p

[4/4] incubator-beam git commit: Closes #896

[bchambers]

Closes #896


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/13b45895
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/13b45895
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/13b45895

Branch: refs/heads/master
Commit: 13b45895eb13cd48557472463404bd097d7097d7
Parents: a5d1293 a0a2488
Author: bchambers 
Authored: Wed Oct 12 17:29:21 2016 -0700
Committer: bchambers 
Committed: Wed Oct 12 17:29:21 2016 -0700

--
 runners/core-java/pom.xml   |   6 +
 .../runners/core/ElementAndRestriction.java |  42 ++
 .../core/ElementAndRestrictionCoder.java|  67 +++
 .../runners/core/GBKIntoKeyedWorkItems.java |  40 ++
 .../beam/runners/core/SplittableParDo.java  | 469 
 .../core/ElementAndRestrictionCoderTest.java| 127 +
 .../beam/runners/core/SplittableParDoTest.java  | 467 
 ...ectGBKIntoKeyedWorkItemsOverrideFactory.java |  66 +++
 .../beam/runners/direct/DirectRunner.java   |   5 +
 .../runners/direct/ParDoOverrideFactory.java|  55 ++
 .../beam/runners/direct/SplittableDoFnTest.java | 225 
 .../beam/sdk/annotations/Experimental.java  |   8 +-
 .../org/apache/beam/sdk/transforms/DoFn.java| 218 +++-
 .../beam/sdk/transforms/DoFnAdapters.java   |  27 +-
 .../apache/beam/sdk/transforms/DoFnTester.java  | 117 ++--
 .../org/apache/beam/sdk/transforms/ParDo.java   |  19 +
 .../sdk/transforms/reflect/DoFnInvoker.java |  26 +-
 .../sdk/transforms/reflect/DoFnInvokers.java| 179 +-
 .../sdk/transforms/reflect/DoFnSignature.java   | 114 +++-
 .../sdk/transforms/reflect/DoFnSignatures.java  | 366 -
 .../splittabledofn/RestrictionTracker.java  |  42 ++
 .../transforms/splittabledofn/package-info.java |  22 +
 .../org/apache/beam/sdk/coders/KvCoderTest.java |  99 ++--
 .../apache/beam/sdk/transforms/ParDoTest.java   |  46 ++
 .../transforms/reflect/DoFnInvokersTest.java| 261 -
 .../DoFnSignaturesProcessElementTest.java   |   4 +-
 .../DoFnSignaturesSplittableDoFnTest.java   | 543 +++
 27 files changed, 3495 insertions(+), 165 deletions(-)
--

[jira] [Closed] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-714.

   Resolution: Fixed
Fix Version/s: Not applicable

> Migrate DatastoreV1 to use AutoValue to reduce boilerplate
> --
>
> Key: BEAM-714
> URL: https://issues.apache.org/jira/browse/BEAM-714
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Eugene Kirpichov
>Priority: Minor
>  Labels: io, simple, starter
> Fix For: Not applicable
>
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570357#comment-15570357
 ] 

Eugene Kirpichov commented on BEAM-714:
---

Should be closed.

> Migrate DatastoreV1 to use AutoValue to reduce boilerplate
> --
>
> Key: BEAM-714
> URL: https://issues.apache.org/jira/browse/BEAM-714
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Eugene Kirpichov
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-714:
-

Assignee: Eugene Kirpichov  (was: Daniel Halperin)

> Migrate DatastoreV1 to use AutoValue to reduce boilerplate
> --
>
> Key: BEAM-714
> URL: https://issues.apache.org/jira/browse/BEAM-714
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Eugene Kirpichov
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-714) Migrate DatastoreV1 to use AutoValue to reduce boilerplate

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570332#comment-15570332
 ] 

ASF GitHub Bot commented on BEAM-714:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1031


> Migrate DatastoreV1 to use AutoValue to reduce boilerplate
> --
>
> Key: BEAM-714
> URL: https://issues.apache.org/jira/browse/BEAM-714
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Reporter: Luke Cwik
>Assignee: Daniel Halperin
>Priority: Minor
>  Labels: io, simple, starter
>
> Use the AutoValue functionality to reduce boilerplate.
> See this PR for an example:
> https://github.com/apache/incubator-beam/pull/1054



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-684) De-Dataflow Beam

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570325#comment-15570325
 ] 

ASF GitHub Bot commented on BEAM-684:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1016


> De-Dataflow Beam
> 
>
> Key: BEAM-684
> URL: https://issues.apache.org/jira/browse/BEAM-684
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>  Labels: easy, easyfix, starter
>
> {code}
> grep -i dataflow sdks/java/core/**/*.java | wc -l
>  136
> {code}
> Most of those uses of the word {{dataflow}} are wrong. We should really 
> finish converting over to Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1031: [BEAM-714] Converts Datastore to use Auto...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1031


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Closes #1031

[dhalperi]

Closes #1031


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/a5d12936
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/a5d12936
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/a5d12936

Branch: refs/heads/master
Commit: a5d12936158f5725a12a7f1f23eb90f86c4e68b1
Parents: e2868d6 251e3e1
Author: Dan Halperin 
Authored: Wed Oct 12 17:12:33 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 17:12:33 2016 -0700

--
 sdks/java/io/google-cloud-platform/pom.xml  |  6 ++
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  | 95 +++-
 2 files changed, 38 insertions(+), 63 deletions(-)
--

[GitHub] incubator-beam pull request #1016: [BEAM-684] pom.xml: replace dataflow.java...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1016


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Converts Datastore to use AutoValue

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master e2868d687 -> a5d129361


Converts Datastore to use AutoValue


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/251e3e1e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/251e3e1e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/251e3e1e

Branch: refs/heads/master
Commit: 251e3e1ef2868b9901be3dbd27793b613855704a
Parents: e2868d6
Author: Eugene Kirpichov 
Authored: Wed Sep 28 17:29:11 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 17:12:32 2016 -0700

--
 sdks/java/io/google-cloud-platform/pom.xml  |  6 ++
 .../beam/sdk/io/gcp/datastore/DatastoreV1.java  | 95 +++-
 2 files changed, 38 insertions(+), 63 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/251e3e1e/sdks/java/io/google-cloud-platform/pom.xml
--
diff --git a/sdks/java/io/google-cloud-platform/pom.xml 
b/sdks/java/io/google-cloud-platform/pom.xml
index 15c0447..1a598bb 100644
--- a/sdks/java/io/google-cloud-platform/pom.xml
+++ b/sdks/java/io/google-cloud-platform/pom.xml
@@ -213,6 +213,12 @@
   avro
 
 
+
+  com.google.auto.value
+  auto-value
+  provided
+
+
 
 
   org.apache.beam

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/251e3e1e/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
index 45b2d6f..bfbff32 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java
@@ -35,6 +35,7 @@ import com.google.api.client.auth.oauth2.Credential;
 import com.google.api.client.util.BackOff;
 import com.google.api.client.util.BackOffUtils;
 import com.google.api.client.util.Sleeper;
+import com.google.auto.value.AutoValue;
 import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.MoreObjects;
 import com.google.common.collect.ImmutableList;
@@ -194,7 +195,7 @@ public class DatastoreV1 {
* {@link DatastoreV1.Read#withNamespace}, {@link 
DatastoreV1.Read#withNumQuerySplits}.
*/
   public DatastoreV1.Read read() {
-return new DatastoreV1.Read(null, null, null, 0);
+return new 
AutoValue_DatastoreV1_Read.Builder().setNumQuerySplits(0).build();
   }
 
   /**
@@ -203,7 +204,8 @@ public class DatastoreV1 {
*
* @see DatastoreIO
*/
-  public static class Read extends PTransform> {
+  @AutoValue
+  public abstract static class Read extends PTransform> {
 private static final Logger LOG = LoggerFactory.getLogger(Read.class);
 
 /** An upper bound on the number of splits for a query. */
@@ -222,16 +224,23 @@ public class DatastoreV1 {
  */
 static final int QUERY_BATCH_LIMIT = 500;
 
-@Nullable
-private final String projectId;
+@Nullable public abstract String getProjectId();
+@Nullable public abstract Query getQuery();
+@Nullable public abstract String getNamespace();
+public abstract int getNumQuerySplits();
 
-@Nullable
-private final Query query;
+public abstract String toString();
 
-@Nullable
-private final String namespace;
+abstract Builder toBuilder();
 
-private final int numQuerySplits;
+@AutoValue.Builder
+abstract static class Builder {
+  abstract Builder setProjectId(String projectId);
+  abstract Builder setQuery(Query query);
+  abstract Builder setNamespace(String namespace);
+  abstract Builder setNumQuerySplits(int numQuerySplits);
+  abstract Read build();
+}
 
 /**
  * Computes the number of splits to be performed on the given query by 
querying the estimated
@@ -345,25 +354,12 @@ public class DatastoreV1 {
 }
 
 /**
- * Note that only {@code namespace} is really {@code @Nullable}. The other 
parameters may be
- * {@code null} as a matter of build order, but if they are {@code null} 
at instantiation time,
- * an error will be thrown.
- */
-private Read(@Nullable String projectId, @Nullable Query query, @Nullable 
String namespace,
-int numQuerySplits) {
-  this.projectId = projectId;
-  this.query = query;
-  this.namespace = namespace;
-  this.numQuerySplits = numQuerySplits;
-}
-
-/**
  * Returns a new {@link DatastoreV1.Read}

incubator-beam git commit: Closes #1016

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c561cd9c1 -> e2868d687


Closes #1016


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/e2868d68
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/e2868d68
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/e2868d68

Branch: refs/heads/master
Commit: e2868d6874f1a0dd5971328f84049277001ef412
Parents: 4d10f86 c561cd9
Author: Dan Halperin 
Authored: Wed Oct 12 17:11:05 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 17:11:05 2016 -0700

--
 examples/java/pom.xml |  2 +-
 pom.xml   |  4 ++--
 runners/google-cloud-dataflow-java/pom.xml|  2 +-
 sdks/java/core/pom.xml|  2 +-
 .../java/org/apache/beam/sdk/coders/Coder.java|  2 +-
 .../org/apache/beam/sdk/coders/package-info.java  |  2 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java | 18 +-
 .../main/java/org/apache/beam/sdk/io/Sink.java|  2 +-
 .../main/java/org/apache/beam/sdk/io/TextIO.java  | 15 ++-
 .../java/org/apache/beam/sdk/io/XmlSource.java| 13 +++--
 .../apache/beam/sdk/io/range/ByteKeyRange.java|  6 +++---
 .../java/org/apache/beam/sdk/package-info.java|  2 +-
 .../org/apache/beam/sdk/runners/package-info.java | 17 +++--
 .../org/apache/beam/sdk/testing/package-info.java |  4 ++--
 .../org/apache/beam/sdk/transforms/OldDoFn.java   | 15 ---
 .../apache/beam/sdk/transforms/PTransform.java|  2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java |  7 +++
 .../org/apache/beam/sdk/util/ReleaseInfo.java |  6 +++---
 .../org/apache/beam/sdk/values/package-info.java  |  4 ++--
 19 files changed, 52 insertions(+), 73 deletions(-)
--

[GitHub] incubator-beam pull request #1090: [BEAM-737][BEAM-738] Updates source API d...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/1090

[BEAM-737][BEAM-738] Updates source API documentation to mention that 
sources should be immutable and updates existing sources accordingly

Updates source API documentation to mention that source objects should not 
be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentrancy tests for text and Avro sources.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
update_boundedsource_stateless

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1090.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1090


commit 5bd56f35d597b142463b1a141e2ac219e6902fc3
Author: Chamikara Jayalath 
Date:   2016-10-12T23:51:20Z

Updates source API documentation to mention that source objects should not 
be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentracy tests for text and Avro sources.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-737) Update TextSource to not maintain local state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570306#comment-15570306
 ] 

ASF GitHub Bot commented on BEAM-737:
-

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/1090

[BEAM-737][BEAM-738] Updates source API documentation to mention that 
sources should be immutable and updates existing sources accordingly

Updates source API documentation to mention that source objects should not 
be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentrancy tests for text and Avro sources.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam 
update_boundedsource_stateless

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1090.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1090


commit 5bd56f35d597b142463b1a141e2ac219e6902fc3
Author: Chamikara Jayalath 
Date:   2016-10-12T23:51:20Z

Updates source API documentation to mention that source objects should not 
be mutated.

Updates  textio._TextSource so that it does not get mutated while reading.

Updates source_test_utils so that sources objects do not get cloned while 
testing. This could help to catch sources that erroneously get modified while 
reading.

Adds reentracy tests for text and Avro sources.




> Update TextSource to not maintain local state
> -
>
> Key: BEAM-737
> URL: https://issues.apache.org/jira/browse/BEAM-737
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently, TextSource [1] maintains a temporary buffer as local state. This 
> could be problematic if a runner implementation tries to pickle the source. 
> We could update TextSource to not maintain any temporary state.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/textio.py#L32



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-742) Move trigger state machines to runners-core, convert triggers to AST

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570036#comment-15570036
 ] 

ASF GitHub Bot commented on BEAM-742:
-

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1089

[BEAM-742] Replicate triggers to runners-core, favoring commit history over 
blame

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

This is identical to #1083 but with the first and third commit squashed 
together. Based on my readings on the subject, I would have expected git to 
actually track the content, at least when using blame, but it does not seem to.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
TriggerReactor-squashed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1089


commit f63f81e2d8fad16d668809c8b291256ae5477ab1
Author: Kenneth Knowles 
Date:   2016-06-24T03:05:27Z

Copy triggers to runners-core

This commit is effectively a "git hint" that all the files
in runners-core are moved from prior files. The moved files
will be replaced.

commit f0913147af072cda06a2da4ef6933e311b94dd90
Author: Kenneth Knowles 
Date:   2016-06-24T03:31:10Z

Rename trigger reactors




> Move trigger state machines to runners-core, convert triggers to AST
> 
>
> Key: BEAM-742
> URL: https://issues.apache.org/jira/browse/BEAM-742
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1089: [BEAM-742] Replicate triggers to runners-...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1089

[BEAM-742] Replicate triggers to runners-core, favoring commit history over 
blame

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @bjchambers 

This is identical to #1083 but with the first and third commit squashed 
together. Based on my readings on the subject, I would have expected git to 
actually track the content, at least when using blame, but it does not seem to.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
TriggerReactor-squashed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1089


commit f63f81e2d8fad16d668809c8b291256ae5477ab1
Author: Kenneth Knowles 
Date:   2016-06-24T03:05:27Z

Copy triggers to runners-core

This commit is effectively a "git hint" that all the files
in runners-core are moved from prior files. The moved files
will be replaced.

commit f0913147af072cda06a2da4ef6933e311b94dd90
Author: Kenneth Knowles 
Date:   2016-06-24T03:31:10Z

Rename trigger reactors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-745) Display Data fails on transforms with nested components of the same type

[Scott Wegner (JIRA)]

Scott Wegner created BEAM-745:
-

 Summary: Display Data fails on transforms with nested components 
of the same type
 Key: BEAM-745
 URL: https://issues.apache.org/jira/browse/BEAM-745
 Project: Beam
  Issue Type: Bug
Reporter: Scott Wegner
Assignee: Scott Wegner


Display Data supports the notion of "sub components", components within a 
transform class which can contribute their own display data. We add a namespace 
to display data items based on the originating component, which keeps the 
display data items unique within the step.

There are instances where a component is included multiple times within a step. 
We handle the case of the same instance being shared by simply ignoring it the 
second time. However, we don't handle the case of a separate instance being 
added of the same class. Currently the separate instances will add display data 
with the same namespace and key, causing a failure.

This can come up for example when infrastructure at different levels wrap and 
re-wrap a component. We saw this with a bounded source being adapted multiple 
times, Bounded -> Unbounded -> Bounded -> Unbounded. The 
BoundedToUnboundedSourceAdapter was included multiple times with separate 
instances and caused a failure while populating display data.

See: https://github.com/apache/incubator-beam/pull/1088 for a failing test case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1088: Support display data for transforms with ...

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/1088

Support display data for transforms with nested components of the same 
instance

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam 
displaydata-nestedclasses

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1088.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1088


commit d1298954b680c40a26c43d204e478561dfc4902f
Author: Scott Wegner 
Date:   2016-10-12T21:49:41Z

Test case for failing display data for nested component structure




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] incubator-beam-site git commit: This closes #43

[frances]

This closes #43


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/34975927
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/34975927
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/34975927

Branch: refs/heads/asf-site
Commit: 3497592741fe701f694dbee1d0e41dd811310696
Parents: 976b030 4165ad1
Author: Frances Perry 
Authored: Wed Oct 12 14:09:33 2016 -0700
Committer: Frances Perry 
Committed: Wed Oct 12 14:09:33 2016 -0700

--
 _data/authors.yml   |   3 +
 .../2016-10-12-strata-hadoop-world-and-beam.md  |  32 +++
 .../10/11/strata-hadoop-world-and-beam.html | 200 +++
 content/blog/index.html |  16 ++
 content/feed.xml|  40 +++-
 content/images/blog/IMG_20160927_170455.jpg | Bin 0 -> 46625 bytes
 content/images/blog/IMG_20160927_170956.jpg | Bin 0 -> 61284 bytes
 content/index.html  |   2 +
 .../learn/runners/capability-matrix/index.html  |   2 +-
 images/blog/IMG_20160927_170455.jpg | Bin 0 -> 46625 bytes
 images/blog/IMG_20160927_170956.jpg | Bin 0 -> 61284 bytes
 11 files changed, 292 insertions(+), 3 deletions(-)
--

[GitHub] incubator-beam-site pull request #43: Blog post about Strata+Hadoop World Tu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/43


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] incubator-beam-site git commit: Added blog post about Strata+Hadoop World tutorial. Added Jesse Anderson as an author. Added the images for the blog post.

[frances]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site 976b0302a -> 349759274


Added blog post about Strata+Hadoop World tutorial. Added Jesse Anderson as an 
author. Added the images for the blog post.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/ccf6bcfe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/ccf6bcfe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/ccf6bcfe

Branch: refs/heads/asf-site
Commit: ccf6bcfed047090d0c03ed44234c8e77cc508395
Parents: 976b030
Author: Jesse Anderson 
Authored: Tue Oct 11 10:23:33 2016 -0700
Committer: Frances Perry 
Committed: Wed Oct 12 14:00:10 2016 -0700

--
 _data/authors.yml   |   3 +
 .../2016-10-12-strata-hadoop-world-and-beam.md  |  32 +++
 .../10/11/strata-hadoop-world-and-beam.html | 200 +++
 content/blog/index.html |  16 ++
 content/feed.xml|  44 +++-
 content/images/blog/IMG_20160927_170455.jpg | Bin 0 -> 46625 bytes
 content/images/blog/IMG_20160927_170956.jpg | Bin 0 -> 61284 bytes
 content/index.html  |   2 +
 images/blog/IMG_20160927_170455.jpg | Bin 0 -> 46625 bytes
 images/blog/IMG_20160927_170956.jpg | Bin 0 -> 61284 bytes
 10 files changed, 293 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ccf6bcfe/_data/authors.yml
--
diff --git a/_data/authors.yml b/_data/authors.yml
index 6324a96..808513f 100644
--- a/_data/authors.yml
+++ b/_data/authors.yml
@@ -26,3 +26,6 @@ takidau:
 name: Tyler Akidau
 email: taki...@apache.org
 twitter: takidau
+jesseanderson:
+name: Jesse Anderson
+twitter: jessetanderson
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/ccf6bcfe/_posts/2016-10-12-strata-hadoop-world-and-beam.md
--
diff --git a/_posts/2016-10-12-strata-hadoop-world-and-beam.md 
b/_posts/2016-10-12-strata-hadoop-world-and-beam.md
new file mode 100644
index 000..7e1d7b9
--- /dev/null
+++ b/_posts/2016-10-12-strata-hadoop-world-and-beam.md
@@ -0,0 +1,32 @@
+---
+layout: post
+title:  "Strata+Hadoop World and Beam"
+date:   2016-10-11 09:00:00 -0800
+excerpt_separator: 
+categories: beam update
+authors:
+- jesseanderson
+---
+
+Tyler Akidau and I gave a [three-hour 
tutorial](http://conferences.oreilly.com/strata/hadoop-big-data-ny/public/schedule/detail/52129)
 on Apache Beam at Strata+Hadoop World 2016. We had a plethora of help from our 
TAs: Kenn Knowles, Reuven Lax, Felipe Hoffa, Slava Chernyak, and Jamie Grier. 
There were a total of 66 people that attended the session.
+
+
+
+If you want to take a look at the tutorial materials, we’ve put them up [on 
GitHub](https://github.com/eljefe6a/beamexample). This includes the [actual 
slides](https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf)
 as well as the 
[exercises](https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game)
 that we covered. If you’re looking to learn a little about Beam, this is a 
good way to start. The exercises are based on an imaginary mobile game where 
data needs processing and are based on code in the [Beam examples 
directory](https://github.com/apache/incubator-beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game).
 The code has TODOs for where you need to fill in code or there are full sample 
solutions to look over our code. You can run these examples on your own machine 
or on a cluster using a runner that Beam supports.
+
+I want to share some of takeaways I had about Beam during the conference.
+
+The Data Engineers are looking to Beam as a way to 
[future-proof](https://www.oreilly.com/ideas/future-proof-and-scale-proof-your-code),
 meaning that code is portable between the various Big Data frameworks. In 
fact, many of the attendees were still on Hadoop MapReduce and looking to 
transition to a new framework. They’re realizing that continually rewriting 
code isn’t the most productive approach.
+
+Data Scientists are really interested in using Beam. They interested in having 
a single API for doing analysis instead of several different APIs. We talked 
about Beam’s progress on the Python API. If you want to take a peek, it’s 
being actively developed on a [feature 
branch](https://github.com/apache/incubator-beam/tree/python-sdk). As Beam 
matures, we’re looking to add other supported languages.
+
+We heard [loud and 
clear](https://twit

[2/3] incubator-beam-site git commit: Regenerate html and make 'rake test' pass.

[frances]

Regenerate html and make 'rake test' pass.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/4165ad18
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/4165ad18
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/4165ad18

Branch: refs/heads/asf-site
Commit: 4165ad181316f1cc3a078bf79490dcb6775c1474
Parents: ccf6bcf
Author: Frances Perry 
Authored: Wed Oct 12 14:06:15 2016 -0700
Committer: Frances Perry 
Committed: Wed Oct 12 14:06:15 2016 -0700

--
 _posts/2016-10-12-strata-hadoop-world-and-beam.md   |  2 +-
 .../update/2016/10/11/strata-hadoop-world-and-beam.html |  4 ++--
 content/feed.xml| 12 ++--
 content/learn/runners/capability-matrix/index.html  |  2 +-
 4 files changed, 10 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4165ad18/_posts/2016-10-12-strata-hadoop-world-and-beam.md
--
diff --git a/_posts/2016-10-12-strata-hadoop-world-and-beam.md 
b/_posts/2016-10-12-strata-hadoop-world-and-beam.md
index 7e1d7b9..acd48d7 100644
--- a/_posts/2016-10-12-strata-hadoop-world-and-beam.md
+++ b/_posts/2016-10-12-strata-hadoop-world-and-beam.md
@@ -28,5 +28,5 @@ On management and thought leader side, Beam went from 
“what’s Beam?” at pr
 
 We rounded out the tutorial with live demonstrations of Beam running on Apache 
Spark, Apache Flink, the local runner, and DataFlow runner. Then, we brought in 
the big brainpower and had a Q and A session.
 
-If you’re attending a conference, we encourage you to look for a Beam 
session. If you want to use these materials to give your own Beam talk or 
tutorial, we’re happy to help you. In addition to this tutorial, we have 
[other presentation 
materials](beam.incubator.apache.org/learn/presentation-materials/). You can 
reach out to us on the [user mailing 
list](http://beam.incubator.apache.org/use/mailing-lists/).
+If you’re attending a conference, we encourage you to look for a Beam 
session. If you want to use these materials to give your own Beam talk or 
tutorial, we’re happy to help you. In addition to this tutorial, we have 
[other presentation materials](/learn/presentation-materials/). You can reach 
out to us on the [user mailing 
list](http://beam.incubator.apache.org/use/mailing-lists/).
 

http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/4165ad18/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
--
diff --git a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html 
b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
index 92f30f0..d402fa2 100644
--- a/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
+++ b/content/beam/update/2016/10/11/strata-hadoop-world-and-beam.html
@@ -154,7 +154,7 @@
 
 
 
-If you want to take a look at the tutorial materials, we’ve put them up 
https://github.com/eljefe6a/beamexample";>on GitHub. This includes 
the https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf";>actual
 slides as well as the https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game";>exercises
 that we covered. If you’re looking to learn a little about Beam, this is a 
good way to start. The exercises are based on an imaginary mobile game and the 
data needs processing and are based on code in the https://github.com/apache/incubator-beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game";>Beam
 examples directory. The code has TODOs for where you need to fill in code 
or there are full sample solutions to look over our code. You can run these 
examples on your own machine.
+If you want to take a look at the tutorial materials, we’ve put them up 
https://github.com/eljefe6a/beamexample";>on GitHub. This includes 
the https://github.com/eljefe6a/beamexample/blob/master/BeamTutorial/slides.pdf";>actual
 slides as well as the https://github.com/eljefe6a/beamexample/tree/master/BeamTutorial/src/main/java/org/apache/beam/examples/tutorial/game";>exercises
 that we covered. If you’re looking to learn a little about Beam, this is a 
good way to start. The exercises are based on an imaginary mobile game where 
data needs processing and are based on code in the https://github.com/apache/incubator-beam/tree/master/examples/java8/src/main/java/org/apache/beam/examples/complete/game";>Beam
 examples directory. The code has TODOs for where you need to fill in code 
or there are full sample solutions to look over our code. You can run these 
examples on your own ma

[jira] [Commented] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569856#comment-15569856
 ] 

Raghu Angadi commented on BEAM-744:
---

> KAFKA_POLL_TIMEOUT - consumer poll timeout, default: 1 second.
This timeout is KafkaIO internal implementation detail and should be ignored 
here. It does not impose any limitations on the reader (i.e. the reader can be 
closed before this timeout and everything is cleaned up properly).

> A runner should be able to override KafkaIO max wait properties.
> 
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has three "wait" properties:
> {{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela updated BEAM-744:
---
Assignee: (was: James Malone)

> A runner should be able to override KafkaIO max wait properties.
> 
>
> Key: BEAM-744
> URL: https://issues.apache.org/jira/browse/BEAM-744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Amit Sela
>
> KafkaIO has three "wait" properties:
> {{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
> default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the 
> DirectRunner, and I can add that they are now limiting the Spark runner 
> (which reads in defined time frames, which may be smaller then the wait time 
> and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a 
> runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
> may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

[Amit Sela (JIRA)]

Amit Sela created BEAM-744:
--

 Summary: A runner should be able to override KafkaIO max wait 
properties.
 Key: BEAM-744
 URL: https://issues.apache.org/jira/browse/BEAM-744
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Amit Sela
Assignee: James Malone


KafkaIO has three "wait" properties:

{{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
{{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 5 seconds.
{{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, 
default: 10 msec.

[~rangadi] mentioned some of these were set to due to limitations of the 
DirectRunner, and I can add that they are now limiting the Spark runner (which 
reads in defined time frames, which may be smaller then the wait time and so 
never actually read).

This feels like defaults should be set for optimal read from Kafka, while a 
runner may override those if it needs to.

[~rangadi] also mentioned that this could be set in {{PipelineOptions}} which 
may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1087: Resolve Registered Runners that don't end...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1087

Resolve Registered Runners that don't end with Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Resolve runners in a case-insensitive manner.

This reduces duplication in specifying a runner

e.g. the DirectRunner can be specified with (among others) any of
"--runner=direct", "--runner=directrunner", "--runner=DirectRunner",
"--runner=Direct", or "--runner=directRunner"

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam auto_runner

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1087.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1087






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-739) Log full exception stack trace in WordCountIT and BigQueryTornadoesIT

[Pei He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pei He resolved BEAM-739.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Log full exception stack trace in WordCountIT and BigQueryTornadoesIT
> -
>
> Key: BEAM-739
> URL: https://issues.apache.org/jira/browse/BEAM-739
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
> Fix For: Not applicable
>
>
> When IT tests are broken, they don't provide the full stack trace, such as in:
> https://issues.apache.org/jira/browse/BEAM-736
> It makes investigating root causes slower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1086: [BEAM-25] Add DoFn.StateId annotation and...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1086

[BEAM-25] Add DoFn.StateId annotation and validation on fields

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This introduces just the annotation with analysis and validation in 
`DoFnSignature`. To keep reviews small and easy (and happening in parallel with 
other functionality) my plan is to gradually introduce the annotations and 
validation, while rejecting pipelines that actually use them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DoFn-State-annotations

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1086.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1086


commit 739720655224211dd1759bbe2c52fed7c7ecee47
Author: Kenneth Knowles 
Date:   2016-10-11T04:16:37Z

Add DoFn.StateId annotation and validation on fields




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569622#comment-15569622
 ] 

ASF GitHub Bot commented on BEAM-25:


GitHub user kennknowles opened a pull request:

https://github.com/apache/incubator-beam/pull/1086

[BEAM-25] Add DoFn.StateId annotation and validation on fields

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [x] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This introduces just the annotation with analysis and validation in 
`DoFnSignature`. To keep reviews small and easy (and happening in parallel with 
other functionality) my plan is to gradually introduce the annotations and 
validation, while rejecting pipelines that actually use them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/incubator-beam 
DoFn-State-annotations

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1086.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1086


commit 739720655224211dd1759bbe2c52fed7c7ecee47
Author: Kenneth Knowles 
Date:   2016-10-11T04:16:37Z

Add DoFn.StateId annotation and validation on fields




> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: Examples :: Java #1503

[Dan Halperin]

Just an FYI that the issues here were legitimate issues in an external
service that have since been resolved. They were present for approximately
90 minutes in a small set of places, and we were affected :)

On Tue, Oct 11, 2016 at 7:37 PM, Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> See  MavenVerify/org.apache.beam$beam-examples-java/1503/>
>
>

incubator-beam git commit: Minor rewording to drop Dataflow from Beam SDK core

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4d10f86cf -> c561cd9c1


Minor rewording to drop Dataflow from Beam SDK core


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c561cd9c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c561cd9c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c561cd9c

Branch: refs/heads/master
Commit: c561cd9c1be6e77568b129792ace3d8d52bc329b
Parents: 4d10f86
Author: Dan Halperin 
Authored: Mon Sep 26 22:43:40 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 11:49:37 2016 -0700

--
 examples/java/pom.xml |  2 +-
 pom.xml   |  4 ++--
 runners/google-cloud-dataflow-java/pom.xml|  2 +-
 sdks/java/core/pom.xml|  2 +-
 .../java/org/apache/beam/sdk/coders/Coder.java|  2 +-
 .../org/apache/beam/sdk/coders/package-info.java  |  2 +-
 .../java/org/apache/beam/sdk/io/PubsubIO.java | 18 +-
 .../main/java/org/apache/beam/sdk/io/Sink.java|  2 +-
 .../main/java/org/apache/beam/sdk/io/TextIO.java  | 15 ++-
 .../java/org/apache/beam/sdk/io/XmlSource.java| 13 +++--
 .../apache/beam/sdk/io/range/ByteKeyRange.java|  6 +++---
 .../java/org/apache/beam/sdk/package-info.java|  2 +-
 .../org/apache/beam/sdk/runners/package-info.java | 17 +++--
 .../org/apache/beam/sdk/testing/package-info.java |  4 ++--
 .../org/apache/beam/sdk/transforms/OldDoFn.java   | 15 ---
 .../apache/beam/sdk/transforms/PTransform.java|  2 +-
 .../org/apache/beam/sdk/transforms/ParDo.java |  7 +++
 .../org/apache/beam/sdk/util/ReleaseInfo.java |  6 +++---
 .../org/apache/beam/sdk/values/package-info.java  |  4 ++--
 19 files changed, 52 insertions(+), 73 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c561cd9c/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index aa7c22a..d18f959 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -296,7 +296,7 @@
   Apache Beam Examples
 
   org.apache.beam.examples
-  -exclude 
org.apache.beam.sdk.runners.worker:org.apache.beam.sdk.runners.dataflow:org.apache.beam.sdk.util
 ${dataflow.javadoc_opts}
+  -exclude 
org.apache.beam.sdk.runners.worker:org.apache.beam.sdk.runners.dataflow:org.apache.beam.sdk.util
 ${beam.javadoc_opts}
   false
   true
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c561cd9c/pom.xml
--
diff --git a/pom.xml b/pom.xml
index c135f17..8a28bd9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -92,7 +92,7 @@
 
   
 UTF-8
-
+
 
 
 true
@@ -224,7 +224,7 @@
 [1.8,)
   
   
--Xdoclint:-missing
+-Xdoclint:-missing
   
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c561cd9c/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index bf4d93d..b035028 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -98,7 +98,7 @@
   Google Cloud Dataflow Runner for Java, version 
${project.version}
 
   org.apache.beam.runners.dataflow
-  -exclude 
org.apache.beam.sdk.runners.dataflow.internal:org.apache.beam.sdk.runners.dataflow.testing:org.apache.beam.sdk.runners.dataflow.util
 ${dataflow.javadoc_opts}
+  -exclude 
org.apache.beam.sdk.runners.dataflow.internal:org.apache.beam.sdk.runners.dataflow.testing:org.apache.beam.sdk.runners.dataflow.util
 ${beam.javadoc_opts}
   false
   true
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c561cd9c/sdks/java/core/pom.xml
--
diff --git a/sdks/java/core/pom.xml b/sdks/java/core/pom.xml
index aa0ad09..d44a494 100644
--- a/sdks/java/core/pom.xml
+++ b/sdks/java/core/pom.xml
@@ -60,7 +60,7 @@
 org.apache.beam.sdk
 -exclude
   
org.apache.beam.sdk.runners.worker:org.apache.beam.sdk.runners.dataflow:org.apache.beam.sdk.util:org.apache.beam.sdk.runners.inprocess
-  ${dataflow.javadoc_opts}
+  ${beam.javadoc_opts}
 false
 true
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c561cd9c/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/Coder.java
--

[3/3] incubator-beam git commit: This closes #1076

[kenn]

This closes #1076


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4d10f86c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4d10f86c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4d10f86c

Branch: refs/heads/master
Commit: 4d10f86cfb26ba7c765e8130ea561c78ebabe52d
Parents: 7675035 02cfaf0
Author: Kenneth Knowles 
Authored: Wed Oct 12 11:27:24 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Oct 12 11:27:24 2016 -0700

--
 .../java/org/apache/beam/runners/core/SideInputHandler.java   | 2 +-
 .../main/java/org/apache/beam/sdk/util/PCollectionViews.java  | 6 --
 .../main/java/org/apache/beam/sdk/values/PCollectionView.java | 7 ---
 .../org/apache/beam/sdk/testing/PCollectionViewTesting.java   | 7 ---
 4 files changed, 1 insertion(+), 21 deletions(-)
--

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569495#comment-15569495
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1076


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam git commit: Remove the deprecated PCollectionView.fromIterableInternal

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7675035be -> 4d10f86cf


Remove the deprecated PCollectionView.fromIterableInternal


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/02cfaf0c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/02cfaf0c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/02cfaf0c

Branch: refs/heads/master
Commit: 02cfaf0cc2338616da35de8403365c7d76c266d4
Parents: 1ab5d00
Author: Kenneth Knowles 
Authored: Mon Oct 10 13:45:58 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Oct 11 20:42:36 2016 -0700

--
 .../main/java/org/apache/beam/sdk/util/PCollectionViews.java  | 6 --
 .../main/java/org/apache/beam/sdk/values/PCollectionView.java | 7 ---
 .../org/apache/beam/sdk/testing/PCollectionViewTesting.java   | 7 ---
 3 files changed, 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/02cfaf0c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PCollectionViews.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PCollectionViews.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PCollectionViews.java
index 3b1fde9..83ccae8 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PCollectionViews.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/PCollectionViews.java
@@ -458,12 +458,6 @@ public class PCollectionViews {
   return untypedViewFn;
 }
 
-@Override
-@Deprecated
-public ViewT fromIterableInternal(Iterable> elements) {
-  return getViewFn().apply(elements);
-}
-
 /**
  * Returns a unique {@link TupleTag} identifying this {@link 
PCollectionView}.
  *

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/02cfaf0c/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
index 0e5f594..457e78a 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionView.java
@@ -52,13 +52,6 @@ public interface PCollectionView extends PValue, 
Serializable {
   public TupleTag>> getTagInternal();
 
   /**
-   * @deprecated use {@link #getViewFn()} for now, but eventually get the 
needed information via the
-   * side input specification on the {@link ParDo} transform.
-   */
-  @Deprecated
-  public T fromIterableInternal(Iterable> contents);
-
-  /**
* @deprecated this method will be removed entirely. The {@link ViewFn} for 
a side input is an
* attribute of the side input's specification with a {@link ParDo} 
transform, which will
* obtain this specification via a package-private channel.

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/02cfaf0c/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
index ea8e0af..99fb1fb 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/testing/PCollectionViewTesting.java
@@ -245,13 +245,6 @@ public final class PCollectionViewTesting {
   return (TupleTag) tag;
 }
 
-@SuppressWarnings({"unchecked", "rawtypes"})
-@Override
-@Deprecated
-public ViewT fromIterableInternal(Iterable> contents) {
-  return getViewFn().apply(contents);
-}
-
 @Override
 public ViewFn>, ViewT> getViewFn() {
   // Safe cast; runners must maintain type safety

[GitHub] incubator-beam pull request #1076: [BEAM-115] Remove PCollectionView#fromIte...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1076


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] incubator-beam git commit: Remove last use of PCollectionView.fromIterableInternal

[kenn]

Remove last use of PCollectionView.fromIterableInternal


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1ab5d00e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1ab5d00e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1ab5d00e

Branch: refs/heads/master
Commit: 1ab5d00e78a124d297c45edc8c6b55d7c2683543
Parents: 135790b
Author: Kenneth Knowles 
Authored: Mon Oct 10 13:22:02 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Oct 11 20:42:36 2016 -0700

--
 .../main/java/org/apache/beam/runners/core/SideInputHandler.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1ab5d00e/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
index 851ed37..ae7f759 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java
@@ -187,7 +187,7 @@ public class SideInputHandler implements 
ReadyCheckingSideInputReader {
 
 Iterable> elements = state.read();
 
-return sideInput.fromIterableInternal(elements);
+return sideInput.getViewFn().apply(elements);
   }
 
   @Override

[GitHub] incubator-beam pull request #1085: Remove Debug Message when Delivering Time...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1085


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Remove Debug Message when Delivering Timers

[dhalperi]

Remove Debug Message when Delivering Timers


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1f1d8524
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1f1d8524
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1f1d8524

Branch: refs/heads/master
Commit: 1f1d85241e423104e4619164678d57602c77f834
Parents: bc9ed7d
Author: Thomas Groh 
Authored: Wed Oct 12 09:21:33 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 11:04:57 2016 -0700

--
 .../apache/beam/runners/direct/ExecutorServiceParallelExecutor.java | 1 -
 1 file changed, 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1f1d8524/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
--
diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
index 567def2..fab6a33 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ExecutorServiceParallelExecutor.java
@@ -453,7 +453,6 @@ final class ExecutorServiceParallelExecutor implements 
PipelineExecutor {
   }
   KeyedWorkItem work =
   KeyedWorkItems.timersWorkItem(keyTimers.getKey().getKey(), 
delivery);
-  LOG.warn("Delivering {} timers for {}", delivery.size(), 
keyTimers.getKey().getKey());
   @SuppressWarnings({"unchecked", "rawtypes"})
   CommittedBundle bundle =
   evaluationContext

[1/2] incubator-beam git commit: Closes #1085

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master bc9ed7dbd -> 7675035be


Closes #1085


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7675035b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7675035b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7675035b

Branch: refs/heads/master
Commit: 7675035bede78fde164dcdeb75feb63083ae0d4b
Parents: bc9ed7d 1f1d852
Author: Dan Halperin 
Authored: Wed Oct 12 11:04:57 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 11:04:57 2016 -0700

--
 .../apache/beam/runners/direct/ExecutorServiceParallelExecutor.java | 1 -
 1 file changed, 1 deletion(-)
--

[jira] [Closed] (BEAM-701) Support Avro Compression for writes

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-701.

Resolution: Fixed

> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: features
> Fix For: 0.3.0-incubating
>
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (BEAM-701) Support Avro Compression for writes

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin reopened BEAM-701:
--
  Assignee: Rafal Wojdyla

Reopening to assign to Rav

> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>Assignee: Rafal Wojdyla
>  Labels: features
> Fix For: 0.3.0-incubating
>
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-701) Support Avro Compression for writes

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-701.

   Resolution: Fixed
 Assignee: (was: Davor Bonaci)
Fix Version/s: 0.3.0-incubating

> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>  Labels: features
> Fix For: 0.3.0-incubating
>
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : beam_PostCommit_RunnableOnService_FlinkLocal #542

[Apache Jenkins Server]

See 

[1/2] incubator-beam git commit: Refactor StateSpec out of StateTag

[kenn]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 142229e37 -> bc9ed7dbd


Refactor StateSpec out of StateTag


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7c1ba2e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7c1ba2e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7c1ba2e1

Branch: refs/heads/master
Commit: 7c1ba2e1062556ac98b29f5bb4f5b75a7e7832e2
Parents: 135790b
Author: Kenneth Knowles 
Authored: Thu Aug 4 20:50:28 2016 -0700
Committer: Kenneth Knowles 
Committed: Tue Oct 11 20:27:12 2016 -0700

--
 .../src/main/resources/beam/findbugs-filter.xml |   2 +-
 .../apache/beam/sdk/util/state/StateBinder.java |  67 +++
 .../apache/beam/sdk/util/state/StateSpec.java   |  39 ++
 .../apache/beam/sdk/util/state/StateSpecs.java  | 452 +++
 .../apache/beam/sdk/util/state/StateTag.java|  82 ++--
 .../apache/beam/sdk/util/state/StateTags.java   | 386 ++--
 6 files changed, 655 insertions(+), 373 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7c1ba2e1/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
--
diff --git a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml 
b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
index 3c01690..c9223a7 100644
--- a/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
+++ b/sdks/java/build-tools/src/main/resources/beam/findbugs-filter.xml
@@ -116,7 +116,7 @@
 
   
   
-
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7c1ba2e1/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
new file mode 100644
index 000..0521e15
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateBinder.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.util.state;
+
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.transforms.Combine;
+import org.apache.beam.sdk.transforms.CombineWithContext;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.transforms.windowing.OutputTimeFn;
+
+/**
+ * Visitor for binding a {@link StateSpec} and to the associated {@link State}.
+ *
+ * @param  the type of key this binder embodies.
+ */
+public interface StateBinder {
+   ValueState bindValue(String id, StateSpec> 
spec, Coder coder);
+
+   BagState bindBag(String id, StateSpec> spec, 
Coder elemCoder);
+
+   AccumulatorCombiningState 
bindCombiningValue(
+  String id,
+  StateSpec> 
spec,
+  Coder accumCoder,
+  Combine.CombineFn combineFn);
+
+  
+  AccumulatorCombiningState 
bindKeyedCombiningValue(
+  String id,
+  StateSpec> spec,
+  Coder accumCoder,
+  Combine.KeyedCombineFn 
combineFn);
+
+  
+  AccumulatorCombiningState 
bindKeyedCombiningValueWithContext(
+  String id,
+  StateSpec> spec,
+  Coder accumCoder,
+  CombineWithContext.KeyedCombineFnWithContext
+  combineFn);
+
+  /**
+   * Bind to a watermark {@link StateSpec}.
+   *
+   * This accepts the {@link OutputTimeFn} that dictates how watermark hold 
timestamps added to
+   * the returned {@link WatermarkHoldState} are to be combined.
+   */
+   WatermarkHoldState bindWatermark(
+  String id,
+  StateSpec> spec,
+  OutputTimeFn outputTimeFn);
+}

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7c1ba2e1/sdks/java/core/src/main/java/org/apache/beam/sdk/util/state/StateSpec.java
--
diff --git 
a/sdks/ja

[jira] [Closed] (BEAM-556) typo in documentation

[Frank Yellin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Frank Yellin closed BEAM-556.
-
   Resolution: Fixed
Fix Version/s: 0.3.0-incubating

> typo in documentation
> -
>
> Key: BEAM-556
> URL: https://issues.apache.org/jira/browse/BEAM-556
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Assignee: Frank Yellin
>Priority: Trivial
> Fix For: 0.3.0-incubating
>
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> transform.py:
> ergument -> argument  
> in documentation for parse_label_and_args



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1044: [BEAM-25] Refactor StateSpec out of State...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1044


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: This closes #1044

[kenn]

This closes #1044


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/bc9ed7db
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/bc9ed7db
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/bc9ed7db

Branch: refs/heads/master
Commit: bc9ed7dbd9da9f7addc365ad511b106cfcc69b01
Parents: 142229e 7c1ba2e
Author: Kenneth Knowles 
Authored: Wed Oct 12 10:17:28 2016 -0700
Committer: Kenneth Knowles 
Committed: Wed Oct 12 10:17:28 2016 -0700

--
 .../src/main/resources/beam/findbugs-filter.xml |   2 +-
 .../apache/beam/sdk/util/state/StateBinder.java |  67 +++
 .../apache/beam/sdk/util/state/StateSpec.java   |  39 ++
 .../apache/beam/sdk/util/state/StateSpecs.java  | 452 +++
 .../apache/beam/sdk/util/state/StateTag.java|  82 ++--
 .../apache/beam/sdk/util/state/StateTags.java   | 386 ++--
 6 files changed, 655 insertions(+), 373 deletions(-)
--

[jira] [Commented] (BEAM-25) Add user-ready API for interacting with state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569301#comment-15569301
 ] 

ASF GitHub Bot commented on BEAM-25:


Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1044


> Add user-ready API for interacting with state
> -
>
> Key: BEAM-25
> URL: https://issues.apache.org/jira/browse/BEAM-25
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: State
>
> Our current state API is targeted at runner implementers, not pipeline 
> authors. As such it has many capabilities that are not necessary nor 
> desirable for simple use cases of stateful ParDo (such as dynamic state tag 
> creation). Implement a simple state intended for user access.
> (Details of our current thoughts in forthcoming design doc)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #915: Propagate Coder in WithTimestamps

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/915


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #1077: Add profile_memory flag

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/incubator-beam/pull/1077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #689: Express ParDo.Bound in terms of ParDo.Boun...

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/689


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #894: Rename TIMESTAMP_MIN/MAX to POSITIVE/NEGAT...

[tgroh]

Github user tgroh closed the pull request at:

https://github.com/apache/incubator-beam/pull/894


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_RunnableOnService_FlinkLocal #541

[Apache Jenkins Server]

See 


Changes:

[dhalperi] Add compression codec for AvroIO.Write

--
[...truncated 1403 lines...]
[WARNING]   - 134 more...
[WARNING] jcip-annotations-1.0.jar, annotations-3.0.1.jar define 4 overlapping 
classes: 
[WARNING]   - net.jcip.annotations.GuardedBy
[WARNING]   - net.jcip.annotations.NotThreadSafe
[WARNING]   - net.jcip.annotations.ThreadSafe
[WARNING]   - net.jcip.annotations.Immutable
[WARNING] beam-runners-core-java-0.3.0-incubating-SNAPSHOT.jar, 
beam-sdks-java-core-0.3.0-incubating-SNAPSHOT.jar define 1717 overlapping 
classes: 
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.TreeRangeSet$ComplementRangesByLowerBound$2
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.WellBehavedMap$EntrySet$1$1
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.util.concurrent.CycleDetectingLockFactory$Policies$1
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Maps$6
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.primitives.UnsignedBytes$LexicographicalComparatorHolder$UnsafeComparator$1
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Range$1
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.collect.Collections2$OrderedPermutationCollection
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.base.Splitter$2
[WARNING]   - 
org.apache.beam.sdk.repackaged.com.google.common.base.Equivalence$Identity
[WARNING]   - org.apache.beam.sdk.repackaged.com.google.common.collect.Lists$1
[WARNING]   - 1707 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Replacing 

 with 

[INFO] 
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
beam-runners-core-java ---
[INFO] No dependency problems found
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Runners :: Flink 0.3.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ 
beam-runners-flink-parent ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @ 
beam-runners-flink-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
beam-runners-flink-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.4:attach-descriptor (attach-descriptor) @ 
beam-runners-flink-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:analyze-only (default) @ 
beam-runners-flink-parent ---
[INFO] Skipping pom project
[INFO] 
[INFO] 
[INFO] Building Apache Beam :: Runners :: Flink :: Core 
0.3.0-incubating-SNAPSHOT
[INFO] 
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ 
beam-runners-flink_2.10 ---
[INFO] Deleting 

[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce) @ 
beam-runners-flink_2.10 ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
beam-runners-flink_2.10 ---
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ 
beam-runners-flink_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ 
beam-runners-flink_2.10 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 63 source files to 

[INFO] 

[2/2] incubator-beam git commit: Closes #1038

[dhalperi]

Closes #1038


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/142229e3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/142229e3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/142229e3

Branch: refs/heads/master
Commit: 142229e3719952451ef9a65c4cfc77c2d27520fd
Parents: 4f991fd 2b22d00
Author: Dan Halperin 
Authored: Wed Oct 12 09:35:49 2016 -0700
Committer: Dan Halperin 
Committed: Wed Oct 12 09:35:49 2016 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIO.java | 158 ---
 .../sdk/io/SerializableAvroCodecFactory.java| 112 +
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 107 -
 .../io/SerializableAvroCodecFactoryTest.java| 100 
 4 files changed, 458 insertions(+), 19 deletions(-)
--

[jira] [Commented] (BEAM-701) Support Avro Compression for writes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569182#comment-15569182
 ] 

ASF GitHub Bot commented on BEAM-701:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1038


> Support Avro Compression for writes
> ---
>
> Key: BEAM-701
> URL: https://issues.apache.org/jira/browse/BEAM-701
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Rafal Wojdyla
>Assignee: Davor Bonaci
>  Labels: features
>
> Support compression on Avro Write.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1038: [BEAM-701] Support Avro write compression

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1038


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Add compression codec for AvroIO.Write

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 4f991fd82 -> 142229e37


Add compression codec for AvroIO.Write

BEHAVIOUR CHANGE: prior to this change Avro output would not use
compression. Starting from this commit, by default Avro output is
compressed using deflate codec (level 6).


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/2b22d003
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/2b22d003
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/2b22d003

Branch: refs/heads/master
Commit: 2b22d003dabb7fddabdc8aaea872478fe13d407a
Parents: 4f991fd
Author: Rafal Wojdyla 
Authored: Mon Oct 3 14:02:59 2016 -0400
Committer: Dan Halperin 
Committed: Wed Oct 12 09:35:23 2016 -0700

--
 .../java/org/apache/beam/sdk/io/AvroIO.java | 158 ---
 .../sdk/io/SerializableAvroCodecFactory.java| 112 +
 .../java/org/apache/beam/sdk/io/AvroIOTest.java | 107 -
 .../io/SerializableAvroCodecFactoryTest.java| 100 
 4 files changed, 458 insertions(+), 19 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/2b22d003/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
index 267265d..eeb4bb7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java
@@ -27,6 +27,7 @@ import java.nio.channels.WritableByteChannel;
 import java.util.regex.Pattern;
 import javax.annotation.Nullable;
 import org.apache.avro.Schema;
+import org.apache.avro.file.CodecFactory;
 import org.apache.avro.file.DataFileWriter;
 import org.apache.avro.generic.GenericRecord;
 import org.apache.avro.reflect.ReflectData;
@@ -443,6 +444,13 @@ public class AvroIO {
 }
 
 /**
+ * Returns a {@link PTransform} that writes Avro file(s) using specified 
codec.
+ */
+public static Bound withCodec(CodecFactory codec) {
+  return new Bound<>(GenericRecord.class).withCodec(codec);
+}
+
+/**
  * A {@link PTransform} that writes a bounded {@link PCollection} to an 
Avro file (or
  * multiple Avro files matching a sharding pattern).
  *
@@ -450,6 +458,8 @@ public class AvroIO {
  */
 public static class Bound extends PTransform, PDone> {
   private static final String DEFAULT_SHARD_TEMPLATE = 
ShardNameTemplate.INDEX_OF_MAX;
+  private static final SerializableAvroCodecFactory DEFAULT_CODEC =
+  new SerializableAvroCodecFactory(CodecFactory.deflateCodec(6));
 
   /** The filename to write to. */
   @Nullable
@@ -467,9 +477,23 @@ public class AvroIO {
   final Schema schema;
   /** An option to indicate if output validation is desired. Default is 
true. */
   final boolean validate;
+  /**
+   * The codec used to encode the blocks in the Avro file. String value 
drawn from those in
+   * 
https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/file/CodecFactory.html
+   */
+  final SerializableAvroCodecFactory codec;
 
   Bound(Class type) {
-this(null, null, "", 0, DEFAULT_SHARD_TEMPLATE, type, null, true);
+this(
+null,
+null,
+"",
+0,
+DEFAULT_SHARD_TEMPLATE,
+type,
+null,
+true,
+DEFAULT_CODEC);
   }
 
   Bound(
@@ -480,7 +504,8 @@ public class AvroIO {
   String shardTemplate,
   Class type,
   Schema schema,
-  boolean validate) {
+  boolean validate,
+  SerializableAvroCodecFactory codec) {
 super(name);
 this.filenamePrefix = filenamePrefix;
 this.filenameSuffix = filenameSuffix;
@@ -489,6 +514,7 @@ public class AvroIO {
 this.type = type;
 this.schema = schema;
 this.validate = validate;
+this.codec = codec;
   }
 
   /**
@@ -503,7 +529,15 @@ public class AvroIO {
   public Bound to(String filenamePrefix) {
 validateOutputComponent(filenamePrefix);
 return new Bound<>(
-name, filenamePrefix, filenameSuffix, numShards, shardTemplate, 
type, schema, validate);
+name,
+filenamePrefix,
+filenameSuffix,
+numShards,
+shardTemplate,
+type,
+schema,
+validate,
+codec);
   }
 
   /**
@@ -517,7 +551,15 @@ public class AvroIO {
   public Bound withSuffix(String filenameSuffix) {
 validateOutputComponent(filena

[GitHub] incubator-beam pull request #1085: Remove Debug Message when Delivering Time...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1085

Remove Debug Message when Delivering Timers

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam rm_warn_timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1085.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1085


commit 5ec1fda3e543dc3a8da5699ac26b3dfacff82e56
Author: Thomas Groh 
Date:   2016-10-12T16:21:33Z

Reduce Level of Debug Message when Delivering Timers




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-739) Log full exception stack trace in WordCountIT and BigQueryTornadoesIT

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569036#comment-15569036
 ] 

ASF GitHub Bot commented on BEAM-739:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1079


> Log full exception stack trace in WordCountIT and BigQueryTornadoesIT
> -
>
> Key: BEAM-739
> URL: https://issues.apache.org/jira/browse/BEAM-739
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Pei He
>Assignee: Pei He
>Priority: Minor
>
> When IT tests are broken, they don't provide the full stack trace, such as in:
> https://issues.apache.org/jira/browse/BEAM-736
> It makes investigating root causes slower.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1079: [BEAM-739] Log exceptions full stack trac...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1079


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: [BEAM-739] Log exceptions full stack track in IT tests

[lcwik]

[BEAM-739] Log exceptions full stack track in IT tests

This closes #1079


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4f991fd8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4f991fd8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4f991fd8

Branch: refs/heads/master
Commit: 4f991fd828b0d03556fee7029a1d19657af318ed
Parents: 135790b 1affccc
Author: Luke Cwik 
Authored: Wed Oct 12 08:22:12 2016 -0700
Committer: Luke Cwik 
Committed: Wed Oct 12 08:22:12 2016 -0700

--
 examples/java/pom.xml  | 2 --
 pom.xml| 4 
 sdks/java/io/google-cloud-platform/pom.xml | 1 -
 sdks/java/io/kinesis/pom.xml   | 1 -
 4 files changed, 4 insertions(+), 4 deletions(-)
--

[1/2] incubator-beam git commit: Disable trimStackTrace in maven-failsafe-plugin

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 135790bc9 -> 4f991fd82


Disable trimStackTrace in maven-failsafe-plugin


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/1affcccd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/1affcccd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/1affcccd

Branch: refs/heads/master
Commit: 1affcccd4f4dafa0c6c6ec2360c5fbcdf55247ce
Parents: 135790b
Author: Pei He 
Authored: Tue Oct 11 15:56:58 2016 -0700
Committer: Luke Cwik 
Committed: Wed Oct 12 08:21:48 2016 -0700

--
 examples/java/pom.xml  | 2 --
 pom.xml| 4 
 sdks/java/io/google-cloud-platform/pom.xml | 1 -
 sdks/java/io/kinesis/pom.xml   | 1 -
 4 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1affcccd/examples/java/pom.xml
--
diff --git a/examples/java/pom.xml b/examples/java/pom.xml
index 1923366..aa7c22a 100644
--- a/examples/java/pom.xml
+++ b/examples/java/pom.xml
@@ -127,7 +127,6 @@
 maven-failsafe-plugin
 
   false
-  true
 
 
   
@@ -404,7 +403,6 @@
 maven-failsafe-plugin
 
   false
-  true
 
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/1affcccd/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 7295261..c135f17 100644
--- a/pom.xml
+++ b/pom.xml
@@ -981,6 +981,10 @@
   org.apache.maven.plugins
   maven-failsafe-plugin
   2.19.1
+  
+false
+true
+  
 
 
 

[jira] [Commented] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569001#comment-15569001
 ] 

Aljoscha Krettek commented on BEAM-696:
---

Flink doesn't use bundles but it just keeps all the elements in fault-tolerant 
state until a trigger fires. So mostly yes to your question. :-)

> Side-Inputs non-deterministic with merging main-input windows
> -
>
> Key: BEAM-696
> URL: https://issues.apache.org/jira/browse/BEAM-696
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
>Reporter: Ben Chambers
>Assignee: Pei He
>
> Side-Inputs are non-deterministic for several reasons:
> 1. Because they depend on triggering of the side-input (this is acceptable 
> because triggers are by their nature non-deterministic).
> 2. They depend on the current state of the main-input window in order to 
> lookup the side-input. This means that with merging
> 3. Any runner optimizations that affect when the side-input is looked up may 
> cause problems with either or both of these.
> This issue focuses on #2 -- the non-determinism of side-inputs that execute 
> within a Merging WindowFn.
> Possible solution would be to defer running anything that looks up the 
> side-input until we need to extract an output, and using the main-window at 
> that point. Specifically, if the main-window is a MergingWindowFn, don't 
> execute any kind of pre-combine, instead buffer all the inputs and combine 
> later.
> This could still run into some non-determinism if there are triggers 
> controlling when we extract output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-743) JdbcIO should not use getConnection() with username and password when using DBCP BasicDataSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568579#comment-15568579
 ] 

ASF GitHub Bot commented on BEAM-743:
-

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1084

[BEAM-743] JdbcIO deals right getConnection() to use with DBCP 
BasicDataSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam 
BEAM-743-JDBCIO_DBCP_USERNAME

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1084.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1084


commit ee5a0b1445a3f3f57535471baf53146a8df66884
Author: Jean-Baptiste Onofré 
Date:   2016-10-12T12:23:51Z

[BEAM-743] JdbcIO deals right getConnection() to use with DBCP 
BasicDataSource




> JdbcIO should not use getConnection() with username and password when using 
> DBCP BasicDataSource
> 
>
> Key: BEAM-743
> URL: https://issues.apache.org/jira/browse/BEAM-743
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> When we use an username and password, {{JdbcIO}} uses 
> {{dataSource.getConnection(username, password)}}.
> However, this method is not supported by DBCP {{BasicDataSource}} (as the 
> username and password are already part of the data source).
> Basically, the test should be if username is not null and if datasource (user 
> provided) is not null, then we use {{getConnection(username, password)}} 
> else, we just use {{getConnection()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #1084: [BEAM-743] JdbcIO deals right getConnecti...

[jbonofre]

GitHub user jbonofre opened a pull request:

https://github.com/apache/incubator-beam/pull/1084

[BEAM-743] JdbcIO deals right getConnection() to use with DBCP 
BasicDataSource

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [X] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [X] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [X] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [X] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbonofre/incubator-beam 
BEAM-743-JDBCIO_DBCP_USERNAME

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1084.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1084


commit ee5a0b1445a3f3f57535471baf53146a8df66884
Author: Jean-Baptiste Onofré 
Date:   2016-10-12T12:23:51Z

[BEAM-743] JdbcIO deals right getConnection() to use with DBCP 
BasicDataSource




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-556) typo in documentation

[Neelesh Srinivas Salian (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15568497#comment-15568497
 ] 

Neelesh Srinivas Salian commented on BEAM-556:
--

[~fyellin] could you please close the JIRA since it has been resolved.

> typo in documentation
> -
>
> Key: BEAM-556
> URL: https://issues.apache.org/jira/browse/BEAM-556
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Frank Yellin
>Assignee: Frank Yellin
>Priority: Trivial
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> transform.py:
> ergument -> argument  
> in documentation for parse_label_and_args



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-743) JdbcIO should not use getConnection() with username and password when using DBCP BasicDataSource

[JIRA]

Jean-Baptiste Onofré created BEAM-743:
-

 Summary: JdbcIO should not use getConnection() with username and 
password when using DBCP BasicDataSource
 Key: BEAM-743
 URL: https://issues.apache.org/jira/browse/BEAM-743
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Jean-Baptiste Onofré
Assignee: Jean-Baptiste Onofré


When we use an username and password, {{JdbcIO}} uses 
{{dataSource.getConnection(username, password)}}.
However, this method is not supported by DBCP {{BasicDataSource}} (as the 
username and password are already part of the data source).

Basically, the test should be if username is not null and if datasource (user 
provided) is not null, then we use {{getConnection(username, password)}} else, 
we just use {{getConnection()}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)