[GitHub] incubator-beam pull request #936: Update Beam examples archetypes

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/incubator-beam/pull/936

Update Beam examples archetypes



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam archetypes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #936


commit e25fb28b36d8f6acc3ff9c48e7a1d82a321c101e
Author: Pei He 
Date:   2016-09-09T02:16:12Z

Update Beam examples archetypes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-620) DataflowPipelineJob IllegalArgumentException from time underflow

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-620.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> DataflowPipelineJob IllegalArgumentException from time underflow
> 
>
> Key: BEAM-620
> URL: https://issues.apache.org/jira/browse/BEAM-620
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: 
> maxCumulativeBackoff PT-0.072S must be at least 1 millisecond
> at 
> com.google.cloud.dataflow.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
> at 
> com.google.cloud.dataflow.sdk.util.FluentBackoff.withMaxCumulativeBackoff(FluentBackoff.java:129)
> at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.waitToFinish(DataflowPipelineJob.java:266)
> at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.waitToFinish(DataflowPipelineJob.java:191)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-620) DataflowPipelineJob IllegalArgumentException from time underflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475500#comment-15475500
 ] 

ASF GitHub Bot commented on BEAM-620:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/932


> DataflowPipelineJob IllegalArgumentException from time underflow
> 
>
> Key: BEAM-620
> URL: https://issues.apache.org/jira/browse/BEAM-620
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>Priority: Minor
>
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: 
> maxCumulativeBackoff PT-0.072S must be at least 1 millisecond
> at 
> com.google.cloud.dataflow.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
> at 
> com.google.cloud.dataflow.sdk.util.FluentBackoff.withMaxCumulativeBackoff(FluentBackoff.java:129)
> at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.waitToFinish(DataflowPipelineJob.java:266)
> at 
> com.google.cloud.dataflow.sdk.runners.DataflowPipelineJob.waitToFinish(DataflowPipelineJob.java:191)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: Closes #932

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 7fcc9444e -> 817515fe4


Closes #932


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/817515fe
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/817515fe
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/817515fe

Branch: refs/heads/master
Commit: 817515fe479b06bcd58c72cc1797255572f10f68
Parents: 7fcc944 ed3b12a
Author: Dan Halperin 
Authored: Thu Sep 8 17:56:43 2016 -0700
Committer: Dan Halperin 
Committed: Thu Sep 8 17:56:43 2016 -0700

--
 .../runners/dataflow/DataflowPipelineJob.java   | 26 +++
 .../dataflow/DataflowPipelineJobTest.java   | 46 
 2 files changed, 63 insertions(+), 9 deletions(-)
--

[2/2] incubator-beam git commit: DataflowPipelineJob: catch an underflow in backoff code

[dhalperi]

DataflowPipelineJob: catch an underflow in backoff code

Forward port of https://github.com/GoogleCloudPlatform/DataflowJavaSDK/pull/422


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ed3b12ab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ed3b12ab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ed3b12ab

Branch: refs/heads/master
Commit: ed3b12ab764d7867813957f22b67a518d5140ecd
Parents: 7fcc944
Author: Daniel Halperin 
Authored: Wed Sep 7 16:57:26 2016 -0700
Committer: Dan Halperin 
Committed: Thu Sep 8 17:56:43 2016 -0700

--
 .../runners/dataflow/DataflowPipelineJob.java   | 26 +++
 .../dataflow/DataflowPipelineJobTest.java   | 46 
 2 files changed, 63 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ed3b12ab/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
index dad59f2..1af8c98 100644
--- 
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
+++ 
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java
@@ -263,19 +263,27 @@ public class DataflowPipelineJob implements 
PipelineResult {
   }
 
   if (!hasError) {
-// Reset the backoff.
+// We can stop if the job is done.
+if (state.isTerminal()) {
+  return state;
+}
+
+// The job is not done, so we must keep polling.
 backoff.reset();
-// If duration is set, update the new cumulative sleep time to be the 
remaining
-// part of the total input sleep duration.
+
+// If a total duration for all backoff has been set, update the new 
cumulative sleep time to
+// be the remaining total backoff duration, stopping if we have 
already exceeded the
+// allotted time.
 if (duration.isLongerThan(Duration.ZERO)) {
   long nanosConsumed = nanoClock.nanoTime() - startNanos;
   Duration consumed = Duration.millis((nanosConsumed + 99) / 
100);
-  backoff =
-  
MESSAGES_BACKOFF_FACTORY.withMaxCumulativeBackoff(duration.minus(consumed)).backoff();
-}
-// Check if the job is done.
-if (state.isTerminal()) {
-  return state;
+  Duration remaining = duration.minus(consumed);
+  if (remaining.isLongerThan(Duration.ZERO)) {
+backoff = 
MESSAGES_BACKOFF_FACTORY.withMaxCumulativeBackoff(remaining).backoff();
+  } else {
+// If there is no time remaining, don't bother backing off.
+backoff = BackOff.STOP_BACKOFF;
+  }
 }
   }
 } while(BackOffUtils.next(sleeper, backoff));

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ed3b12ab/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
--
diff --git 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
index 226140a..4c70d12 100644
--- 
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
+++ 
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowPipelineJobTest.java
@@ -32,6 +32,8 @@ import static org.mockito.Matchers.eq;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
 
+import com.google.api.client.util.NanoClock;
+import com.google.api.client.util.Sleeper;
 import com.google.api.services.dataflow.Dataflow;
 import com.google.api.services.dataflow.Dataflow.Projects.Jobs.Get;
 import com.google.api.services.dataflow.Dataflow.Projects.Jobs.GetMetrics;
@@ -46,6 +48,7 @@ import com.google.common.collect.ImmutableSetMultimap;
 import java.io.IOException;
 import java.math.BigDecimal;
 import java.net.SocketTimeoutException;
+import java.util.concurrent.ThreadLocalRandom;
 import java.util.concurrent.TimeUnit;
 import org.apache.beam.runners.dataflow.internal.DataflowAggregatorTransforms;
 import org.apache.beam.runners.dataflow.testing.TestDataflowPipelineOptions;
@@ -249,6 +252,30 @@ 

[jira] [Commented] (BEAM-618) Python SDKs writes non RFC compliant JSON files for BQ Export

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475458#comment-15475458
 ] 

ASF GitHub Bot commented on BEAM-618:
-

Github user ajamato closed the pull request at:

https://github.com/apache/incubator-beam/pull/935


> Python SDKs writes non RFC compliant JSON files for BQ Export
> -
>
> Key: BEAM-618
> URL: https://issues.apache.org/jira/browse/BEAM-618
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Alex Amato
>Assignee: Frances Perry
>
> Python SDK uses the built in json.dumps to write JSON files to GCS for the BQ 
> Exporter. BigQuery can fail to parse these files when it tries to load these 
> files into a BQ table because json.dumps can export JSON which does not 
> conform to the IEEE RFC.
> There are a few cases which are not RFC compilant listed in that module.
> https://docs.python.org/2/library/json.html#standard-compliance-and-interoperability
> The main issue we run into is the NAN, INF and -INF values.
> These fails with a confusing error (and we delete the GCS files making it 
> hard to debug):
> JSON table encountered too many errors, giving up. Rows JSON parsing error in 
> row starting at position
> We can set the allow_nan argument to json.dumps to false to address these 
> issues. So that when a user tries to write a file with INF, -INF or NAN
> Setting this argument will produce this type of error when json.dumps is 
> called with NAN/INF values. We may want to catch this error to mention the 
> fact that INF and NAN are not allowed.
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
> sort_keys=sort_keys, **kw).encode(obj)
>   File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
> chunks = self.iterencode(o, _one_shot=True)
>   File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
> return _iterencode(o, 0)
> ValueError: Out of range float values are not JSON compliant



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #935: [BEAM-618] Disallow NAN, INF and -INF inva...

[ajamato]

GitHub user ajamato opened a pull request:

https://github.com/apache/incubator-beam/pull/935

[BEAM-618] Disallow NAN, INF and -INF invalid JSON values in bigquery 
exporter




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/incubator-beam python-sdk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #935


commit 11d78a4c1791c1dfd88f0ac348c9c07cd48cafc8
Author: Ian Zhou 
Date:   2016-06-09T21:17:14Z

Modified range tracker to use first response seen as start key

commit ec6d88a787dfdab064bceb70d48b2ce1c5bfa9bb
Author: Thomas Groh 
Date:   2016-06-14T01:34:49Z

Reuse UnboundedReaders in the InProcessRunner

Reuse up to a point, and then discard the reader to exercise resume from
checkpoint.

commit d2ceaf5e5a778fad18472ab0d7c02a14259015d7
Author: Scott Wegner 
Date:   2016-06-14T16:00:49Z

Update DataflowPipelineRunner worker container version

commit 90bb20ee6738c57bc25f47e2d80690fb721b562e
Author: Thomas Groh 
Date:   2016-06-14T22:49:34Z

Explicitly set the Runner in TestFlinkPipelineRunner

This ensures that the created PipelineOptions are valid if the
DirectRunner is not on the classpath.

commit 45e57e0612ae692418e07d9c4483321f040cb4a7
Author: Thomas Groh 
Date:   2016-06-15T00:51:48Z

Remove DoFnRunner from GroupAlsoByWindowsProperties

DoFnRunner is a runner implementation detail, and core SDK code should
instead use DoFnTester.

commit 99654ca4bed6758d7128d0f0ad376e8b479d4eba
Author: Thomas Groh 
Date:   2016-06-15T00:52:49Z

Remove the DirectPipelineRunner from the Core SDK

commit d5e3dfaa864744ec9a011c51707d15f1ab68a734
Author: Scott Wegner 
Date:   2016-06-15T16:51:59Z

Fix NullPointerException in AfterWatermark display data

Window transforms register display data for the associated trigger
function by calling its .toString() method. The AfterWatermark
trigger .toString() method was not properly handling cases where
there is no late firings registered.

commit 340fe3ebcfef0b57b163483d7d7243ad5456ae72
Author: Scott Wegner 
Date:   2016-06-15T17:17:01Z

Package javadoc for org.apache.beam.sdk.transforms.display

commit 6ada1a635382fcddc42a7580e74e755839f7172e
Author: Thomas Groh 
Date:   2016-06-15T19:01:56Z

Run NeedsRunner tests in Runner Core on the DirectRunner

This ensures that all runner tests in runners/core-java are executed in
the standard maven build.

commit e90a1b9d74cbc06d7818bae8dfe2af81acd73222
Author: Kenneth Knowles 
Date:   2016-06-08T22:07:52Z

Roll-forwards: Base PAssert on GBK instead of side inputs

Previously PAssert - hence all RunnableOnService/NeedsRunner
tests - required side input support. This created a very steep
on ramp for new runners.

GroupByKey is a bit more fundamental and most backends will be
able to group by key in the global window very quickly. So switching
the primitive used to gather all the contents of a PCollection for
assertions should make it a bit easier to get early feedback during
runner development.

commit 0a7246d268969cb1b7f46149e38361802c95e70a
Author: Scott Wegner 
Date:   2016-06-13T18:05:52Z

Improve BigQueryIO validation for streaming WriteDisposition

commit 605833071a7034aa3b723776a0f9e24330f64c8b
Author: Pei He 
Date:   2016-06-13T23:58:01Z

Replace GcsPath by IOChannelFactory in WordCount.

commit 5bf732cd3e598321a5c51e1239eda0fe2877a65d
Author: Kenneth Knowles 
Date:   2016-06-14T23:04:10Z

Add test for ReduceFnRunner GC time overflow

commit cfa217a894575f392f1dfe1612e10e393df5c7ab
Author: Kenneth Knowles 
Date:   2016-06-14T23:12:11Z

Fix type error in Eclipse

This type error occurs in my Eclipse installation. It apparently
does not bother the various JDKs we test with. But this is an
accurate typing, so it may help other Eclipse-using contributors,
too.

commit 8278e5f78f36fb48fae994ee7abcc1485db84189
Author: Kenneth Knowles 
Date:   2016-06-15T17:42:59Z

[Spark] Elide assigning windows when WindowFn is null

Previously, when translating a Window.Bound transform, the case
where the WindowFn was null was missed, resulting in a
NullPointerException.

commit 9400fc9a699f218a7948c21639428f5f00134ec5
Author: Thomas Groh 
Date:   2016-06-15T17:45:15Z

Rename InProcessPipelineRunner to DirectRunner

Completes BEAM-243

commit 

[jira] [Commented] (BEAM-618) Python SDKs writes non RFC compliant JSON files for BQ Export

[Alex Amato (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475443#comment-15475443
 ] 

Alex Amato commented on BEAM-618:
-

How do I assign to me? I am going to send a change for review for this

> Python SDKs writes non RFC compliant JSON files for BQ Export
> -
>
> Key: BEAM-618
> URL: https://issues.apache.org/jira/browse/BEAM-618
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Alex Amato
>Assignee: Frances Perry
>
> Python SDK uses the built in json.dumps to write JSON files to GCS for the BQ 
> Exporter. BigQuery can fail to parse these files when it tries to load these 
> files into a BQ table because json.dumps can export JSON which does not 
> conform to the IEEE RFC.
> There are a few cases which are not RFC compilant listed in that module.
> https://docs.python.org/2/library/json.html#standard-compliance-and-interoperability
> The main issue we run into is the NAN, INF and -INF values.
> These fails with a confusing error (and we delete the GCS files making it 
> hard to debug):
> JSON table encountered too many errors, giving up. Rows JSON parsing error in 
> row starting at position
> We can set the allow_nan argument to json.dumps to false to address these 
> issues. So that when a user tries to write a file with INF, -INF or NAN
> Setting this argument will produce this type of error when json.dumps is 
> called with NAN/INF values. We may want to catch this error to mention the 
> fact that INF and NAN are not allowed.
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
> sort_keys=sort_keys, **kw).encode(obj)
>   File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
> chunks = self.iterencode(o, _one_shot=True)
>   File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
> return _iterencode(o, 0)
> ValueError: Out of range float values are not JSON compliant



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-603) Update python sdk README examples

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

María GH closed BEAM-603.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Update python sdk README examples
> -
>
> Key: BEAM-603
> URL: https://issues.apache.org/jira/browse/BEAM-603
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
>  Labels: documentation
> Fix For: Not applicable
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Udate examples and text at:
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/README.md



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-544) Add header/footer support to TextIO.Write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474559#comment-15474559
 ] 

ASF GitHub Bot commented on BEAM-544:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/934


> Add header/footer support to TextIO.Write
> -
>
> Key: BEAM-544
> URL: https://issues.apache.org/jira/browse/BEAM-544
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Being able to add a header/footer to each file that is written via TextIO 
> would cover several simple text file format issues.
> Original ask:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: !fixup Fix missed @Nullable annotations

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 8b39a2ea3 -> 7fcc9444e


!fixup Fix missed @Nullable annotations


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7aac4c3e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7aac4c3e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7aac4c3e

Branch: refs/heads/master
Commit: 7aac4c3e2618f8592eaf4c8e877e5764e708710d
Parents: 8b39a2e
Author: Luke Cwik 
Authored: Thu Sep 8 10:52:24 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 8 10:52:24 2016 -0700

--
 .../java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/7aac4c3e/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
index 3345cd3..79967d1 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
@@ -511,9 +511,9 @@ public class TextIO {
 this(null, null, "", null, null, coder, 0, DEFAULT_SHARD_TEMPLATE, 
true);
   }
 
-  private Bound(String name, String filenamePrefix, String filenameSuffix, 
String header,
-String footer, Coder coder, int numShards, String 
shardTemplate,
-boolean validate) {
+  private Bound(String name, String filenamePrefix, String filenameSuffix,
+  @Nullable String header, @Nullable String footer, Coder coder, 
int numShards,
+  String shardTemplate, boolean validate) {
 super(name);
 this.header = header;
 this.footer = footer;

[2/2] incubator-beam git commit: [BEAM-544] Address comments during backport Dataflow PR/423 for Apache Beam PR/918

[lcwik]

[BEAM-544] Address comments during backport Dataflow PR/423 for Apache Beam 
PR/918

This closes #934


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/7fcc9444
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/7fcc9444
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/7fcc9444

Branch: refs/heads/master
Commit: 7fcc9444efd02d4a93f54e338ed97f5340ae1d77
Parents: 8b39a2e 7aac4c3
Author: Luke Cwik 
Authored: Thu Sep 8 10:53:18 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 8 10:53:18 2016 -0700

--
 .../java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--

[GitHub] incubator-beam pull request #934: [BEAM-544] Address comments during backpor...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/934


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-544) Add header/footer support to TextIO.Write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474476#comment-15474476
 ] 

ASF GitHub Bot commented on BEAM-544:
-

GitHub user lukecwik reopened a pull request:

https://github.com/apache/incubator-beam/pull/934

[BEAM-544] Address comments during backport Dataflow PR/423 for Apache Beam 
PR/918

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam fixup-pr-918

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #934


commit fea99b0466646e553f773d1bfc0a6553c48b49b4
Author: Luke Cwik 
Date:   2016-09-08T16:34:43Z

Address comments during backport Dataflow PR/423 for Apache Beam PR/918

commit facabfca910cb215c7044b97becf58dcb4fd933b
Author: Luke Cwik 
Date:   2016-09-08T16:40:19Z

!fixup Fix changes done by IDE. new line -> newline




> Add header/footer support to TextIO.Write
> -
>
> Key: BEAM-544
> URL: https://issues.apache.org/jira/browse/BEAM-544
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Being able to add a header/footer to each file that is written via TextIO 
> would cover several simple text file format issues.
> Original ask:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #934: [BEAM-544] Address comments during backpor...

[lukecwik]

GitHub user lukecwik reopened a pull request:

https://github.com/apache/incubator-beam/pull/934

[BEAM-544] Address comments during backport Dataflow PR/423 for Apache Beam 
PR/918

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam fixup-pr-918

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #934


commit fea99b0466646e553f773d1bfc0a6553c48b49b4
Author: Luke Cwik 
Date:   2016-09-08T16:34:43Z

Address comments during backport Dataflow PR/423 for Apache Beam PR/918

commit facabfca910cb215c7044b97becf58dcb4fd933b
Author: Luke Cwik 
Date:   2016-09-08T16:40:19Z

!fixup Fix changes done by IDE. new line -> newline




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #934: [BEAM-544] Address comments during backpor...

[lukecwik]

Github user lukecwik closed the pull request at:

https://github.com/apache/incubator-beam/pull/934


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-544) Add header/footer support to TextIO.Write

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474468#comment-15474468
 ] 

ASF GitHub Bot commented on BEAM-544:
-

Github user lukecwik closed the pull request at:

https://github.com/apache/incubator-beam/pull/934


> Add header/footer support to TextIO.Write
> -
>
> Key: BEAM-544
> URL: https://issues.apache.org/jira/browse/BEAM-544
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Stas Levin
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Being able to add a header/footer to each file that is written via TextIO 
> would cover several simple text file format issues.
> Original ask:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

incubator-beam git commit: Address comments during backport Dataflow PR/423 for Apache Beam PR/918

[lcwik]

Repository: incubator-beam
Updated Branches:
  refs/heads/master fb322cc73 -> 8b39a2ea3


Address comments during backport Dataflow PR/423 for Apache Beam PR/918


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/8b39a2ea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/8b39a2ea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/8b39a2ea

Branch: refs/heads/master
Commit: 8b39a2ea3e2a70b9c07c91c375f79b9f966ab288
Parents: fb322cc
Author: Luke Cwik 
Authored: Thu Sep 8 09:34:43 2016 -0700
Committer: Luke Cwik 
Committed: Thu Sep 8 10:18:49 2016 -0700

--
 .../java/org/apache/beam/sdk/io/TextIO.java | 71 ++--
 .../java/org/apache/beam/sdk/io/TextIOTest.java | 13 ++--
 2 files changed, 40 insertions(+), 44 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/8b39a2ea/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
index c0761b1..3345cd3 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
@@ -450,25 +450,25 @@ public class TextIO {
 
 /**
  * Returns a transform for writing to text files that adds a header string 
to the files
- * it writes.
+ * it writes. Note that a newline character will be added after the header.
  *
  * A {@code null} value will clear any previously configured header.
  *
  * @param header the string to be added as file header
  */
-public static Bound withHeader(String header) {
+public static Bound withHeader(@Nullable String header) {
   return new Bound<>(DEFAULT_TEXT_CODER).withHeader(header);
 }
 
 /**
  * Returns a transform for writing to text files that adds a footer string 
to the files
- * it writes.
+ * it writes. Note that a newline character will be added after the header.
  *
  * A {@code null} value will clear any previously configured footer.
  *
  * @param footer the string to be added as file footer
  */
-public static Bound withFooter(String footer) {
+public static Bound withFooter(@Nullable String footer) {
   return new Bound<>(DEFAULT_TEXT_CODER).withFooter(footer);
 }
 
@@ -490,10 +490,10 @@ public class TextIO {
   private final String filenameSuffix;
 
   /** An optional header to add to each file. */
-  private final String header;
+  @Nullable private final String header;
 
   /** An optional footer to add to each file. */
-  private final String footer;
+  @Nullable private final String footer;
 
   /** The Coder to use to decode each line. */
   private final Coder coder;
@@ -634,7 +634,7 @@ public class TextIO {
 
   /**
* Returns a transform for writing to text files that adds a header 
string to the files
-   * it writes.
+   * it writes. Note that a newline character will be added after the 
header.
*
* A {@code null} value will clear any previously configured header.
*
@@ -642,14 +642,14 @@ public class TextIO {
*
* @param header the string to be added as file header
*/
-  public Bound withHeader(String header) {
+  public Bound withHeader(@Nullable String header) {
 return new Bound<>(name, filenamePrefix, filenameSuffix, header, 
footer, coder, numShards,
 shardTemplate, false);
   }
 
   /**
* Returns a transform for writing to text files that adds a footer 
string to the files
-   * it writes.
+   * it writes. Note that a newline character will be added after the 
header.
*
* A {@code null} value will clear any previously configured footer.
*
@@ -657,7 +657,7 @@ public class TextIO {
*
* @param footer the string to be added as file footer
*/
-  public Bound withFooter(String footer) {
+  public Bound withFooter(@Nullable String footer) {
 return new Bound<>(name, filenamePrefix, filenameSuffix, header, 
footer, coder, numShards,
 shardTemplate, false);
   }
@@ -733,10 +733,12 @@ public class TextIO {
 return coder;
   }
 
+  @Nullable
   public String getHeader() {
 return header;
   }
 
+  @Nullable
   public String getFooter() {
 return footer;
   }
@@ -805,7 +807,7 @@ public class TextIO {
   private TextIO() {}
 
   /**
-   * A {@link FileBasedSource} which can decode records delimited by new line 
characters.
+   

[GitHub] incubator-beam pull request #934: Address comments during backport Dataflow ...

[lukecwik]

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/934

Address comments during backport Dataflow PR/423 for Apache Beam PR/918

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam fixup-pr-918

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #934


commit fea99b0466646e553f773d1bfc0a6553c48b49b4
Author: Luke Cwik 
Date:   2016-09-08T16:34:43Z

Address comments during backport Dataflow PR/423 for Apache Beam PR/918

commit facabfca910cb215c7044b97becf58dcb4fd933b
Author: Luke Cwik 
Date:   2016-09-08T16:40:19Z

!fixup Fix changes done by IDE. new line -> newline




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #933: Update 404 link to setuptools docs

[swegner]

GitHub user swegner opened a pull request:

https://github.com/apache/incubator-beam/pull/933

Update 404 link to setuptools docs

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/swegner/incubator-beam setuptools_link

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #933


commit 6d0507a33e211cdfeba8a216afdd336f016df8fc
Author: Scott Wegner 
Date:   2016-09-08T16:32:14Z

Update 404 link to setuptools docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam-site pull request #39: [BEAM-102] update capability matrix

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/39


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-102) Side Inputs for Streaming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474155#comment-15474155
 ] 

ASF GitHub Bot commented on BEAM-102:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam-site/pull/39


> Side Inputs for Streaming
> -
>
> Key: BEAM-102
> URL: https://issues.apache.org/jira/browse/BEAM-102
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 0.3.0-incubating
>
>
> The Flink Runner supports side inputs for batch mode but its missing support 
> for streaming.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/3] incubator-beam-site git commit: [BEAM-102] update capability matrix

[mxm]

Repository: incubator-beam-site
Updated Branches:
  refs/heads/asf-site e2430eb4d -> dcdd8b742


[BEAM-102] update capability matrix


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/8459da13
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/8459da13
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/8459da13

Branch: refs/heads/asf-site
Commit: 8459da13fccbd16e850ea455873812eea974b6dc
Parents: e2430eb
Author: Maximilian Michels 
Authored: Mon Sep 5 13:04:49 2016 +0200
Committer: Maximilian Michels 
Committed: Thu Sep 8 17:24:27 2016 +0200

--
 _data/capability-matrix.yml | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam-site/blob/8459da13/_data/capability-matrix.yml
--
diff --git a/_data/capability-matrix.yml b/_data/capability-matrix.yml
index e2f66b9..89da73b 100644
--- a/_data/capability-matrix.yml
+++ b/_data/capability-matrix.yml
@@ -118,12 +118,11 @@ categories:
   - class: dataflow
 l1: 'Yes'
 l2: some size restrictions in streaming
-l3: Batch implemented supports a distributed implementation, but 
streaming mode may force some size restrictions. Neither mode is able to push 
lookups directly up into key-based sources.
+l3: Batch mode supports a distributed implementation, but 
streaming mode may force some size restrictions. Neither mode is able to push 
lookups directly up into key-based sources.
   - class: flink
-jira: BEAM-102
-l1: 'Partially'
-l2: no supported in streaming
-l3: Supported in batch. Side inputs for streaming are currently 
WiP.
+l1: 'Yes'
+l2: some size restrictions in streaming
+l3: Batch mode supports a distributed implementation, but 
streaming mode may force some size restrictions. Neither mode is able to push 
lookups directly up into key-based sources.
   - class: spark
 l1: 'Partially'
 l2: not supported in streaming

[3/3] incubator-beam-site git commit: This closes #39

[mxm]

This closes #39


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/repo
Commit: 
http://git-wip-us.apache.org/repos/asf/incubator-beam-site/commit/dcdd8b74
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/tree/dcdd8b74
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam-site/diff/dcdd8b74

Branch: refs/heads/asf-site
Commit: dcdd8b742bd5f1463815638ef1c33aca3a523308
Parents: e2430eb bb1106b
Author: Maximilian Michels 
Authored: Thu Sep 8 17:24:51 2016 +0200
Committer: Maximilian Michels 
Committed: Thu Sep 8 17:24:51 2016 +0200

--
 _data/capability-matrix.yml| 9 -
 content/learn/runners/capability-matrix/index.html | 8 
 2 files changed, 8 insertions(+), 9 deletions(-)
--

[1/4] incubator-beam git commit: [BEAM-616] Update Flink Runner to Flink 1.1.2

[aljoscha]

Repository: incubator-beam
Updated Branches:
  refs/heads/master f33296c7f -> fb322cc73


[BEAM-616] Update Flink Runner to Flink 1.1.2


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c66caf33
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c66caf33
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c66caf33

Branch: refs/heads/master
Commit: c66caf3364ae5c20bdf0fbf41a8ef61d4e53c495
Parents: f33296c
Author: Aljoscha Krettek 
Authored: Mon Sep 5 18:17:11 2016 +0200
Committer: Aljoscha Krettek 
Committed: Thu Sep 8 10:10:50 2016 +0200

--
 runners/flink/pom.xml   |  2 +-
 runners/flink/runner/pom.xml|  8 +++
 .../wrappers/streaming/DoFnOperator.java| 24 
 .../wrappers/streaming/FlinkStateInternals.java |  8 ---
 .../wrappers/streaming/WindowDoFnOperator.java  |  4 ++--
 5 files changed, 30 insertions(+), 16 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c66caf33/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index b2f3aaa..68e82d2 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -39,7 +39,7 @@
   
 UTF-8
 UTF-8
-1.0.3
+1.1.2
   
 
   

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c66caf33/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index 7c32280..8759591 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -127,6 +127,14 @@
   test
 
 
+
+  org.apache.flink
+  flink-runtime_2.10
+  ${flink.version}
+  test-jar
+  test
+
+
 
 
 

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/c66caf33/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 3b917e2..79aab9c 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -199,7 +199,7 @@ public class DoFnOperator
   if (restoredSideInputState != null) {
 @SuppressWarnings("unchecked,rawtypes")
 HashMap castRestored = (HashMap) 
restoredSideInputState;
-sideInputStateBackend.injectKeyValueStateSnapshots(castRestored, 0L);
+sideInputStateBackend.injectKeyValueStateSnapshots(castRestored);
 restoredSideInputState = null;
   }
 
@@ -306,15 +306,19 @@ public class DoFnOperator
 pushedBackDescriptor);
 
 List newPushedBack = new ArrayList<>();
-for (WindowedValue elem: pushedBack.get()) {
 
-  // we need to set the correct key in case the operator is
-  // a (keyed) window operator
-  setKeyContextElement1(new StreamRecord<>(elem));
+Iterable pushedBackContents = pushedBack.get();
+if (pushedBackContents != null) {
+  for (WindowedValue elem : pushedBackContents) {
 
-  Iterable justPushedBack =
-  pushbackDoFnRunner.processElementInReadyWindows(elem);
-  Iterables.addAll(newPushedBack, justPushedBack);
+// we need to set the correct key in case the operator is
+// a (keyed) window operator
+setKeyContextElement1(new StreamRecord<>(elem));
+
+Iterable justPushedBack =
+pushbackDoFnRunner.processElementInReadyWindows(elem);
+Iterables.addAll(newPushedBack, justPushedBack);
+  }
 }
 
 
@@ -385,8 +389,8 @@ public class DoFnOperator
   }
 
   @Override
-  public void restoreState(StreamTaskState state, long recoveryTimestamp) 
throws Exception {
-super.restoreState(state, recoveryTimestamp);
+  public void restoreState(StreamTaskState state) throws Exception {
+super.restoreState(state);
 
 @SuppressWarnings("unchecked,rawtypes")
 StateHandle> 
sideInputStateHandle =

[GitHub] incubator-beam pull request #922: Fix shaded imports in Flink Runner

[aljoscha]

Github user aljoscha closed the pull request at:

https://github.com/apache/incubator-beam/pull/922


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/4] incubator-beam git commit: Merge branch 'flink-1.1.2'

[aljoscha]

Merge branch 'flink-1.1.2'

This closes #921


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/31d09eb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/31d09eb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/31d09eb4

Branch: refs/heads/master
Commit: 31d09eb45c4fcd5f88012145e64734785fe4c022
Parents: f33296c c66caf3
Author: Aljoscha Krettek 
Authored: Thu Sep 8 10:11:02 2016 +0200
Committer: Aljoscha Krettek 
Committed: Thu Sep 8 10:11:02 2016 +0200

--
 runners/flink/pom.xml   |  2 +-
 runners/flink/runner/pom.xml|  8 +++
 .../wrappers/streaming/DoFnOperator.java| 24 
 .../wrappers/streaming/FlinkStateInternals.java |  8 ---
 .../wrappers/streaming/WindowDoFnOperator.java  |  4 ++--
 5 files changed, 30 insertions(+), 16 deletions(-)
--

[3/4] incubator-beam git commit: Fix shaded imports in Flink Runner

[aljoscha]

Fix shaded imports in Flink Runner


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/27259599
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/27259599
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/27259599

Branch: refs/heads/master
Commit: 2725959996d131b6d189f7f62df3d05e0d361ba0
Parents: 31d09eb
Author: Aljoscha Krettek 
Authored: Mon Sep 5 10:50:14 2016 +0200
Committer: Aljoscha Krettek 
Committed: Thu Sep 8 10:12:06 2016 +0200

--
 .../flink/translation/wrappers/streaming/DoFnOperator.java  | 2 +-
 .../beam/runners/flink/streaming/DoFnOperatorTest.java  | 9 +
 2 files changed, 6 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/27259599/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index 79aab9c..ee23ae4 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -17,7 +17,7 @@
  */
 package org.apache.beam.runners.flink.translation.wrappers.streaming;
 
-import avro.shaded.com.google.common.base.Preconditions;
+import com.google.common.base.Preconditions;
 import com.google.common.collect.Iterables;
 import java.io.IOException;
 import java.io.Serializable;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/27259599/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
--
diff --git 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
index 5f1b066..913fb8b 100644
--- 
a/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
+++ 
b/runners/flink/runner/src/test/java/org/apache/beam/runners/flink/streaming/DoFnOperatorTest.java
@@ -20,7 +20,12 @@ package org.apache.beam.runners.flink.streaming;
 import static org.hamcrest.collection.IsIterableContainingInOrder.contains;
 import static org.junit.Assert.assertThat;
 
+import com.google.common.base.Function;
+import com.google.common.base.Predicate;
+import com.google.common.collect.FluentIterable;
 import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+
 import java.util.Collections;
 import java.util.HashMap;
 import javax.annotation.Nullable;
@@ -40,10 +45,6 @@ import org.apache.beam.sdk.util.WindowedValue;
 import org.apache.beam.sdk.util.WindowingStrategy;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.TupleTag;
-import org.apache.flink.shaded.com.google.common.base.Function;
-import org.apache.flink.shaded.com.google.common.base.Predicate;
-import org.apache.flink.shaded.com.google.common.collect.FluentIterable;
-import org.apache.flink.shaded.com.google.common.collect.ImmutableMap;
 import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
 import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
 import org.apache.flink.streaming.util.TwoInputStreamOperatorTestHarness;

[4/4] incubator-beam git commit: Merge branch 'flink-fix-imports'

[aljoscha]

Merge branch 'flink-fix-imports'

This closes #922


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/fb322cc7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/fb322cc7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/fb322cc7

Branch: refs/heads/master
Commit: fb322cc73aac6ed42b45371751388270d4f0db16
Parents: 31d09eb 2725959
Author: Aljoscha Krettek 
Authored: Thu Sep 8 10:12:15 2016 +0200
Committer: Aljoscha Krettek 
Committed: Thu Sep 8 10:12:15 2016 +0200

--
 .../flink/translation/wrappers/streaming/DoFnOperator.java  | 2 +-
 .../beam/runners/flink/streaming/DoFnOperatorTest.java  | 9 +
 2 files changed, 6 insertions(+), 5 deletions(-)
--

[GitHub] incubator-beam pull request #921: [FLINK-616] Update Flink Runner to Flink 1...

[aljoscha]

Github user aljoscha closed the pull request at:

https://github.com/apache/incubator-beam/pull/921


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---