[jira] [Resolved] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-242.
---
Resolution: Fixed

> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-242:
--
Fix Version/s: 0.3.0-incubating

> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-242) Enable Checkstyle check and Javadoc build for the Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469466#comment-15469466
 ] 

ASF GitHub Bot commented on BEAM-242:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/919


> Enable Checkstyle check and Javadoc build for the Flink Runner 
> ---
>
> Key: BEAM-242
> URL: https://issues.apache.org/jira/browse/BEAM-242
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Maximilian Michels
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> We don't have a Checkstyle check in place for the Flink Runner. I would like 
> to use the SDK's checkstyle rules.
> We could also think about a unified Checkstyle for all Runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[1/2] incubator-beam git commit: [BEAM-242] Enable and fix checkstyle in Flink runner examples

[jbonofre]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 387854624 -> 26635d7fb


[BEAM-242] Enable and fix checkstyle in Flink runner examples


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/dafb8055
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/dafb8055
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/dafb8055

Branch: refs/heads/master
Commit: dafb80556c1d984630c6ccf615ba982903f176df
Parents: 3878546
Author: Jean-Baptiste Onofré 
Authored: Tue Sep 6 07:26:45 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed Sep 7 05:55:50 2016 +0200

--
 runners/flink/examples/pom.xml  |  2 --
 .../beam/runners/flink/examples/WordCount.java  |  9 ++
 .../runners/flink/examples/package-info.java| 22 +
 .../flink/examples/streaming/AutoComplete.java  |  5 +--
 .../flink/examples/streaming/JoinExamples.java  |  3 +-
 .../examples/streaming/KafkaIOExamples.java | 34 ++--
 .../KafkaWindowedWordCountExample.java  | 27 +---
 .../examples/streaming/WindowedWordCount.java   | 19 +++
 .../flink/examples/streaming/package-info.java  | 22 +
 9 files changed, 110 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafb8055/runners/flink/examples/pom.xml
--
diff --git a/runners/flink/examples/pom.xml b/runners/flink/examples/pom.xml
index 9f705db..b8489fc 100644
--- a/runners/flink/examples/pom.xml
+++ b/runners/flink/examples/pom.xml
@@ -109,12 +109,10 @@
 
   
 
-  
 
   
 org.apache.maven.plugins

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafb8055/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/WordCount.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/WordCount.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/WordCount.java
index ab9297f..9cce757 100644
--- 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/WordCount.java
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/WordCount.java
@@ -36,8 +36,14 @@ import org.apache.beam.sdk.transforms.Sum;
 import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PCollection;
 
+/**
+ * Wordcount pipeline.
+ */
 public class WordCount {
 
+  /**
+   * Function to extract words.
+   */
   public static class ExtractWordsFn extends DoFn {
 private final Aggregator emptyLines =
 createAggregator("emptyLines", new Sum.SumLongFn());
@@ -60,6 +66,9 @@ public class WordCount {
 }
   }
 
+  /**
+   * PTransform counting words.
+   */
   public static class CountWords extends PTransform>> {
 @Override

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafb8055/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/package-info.java
--
diff --git 
a/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/package-info.java
 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/package-info.java
new file mode 100644
index 000..b0ecb56
--- /dev/null
+++ 
b/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/package-info.java
@@ -0,0 +1,22 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/**
+ * Flink Beam runner exemple.
+ */
+package org.apache.beam.runners.flink.examples;

http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/dafb8055/runners/flink/examples/src/main/java/org/apache/beam/runners/flink/examples/streaming/AutoComplete.java

[2/2] incubator-beam git commit: [BEAM-242] This closes #919

[jbonofre]

[BEAM-242] This closes #919


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/26635d7f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/26635d7f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/26635d7f

Branch: refs/heads/master
Commit: 26635d7fb3d92185845d269909a3d399099df7da
Parents: 3878546 dafb805
Author: Jean-Baptiste Onofré 
Authored: Wed Sep 7 06:11:02 2016 +0200
Committer: Jean-Baptiste Onofré 
Committed: Wed Sep 7 06:11:02 2016 +0200

--
 runners/flink/examples/pom.xml  |  2 --
 .../beam/runners/flink/examples/WordCount.java  |  9 ++
 .../runners/flink/examples/package-info.java| 22 +
 .../flink/examples/streaming/AutoComplete.java  |  5 +--
 .../flink/examples/streaming/JoinExamples.java  |  3 +-
 .../examples/streaming/KafkaIOExamples.java | 34 ++--
 .../KafkaWindowedWordCountExample.java  | 27 +---
 .../examples/streaming/WindowedWordCount.java   | 19 +++
 .../flink/examples/streaming/package-info.java  | 22 +
 9 files changed, 110 insertions(+), 33 deletions(-)
--

[GitHub] incubator-beam pull request #919: [BEAM-242] Enable and fix checkstyle in Fl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/919


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : beam_PostCommit_RunnableOnService_GoogleCloudDataflow #1098

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-577) Update filebasedsource to support compressed files

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468784#comment-15468784
 ] 

ASF GitHub Bot commented on BEAM-577:
-

Github user sbilac closed the pull request at:

https://github.com/apache/incubator-beam/pull/912


> Update filebasedsource to support compressed files
> --
>
> Key: BEAM-577
> URL: https://issues.apache.org/jira/browse/BEAM-577
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> FileBasedSource framework [1] should be updated to properly read compressed 
> files.
> One possible way to do this might be to update FileBasedSource.open_file() 
> [2] to return a CompressedFile [3].
> Similar to Java implementation, we may not be able to support dynamic work 
> rebalancing for compressed files.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/filebasedsource.py
> [2] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/filebasedsource.py#L125
> [3] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L300



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #912: [BEAM-577] Add support for reading compres...

[sbilac]

Github user sbilac closed the pull request at:

https://github.com/apache/incubator-beam/pull/912


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #925

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master c5956318e -> 387854624


Closes #925


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/38785462
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/38785462
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/38785462

Branch: refs/heads/master
Commit: 38785462480aae2c3e2c46f65d6131c7b9b8fea3
Parents: c595631 88a6cf5
Author: Dan Halperin 
Authored: Tue Sep 6 15:09:46 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 6 15:09:46 2016 -0700

--
 .../java/org/apache/beam/sdk/coders/JAXBCoderTest.java | 13 +
 1 file changed, 5 insertions(+), 8 deletions(-)
--

[GitHub] incubator-beam pull request #925: Remove timeout in JAXBCoderTest

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/925


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] incubator-beam git commit: Remove timeout in JAXBCoderTest

[dhalperi]

Remove timeout in JAXBCoderTest

The infrastructure should kill the test if the await does not return.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/88a6cf5e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/88a6cf5e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/88a6cf5e

Branch: refs/heads/master
Commit: 88a6cf5e3793c9ecff664119be88349f94a37694
Parents: c595631
Author: Thomas Groh 
Authored: Tue Sep 6 11:13:07 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 6 15:09:46 2016 -0700

--
 .../java/org/apache/beam/sdk/coders/JAXBCoderTest.java | 13 +
 1 file changed, 5 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/88a6cf5e/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/JAXBCoderTest.java
--
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/JAXBCoderTest.java 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/JAXBCoderTest.java
index 129be61..36190f9 100644
--- a/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/JAXBCoderTest.java
+++ b/sdks/java/core/src/test/java/org/apache/beam/sdk/coders/JAXBCoderTest.java
@@ -18,7 +18,6 @@
 package org.apache.beam.sdk.coders;
 
 import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
 
 import com.google.common.collect.ImmutableList;
 import java.io.IOException;
@@ -28,7 +27,6 @@ import java.util.List;
 import java.util.concurrent.CountDownLatch;
 import java.util.concurrent.Executor;
 import java.util.concurrent.Executors;
-import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicReference;
 import javax.xml.bind.annotation.XmlRootElement;
 import org.apache.beam.sdk.testing.CoderProperties;
@@ -119,7 +117,7 @@ public class JAXBCoderTest {
   @Test
   public void testEncodeDecodeMultithreaded() throws Throwable {
 final JAXBCoder coder = JAXBCoder.of(TestType.class);
-int numThreads = 1000;
+int numThreads = 100;
 
 final CountDownLatch ready = new CountDownLatch(numThreads);
 final CountDownLatch start = new CountDownLatch(1);
@@ -155,11 +153,10 @@ public class JAXBCoderTest {
 ready.await();
 start.countDown();
 
-if (!done.await(10L, TimeUnit.SECONDS)) {
-  fail("Should be able to clone " + numThreads + " elements in 10 
seconds");
-}
-if (thrown.get() != null) {
-  throw thrown.get();
+done.await();
+Throwable actuallyThrown = thrown.get();
+if (actuallyThrown != null) {
+  throw actuallyThrown;
 }
   }
 

[GitHub] incubator-beam pull request #926: More changes to DoFn{Signatures,Invokers}.

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/incubator-beam/pull/926

More changes to DoFn{Signatures,Invokers}.

In preparation for Splittable DoFn.
* More generic code generation in DoFnInvokers:
  supports methods with return values (thanks @bjchambers).
* Uses AutoValue builder in DoFnSignature.
* Contextual error reporting in DoFnSignatures parsing code.
* Rewrote DoFnInvokers tests to use Mockito.
* Changed DoFnSignatures tests to use local classes
  and an "AnonymousMethod" class for testing analysis of
  single methods.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam do-fn-reflector

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/926.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #926


commit 9c88a510481df5cdfb229112cd35c364072c9e6f
Author: Eugene Kirpichov 
Date:   2016-08-12T00:13:53Z

More changes to DoFn{Signatures,Invokers}.

In preparation for Splittable DoFn.
* More generic code generation in DoFnInvokers:
  supports methods with return values (thanks @bjchambers).
* Uses AutoValue builder in DoFnSignature.
* Contextual error reporting in DoFnSignatures parsing code.
* Rewrote DoFnInvokers tests to use Mockito.
* Changed DoFnSignatures tests to use local classes
  and an "AnonymousMethod" class for testing analysis of
  single methods.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1276

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_MavenVerify » Apache Beam :: SDKs :: Java :: IO :: Google Cloud Platform #1275

[Apache Jenkins Server]

See 

[GitHub] incubator-beam pull request #914: Be more accepting in UnboundedReadDeduplic...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/914


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #914

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master 743a534a0 -> c5956318e


Closes #914


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/c5956318
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/c5956318
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/c5956318

Branch: refs/heads/master
Commit: c5956318e12f24a998624fec5476240a9657e784
Parents: 743a534 ae897c0
Author: Dan Halperin 
Authored: Tue Sep 6 11:33:18 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 6 11:33:18 2016 -0700

--
 .../runners/direct/UnboundedReadDeduplicatorTest.java| 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)
--

[2/2] incubator-beam git commit: Be more accepting in UnboundedReadDeduplicatorTest

[dhalperi]

Be more accepting in UnboundedReadDeduplicatorTest

Don't depend on all the threads failing. Instead, assert that at most
one success was encountered, and we saw at most numThreads - 1 failures.


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/ae897c06
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/ae897c06
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/ae897c06

Branch: refs/heads/master
Commit: ae897c063bab07dcfba08ce164898688b257b674
Parents: 743a534
Author: Thomas Groh 
Authored: Fri Sep 2 10:43:43 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 6 11:33:18 2016 -0700

--
 .../runners/direct/UnboundedReadDeduplicatorTest.java| 11 +++
 1 file changed, 7 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/ae897c06/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
--
diff --git 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
index 7d2a95c..0aa2c49 100644
--- 
a/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
+++ 
b/runners/direct-java/src/test/java/org/apache/beam/runners/direct/UnboundedReadDeduplicatorTest.java
@@ -19,6 +19,7 @@ package org.apache.beam.runners.direct;
 
 import static org.hamcrest.Matchers.equalTo;
 import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.lessThan;
 import static org.junit.Assert.assertThat;
 
 import java.util.concurrent.CountDownLatch;
@@ -60,18 +61,18 @@ public class UnboundedReadDeduplicatorTest {
 byte[] id = new byte[] {-1, 2, 4, 22};
 UnboundedReadDeduplicator dedupper = CachedIdDeduplicator.create();
 final CountDownLatch startSignal = new CountDownLatch(1);
-int numThreads = 1000;
+int numThreads = 50;
 final CountDownLatch readyLatch = new CountDownLatch(numThreads);
 final CountDownLatch finishLine = new CountDownLatch(numThreads);
 
 ExecutorService executor = Executors.newCachedThreadPool();
 AtomicInteger successCount = new AtomicInteger();
-AtomicInteger failureCount = new AtomicInteger();
+AtomicInteger noOutputCount = new AtomicInteger();
 for (int i = 0; i < numThreads; i++) {
   executor.submit(new TryOutputIdRunnable(dedupper,
   id,
   successCount,
-  failureCount,
+  noOutputCount,
   readyLatch,
   startSignal,
   finishLine));
@@ -82,8 +83,10 @@ public class UnboundedReadDeduplicatorTest {
 finishLine.await(10L, TimeUnit.SECONDS);
 executor.shutdownNow();
 
+// The first thread to run will succeed, and no others will
 assertThat(successCount.get(), equalTo(1));
-assertThat(failureCount.get(), equalTo(numThreads - 1));
+// The threads may not all complete; all of the threads that do not 
succeed must not output
+assertThat(noOutputCount.get(), lessThan(numThreads));
   }
 
   private static class TryOutputIdRunnable implements Runnable {

[GitHub] incubator-beam pull request #917: [BEAM-383] BigQuery: limit max job polling...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/917


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] incubator-beam git commit: Closes #917

[dhalperi]

Repository: incubator-beam
Updated Branches:
  refs/heads/master be689df07 -> 743a534a0


Closes #917


Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/743a534a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/743a534a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/743a534a

Branch: refs/heads/master
Commit: 743a534a0cf915e2d9d61e8fa9eb048302a13107
Parents: be689df 8d306bb
Author: Dan Halperin 
Authored: Tue Sep 6 11:31:38 2016 -0700
Committer: Dan Halperin 
Committed: Tue Sep 6 11:31:38 2016 -0700

--
 .../apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--

[jira] [Created] (BEAM-618) Python SDKs writes non RFC compliant JSON files for BQ Export

[Alex Amato (JIRA)]

Alex Amato created BEAM-618:
---

 Summary: Python SDKs writes non RFC compliant JSON files for BQ 
Export
 Key: BEAM-618
 URL: https://issues.apache.org/jira/browse/BEAM-618
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Alex Amato
Assignee: Frances Perry


Python SDK uses the built in json.dumps to write JSON files to GCS for the BQ 
Exporter. BigQuery can fail to parse these files when it tries to load these 
files into a BQ table because json.dumps can export JSON which does not conform 
to the IEEE RFC.

There are a few cases which are not RFC compilant listed in that module.
https://docs.python.org/2/library/json.html#standard-compliance-and-interoperability

The main issue we run into is the NAN, INF and -INF values.
These fails with a confusing error (and we delete the GCS files making it hard 
to debug):
JSON table encountered too many errors, giving up. Rows JSON parsing error in 
row starting at position

We can set the allow_nan argument to json.dumps to false to address these 
issues. So that when a user tries to write a file with INF, -INF or NAN

Setting this argument will produce this type of error when json.dumps is called 
with NAN/INF values. We may want to catch this error to mention the fact that 
INF and NAN are not allowed.
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
ValueError: Out of range float values are not JSON compliant




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-544) Add header/footer support to TextIO.Write

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-544:
---
Assignee: Stas Levin

> Add header/footer support to TextIO.Write
> -
>
> Key: BEAM-544
> URL: https://issues.apache.org/jira/browse/BEAM-544
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Luke Cwik
>Assignee: Stas Levin
>Priority: Minor
>
> Being able to add a header/footer to each file that is written via TextIO 
> would cover several simple text file format issues.
> Original ask:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues/360



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-333) Support "stop" for streaming pipelines

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467566#comment-15467566
 ] 

ASF GitHub Bot commented on BEAM-333:
-

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/924

[BEAM-333][flink] make unbounded sources stoppable




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit 557f1fe909047a72492faa67c6f7d2d24c3cf729
Author: Maximilian Michels 
Date:   2016-09-06T14:38:43Z

[BEAM-333][flink] make unbounded sources stoppable




> Support "stop" for streaming pipelines
> --
>
> Key: BEAM-333
> URL: https://issues.apache.org/jira/browse/BEAM-333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Emanuele Cesena
>Assignee: Maximilian Michels
>
> Run:
> {code}
> $ flink run -c 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample
>  /path/to/fat.jar
> {code}
> The job properly runs. Trying to stop it fails. (Cancelling works fine.)
> {code}
> $ flink list
> -- Running/Restarting Jobs ---
> 10.06.2016 06:24:46 : cf004f703cb3d4f89254eedbc04718cc : KafkaExample - 
> WindowSize: 10 seconds (RUNNING)
> --
> No scheduled jobs.
> $ flink stop cf004f703cb3d4f89254eedbc04718cc
> 
>  The program finished with the following exception:
> java.lang.Exception: Stopping the job with ID 
> cf004f703cb3d4f89254eedbc04718cc failed.
>   at org.apache.flink.client.CliFrontend.stop(CliFrontend.java:629)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1201)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243)
> Caused by: java.lang.IllegalStateException: Job with ID 
> cf004f703cb3d4f89254eedbc04718cc is not stoppable.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:483)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:107)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #924: [BEAM-333][flink] make unbounded sources s...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/924

[BEAM-333][flink] make unbounded sources stoppable




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-333

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/924.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #924


commit 557f1fe909047a72492faa67c6f7d2d24c3cf729
Author: Maximilian Michels 
Date:   2016-09-06T14:38:43Z

[BEAM-333][flink] make unbounded sources stoppable




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-333) Support "stop" for streaming pipelines

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467560#comment-15467560
 ] 

Maximilian Michels commented on BEAM-333:
-

You can use {{cancel()}} instead which should achieve the same for Beam sources.

> Support "stop" for streaming pipelines
> --
>
> Key: BEAM-333
> URL: https://issues.apache.org/jira/browse/BEAM-333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Emanuele Cesena
>Assignee: Maximilian Michels
>
> Run:
> {code}
> $ flink run -c 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample
>  /path/to/fat.jar
> {code}
> The job properly runs. Trying to stop it fails. (Cancelling works fine.)
> {code}
> $ flink list
> -- Running/Restarting Jobs ---
> 10.06.2016 06:24:46 : cf004f703cb3d4f89254eedbc04718cc : KafkaExample - 
> WindowSize: 10 seconds (RUNNING)
> --
> No scheduled jobs.
> $ flink stop cf004f703cb3d4f89254eedbc04718cc
> 
>  The program finished with the following exception:
> java.lang.Exception: Stopping the job with ID 
> cf004f703cb3d4f89254eedbc04718cc failed.
>   at org.apache.flink.client.CliFrontend.stop(CliFrontend.java:629)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1201)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243)
> Caused by: java.lang.IllegalStateException: Job with ID 
> cf004f703cb3d4f89254eedbc04718cc is not stoppable.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:483)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:107)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-593) Support cancel() and waitUntilFinish() in FlinkRunnerResult

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467548#comment-15467548
 ] 

Maximilian Michels commented on BEAM-593:
-

+1 These need to be supported. We need to change the execution behavior which 
defaults to blocking upon {{Pipeline.run()}}.

> Support cancel() and waitUntilFinish() in FlinkRunnerResult
> ---
>
> Key: BEAM-593
> URL: https://issues.apache.org/jira/browse/BEAM-593
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Pei He
>
> We introduced both functions to PipelineResult.
> Currently, both of them throw UnsupportedOperationException in Flink runner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-617) Allow setting of StateBackend on FlinkPipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15467538#comment-15467538
 ] 

ASF GitHub Bot commented on BEAM-617:
-

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/923

[BEAM-617][flink] introduce option to set state backend

CC @aljoscha 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-617

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #923


commit 2eba5034e7d5abbae737025a125805af7744aefd
Author: Maximilian Michels 
Date:   2016-09-06T14:25:32Z

[BEAM-617][flink] introduce option to set state backend




> Allow setting of StateBackend on FlinkPipelineOptions
> -
>
> Key: BEAM-617
> URL: https://issues.apache.org/jira/browse/BEAM-617
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Affects Versions: 0.3.0-incubating
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>
> The Flink Runner defaults to the in-memory state backend if not set otherwise 
> in the Flink config. In order to set the RocksDB state backend or allow 
> custom state backends, we need to introduce a new option in 
> FlinkPipelineOptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #923: [BEAM-617][flink] introduce option to set ...

[mxm]

GitHub user mxm opened a pull request:

https://github.com/apache/incubator-beam/pull/923

[BEAM-617][flink] introduce option to set state backend

CC @aljoscha 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mxm/incubator-beam BEAM-617

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/923.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #923


commit 2eba5034e7d5abbae737025a125805af7744aefd
Author: Maximilian Michels 
Date:   2016-09-06T14:25:32Z

[BEAM-617][flink] introduce option to set state backend




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-616) Update Flink Runner to 1.1.2

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels updated BEAM-616:

Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-273

> Update Flink Runner to 1.1.2
> 
>
> Key: BEAM-616
> URL: https://issues.apache.org/jira/browse/BEAM-616
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-283) CheckpointMark.finalize() is not called in Flink Source Wrapper

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned BEAM-283:
---

Assignee: Maximilian Michels

> CheckpointMark.finalize() is not called in Flink Source Wrapper
> ---
>
> Key: BEAM-283
> URL: https://issues.apache.org/jira/browse/BEAM-283
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Maximilian Michels
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (BEAM-333) Support "stop" for streaming pipelines

[Maximilian Michels (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Maximilian Michels reassigned BEAM-333:
---

Assignee: Maximilian Michels

> Support "stop" for streaming pipelines
> --
>
> Key: BEAM-333
> URL: https://issues.apache.org/jira/browse/BEAM-333
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Emanuele Cesena
>Assignee: Maximilian Michels
>
> Run:
> {code}
> $ flink run -c 
> org.apache.beam.runners.flink.examples.streaming.KafkaWindowedWordCountExample
>  /path/to/fat.jar
> {code}
> The job properly runs. Trying to stop it fails. (Cancelling works fine.)
> {code}
> $ flink list
> -- Running/Restarting Jobs ---
> 10.06.2016 06:24:46 : cf004f703cb3d4f89254eedbc04718cc : KafkaExample - 
> WindowSize: 10 seconds (RUNNING)
> --
> No scheduled jobs.
> $ flink stop cf004f703cb3d4f89254eedbc04718cc
> 
>  The program finished with the following exception:
> java.lang.Exception: Stopping the job with ID 
> cf004f703cb3d4f89254eedbc04718cc failed.
>   at org.apache.flink.client.CliFrontend.stop(CliFrontend.java:629)
>   at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1201)
>   at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1243)
> Caused by: java.lang.IllegalStateException: Job with ID 
> cf004f703cb3d4f89254eedbc04718cc is not stoppable.
>   at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:483)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
>   at 
> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>   at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
>   at 
> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>   at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>   at 
> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:107)
>   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>   at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>   at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>   at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>   at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>   at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>   at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #922: Fix shaded imports in Flink Runner

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/922

Fix shaded imports in Flink Runner

R: @mxm for review

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam flink-fix-imports

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/922.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #922


commit 051f6df5d8eb18d7694f2cf775792413e03bb42b
Author: Aljoscha Krettek 
Date:   2016-09-05T08:50:14Z

Fix shaded imports in Flink Runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-beam pull request #921: [FLINK-616] Update Flink Runner to Flink 1...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/incubator-beam/pull/921

[FLINK-616] Update Flink Runner to Flink 1.1.2

R: @mxm for review


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/incubator-beam flink-1.1.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #921


commit 129ff811754f33313ddfe7021d6c2d49f6c57f35
Author: Aljoscha Krettek 
Date:   2016-09-05T16:17:11Z

[FLINK-616] Update Flink Runner to Flink 1.1.2




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-616) Update Flink Runner to 1.1.2

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-616:
-

 Summary: Update Flink Runner to 1.1.2
 Key: BEAM-616
 URL: https://issues.apache.org/jira/browse/BEAM-616
 Project: Beam
  Issue Type: Improvement
  Components: runner-flink
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-553) Add a custom text source

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466569#comment-15466569
 ] 

ASF GitHub Bot commented on BEAM-553:
-

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/920

[BEAM-553] Adds a text source for Python SDK.

Current text source (fileio.TextFileSource) is specific to Dataflow runner. 
This adds a runner independent TextSource that is based on iobase.BoundedSource 
interface.

Adds a textio module that contains text source, text sink, and PTransforms 
that can be used to read and write text files.

Adds a significant number of tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam text_source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #920


commit bb1ff90307b563656a54731cada05e41cd9e82b8
Author: Chamikara Jayalath 
Date:   2016-08-30T01:08:46Z

Adds a text source to Python SDK.




> Add a custom text source
> 
>
> Key: BEAM-553
> URL: https://issues.apache.org/jira/browse/BEAM-553
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently, the text source implementation available for Python SDK [1] is a 
> Dataflow native source which only works efficiently for Dataflow runner. We 
> should add a custom text source on top of custom file-based source framework 
> [2] so that other runner implementations can potentially use the same text 
> source implementation.
> Custom text source implementation for Java SDK is at [3].
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L70
> [2] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/filebasedsource.py
> [3] 
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L745



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] incubator-beam pull request #920: [BEAM-553] Adds a text source for Python S...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/incubator-beam/pull/920

[BEAM-553] Adds a text source for Python SDK.

Current text source (fileio.TextFileSource) is specific to Dataflow runner. 
This adds a runner independent TextSource that is based on iobase.BoundedSource 
interface.

Adds a textio module that contains text source, text sink, and PTransforms 
that can be used to read and write text files.

Adds a significant number of tests.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/incubator-beam text_source

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #920


commit bb1ff90307b563656a54731cada05e41cd9e82b8
Author: Chamikara Jayalath 
Date:   2016-08-30T01:08:46Z

Adds a text source to Python SDK.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---