Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2008

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Dataflow #2008

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1342

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1342

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_RunnableOnService_Flink #1341

[Apache Jenkins Server]

See 

[1/2] beam git commit: Captures assertion site and message in PAssert

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 055f452c0 -> 51820cbe0


Captures assertion site and message in PAssert

This makes PAssert failures quite a bit easier to debug.
Example message after this commit:

java.lang.AssertionError: Some message

at 
org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:384)
at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:279)
at 
org.apache.beam.sdk.transforms.SplittableDoFnTest.testPairWithIndexWindowedTimestamped(SplittableDoFnTest.java:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
Caused by: java.lang.AssertionError:
Expected: iterable over [, , , , , , , ] in any order
 but: Not matched: 
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.beam.sdk.testing.PAssert$AssertContainsInAnyOrder.apply(PAssert.java:1270)
at 
org.apache.beam.sdk.testing.PAssert$AssertContainsInAnyOrder.apply(PAssert.java:1)
...

(as opposed to, basically, just the "Caused by" part)


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c62611c7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c62611c7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c62611c7

Branch: refs/heads/master
Commit: c62611c73ab0f9a5769f3ee9b28b11e917628f78
Parents: b81bd25
Author: Eugene Kirpichov 
Authored: Mon Jan 9 16:38:50 2017 -0800
Committer: Kenneth Knowles 
Committed: Tue Jan 10 21:45:21 2017 -0800

--
 .../org/apache/beam/sdk/testing/PAssert.java| 163 +++
 .../apache/beam/sdk/testing/PAssertTest.java|  44 +
 2 files changed, 174 insertions(+), 33 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/c62611c7/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
index b57f4a9..89d6fea 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/PAssert.java
@@ -275,8 +275,13 @@ public class PAssert {
   /**
* Constructs an {@link IterableAssert} for the elements of the provided 
{@link PCollection}.
*/
+  public static  IterableAssert that(String message, PCollection 
actual) {
+return new PCollectionContentsAssert<>(PAssertionSite.capture(message), 
actual);
+  }
+
+  /** @see #that(String, PCollection) */
   public static  IterableAssert that(PCollection actual) {
-return new PCollectionContentsAssert<>(actual);
+return that("", actual);
   }
 
   /**
@@ -284,7 +289,7 @@ public class PAssert {
* must contain a single {@code Iterable} value.
*/
   public static  IterableAssert thatSingletonIterable(
-  PCollection> actual) {
+  String message, PCollection> actual) {
 
 try {
 } catch (NoSuchElementException | IllegalArgumentException exc) {
@@ -297,15 +302,29 @@ public class PAssert {
 @SuppressWarnings("unchecked") // Safe covariant cast
 PCollection actualIterables = (PCollection) 
actual;
 
-return new PCollectionSingletonIterableAssert<>(actualIterables);
+return new PCollectionSingletonIterableAssert<>(
+PAssertionSite.capture(message), actualIterables);
   }
 
-  /**
-   * Constructs a {@link SingletonAssert} for the value of the provided
-   * {@code PCollection PCollection}, which must be a singleton.
-   */
+  /** @see #thatSingletonIterable(String, PCollection)  */
+  public static  IterableAssert thatSingletonIterable(
+  PCollection> actual) {
+return thatSingletonIterable("", actual);
+  }
+
+/**
+ * Constructs a {@link SingletonAssert} for the value of the provided
+ * {@code PCollection PCollection}, which must be a singleton.
+ */
+  public static  SingletonAssert thatSingleton(String message, 
PCollection actual) {
+return new PCollectionViewAssert<>(
+

[jira] [Resolved] (BEAM-1258) BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1258.
---
Resolution: Fixed

> BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken
> --
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.4.0
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.5.0
>
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table myproject:mytable",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table myproject:mytable"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:299)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> at 
> com.google.cloud.training.dataanalyst.javahelp.StreamDemoConsumer.main(StreamDemoConsumer.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1258) BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1258:
--
Fix Version/s: 0.5.0

> BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken
> --
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.4.0
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.5.0
>
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table myproject:mytable",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table myproject:mytable"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:299)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> at 
> com.google.cloud.training.dataanalyst.javahelp.StreamDemoConsumer.main(StreamDemoConsumer.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1258) BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin updated BEAM-1258:
--
Affects Version/s: 0.4.0

> BigQueryIO.Write: CREATE_IF_NEEDED and per-window tables is broken
> --
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 0.4.0
>Reporter: Pei He
>Assignee: Pei He
> Fix For: 0.5.0
>
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table myproject:mytable",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table myproject:mytable"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:299)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> at 
> com.google.cloud.training.dataanalyst.javahelp.StreamDemoConsumer.main(StreamDemoConsumer.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1759: [BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryService...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1759


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1759

[dhalperi]

This closes #1759


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/055f452c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/055f452c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/055f452c

Branch: refs/heads/master
Commit: 055f452c0a2f62920f58928e66aebe21744eaf41
Parents: 68b4c34 3490a36
Author: Dan Halperin 
Authored: Tue Jan 10 18:14:09 2017 -0800
Committer: Dan Halperin 
Committed: Tue Jan 10 18:14:09 2017 -0800

--
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 18 +++--
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 74 
 2 files changed, 88 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 68b4c34a4 -> 055f452c0


[BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3490a36e
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3490a36e
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3490a36e

Branch: refs/heads/master
Commit: 3490a36e5877227943c5635b4d92706f1d128d22
Parents: 68b4c34
Author: Pei He 
Authored: Tue Jan 10 10:57:38 2017 -0800
Committer: Dan Halperin 
Committed: Tue Jan 10 18:13:52 2017 -0800

--
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 18 +++--
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 74 
 2 files changed, 88 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3490a36e/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
index c4c7344..2098148 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java
@@ -527,14 +527,24 @@ class BigQueryServicesImpl implements BigQueryServices {
   BackOff backoff =
   FluentBackoff.DEFAULT
   
.withMaxRetries(MAX_RPC_RETRIES).withInitialBackoff(INITIAL_RPC_BACKOFF).backoff();
+  return isTableEmpty(
+  new 
TableReference().setProjectId(projectId).setDatasetId(datasetId).setTableId(tableId),
+  backoff,
+  Sleeper.DEFAULT);
+}
+
+@VisibleForTesting
+boolean isTableEmpty(TableReference tableRef, BackOff backoff, Sleeper 
sleeper)
+throws IOException, InterruptedException {
   TableDataList dataList = executeWithRetries(
-  client.tabledata().list(projectId, datasetId, tableId),
+  client.tabledata().list(
+  tableRef.getProjectId(), tableRef.getDatasetId(), 
tableRef.getTableId()),
   String.format(
   "Unable to list table data: %s, aborting after %d retries.",
-  tableId, MAX_RPC_RETRIES),
-  Sleeper.DEFAULT,
+  tableRef.getTableId(), MAX_RPC_RETRIES),
+  sleeper,
   backoff,
-  ALWAYS_RETRY);
+  DONT_RETRY_NOT_FOUND);
   return dataList.getRows() == null || dataList.getRows().isEmpty();
 }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/3490a36e/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
index bfd1319..1ce10f1 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImplTest.java
@@ -21,6 +21,7 @@ import static com.google.common.base.Verify.verifyNotNull;
 import static org.hamcrest.Matchers.containsString;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertThat;
 import static org.junit.Assert.fail;
@@ -49,6 +50,7 @@ import com.google.api.services.bigquery.model.JobStatus;
 import com.google.api.services.bigquery.model.Table;
 import com.google.api.services.bigquery.model.TableDataInsertAllResponse;
 import 
com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors;
+import com.google.api.services.bigquery.model.TableDataList;
 import com.google.api.services.bigquery.model.TableFieldSchema;
 import com.google.api.services.bigquery.model.TableReference;
 import com.google.api.services.bigquery.model.TableRow;
@@ -385,6 +387,78 @@ public class BigQueryServicesImplTest {
   }
 
   @Test
+  public void testIsTableEmptySucceeds() throws Exception {
+TableReference tableRef = new TableReference()
+.setProjectId("projectId")
+.setDatasetId("datasetId")
+.setTableId("tableId");
+
+

[jira] [Resolved] (BEAM-1257) Compressed text source emit an extra element when reading records

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1257.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Compressed text source emit an extra element when reading records
> -
>
> Key: BEAM-1257
> URL: https://issues.apache.org/jira/browse/BEAM-1257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> Way to reproduce: 
> Create a text file that has N lines. 
> Create a pipeline that reads the file and counts the number of elements.
> The plain text file will output N while a compressed file outputs N+1 which 
> is incorrect. 
> Also need to add unittests to prevent this in the future. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1265) Add streaming support to DirectRunner

[Ahmet Altay (JIRA)]

Ahmet Altay created BEAM-1265:
-

 Summary: Add streaming support to DirectRunner
 Key: BEAM-1265
 URL: https://issues.apache.org/jira/browse/BEAM-1265
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Reporter: Ahmet Altay
Assignee: Charles Chen


Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (BEAM-428) InProcessRunner - Bundle based local runner

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816836#comment-15816836
 ] 

Ahmet Altay edited comment on BEAM-428 at 1/11/17 1:59 AM:
---

Streaming support is tracked in https://issues.apache.org/jira/browse/BEAM-1265


was (Author: altay):
Streamin support is tracked in https://issues.apache.org/jira/browse/BEAM-1265

> InProcessRunner - Bundle based local runner
> ---
>
> Key: BEAM-428
> URL: https://issues.apache.org/jira/browse/BEAM-428
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Ahmet Altay
> Fix For: Not applicable
>
>
> InProcessRunner is a bundle based drop in replacement for DirectRunner.
> Similar to its Java equivalent it improves DirectRunner by executing 
> transforms in parallel using bundles similar to a service based 
> implementations. It offers better performance and more validation options.
> Initially it will be a runner for executing batch jobs only. The target of 
> this phase is to develop a drop in replacement for DirectRunner. Later it 
> will be improved by adding streaming execution.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1106) Remove no_pipeline_type_check flag from Python SDK

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816759#comment-15816759
 ] 

Ahmet Altay commented on BEAM-1106:
---

https://github.com/apache/beam/pull/1746 fixed this.

> Remove no_pipeline_type_check flag from Python SDK
> --
>
> Key: BEAM-1106
> URL: https://issues.apache.org/jira/browse/BEAM-1106
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Frances Perry
>
> It's already the default behavior. It should be possible to remove it without 
> trouble.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1764: Standard coder tests

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/1764

Standard coder tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam standard-coders

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1764.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1764






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1257) Compressed text source emit an extra element when reading records

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816711#comment-15816711
 ] 

ASF GitHub Bot commented on BEAM-1257:
--

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1757


> Compressed text source emit an extra element when reading records
> -
>
> Key: BEAM-1257
> URL: https://issues.apache.org/jira/browse/BEAM-1257
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>
> Way to reproduce: 
> Create a text file that has N lines. 
> Create a pipeline that reads the file and counts the number of elements.
> The plain text file will output N while a compressed file outputs N+1 which 
> is incorrect. 
> Also need to add unittests to prevent this in the future. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1757: [BEAM-1257] Compressed text source emit an extra el...

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/1757


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: Closes #1757

[robertwb]

Closes #1757


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/86d42037
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/86d42037
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/86d42037

Branch: refs/heads/python-sdk
Commit: 86d420376b14a9ff19b4f078470d40a7fd4267a5
Parents: 2d190a2 fb80e09
Author: Robert Bradshaw 
Authored: Tue Jan 10 16:56:29 2017 -0800
Committer: Robert Bradshaw 
Committed: Tue Jan 10 16:56:29 2017 -0800

--
 sdks/python/apache_beam/io/textio.py  | 10 ++
 sdks/python/apache_beam/io/textio_test.py | 44 --
 2 files changed, 52 insertions(+), 2 deletions(-)
--

[2/2] beam git commit: Closes #1754

[robertwb]

Closes #1754


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2d190a29
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2d190a29
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2d190a29

Branch: refs/heads/python-sdk
Commit: 2d190a29bc8327143db20ef705894d084c040a4a
Parents: cb06349 ecd6b5e
Author: Robert Bradshaw 
Authored: Tue Jan 10 16:54:49 2017 -0800
Committer: Robert Bradshaw 
Committed: Tue Jan 10 16:54:49 2017 -0800

--
 sdks/python/apache_beam/coders/typecoders.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
--

[GitHub] beam pull request #1761: [BEAM-147] Adding proto buff matchers for dataflow ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1761


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1264) Python ChannelFactory Raise Inconsistent Error for Local FS and GCS

[Mark Liu (JIRA)]

Mark Liu created BEAM-1264:
--

 Summary: Python ChannelFactory Raise Inconsistent Error for Local 
FS and GCS
 Key: BEAM-1264
 URL: https://issues.apache.org/jira/browse/BEAM-1264
 Project: Beam
  Issue Type: Bug
  Components: beam-model, sdk-py
Reporter: Mark Liu
Assignee: Ahmet Altay


ChannelFactory raises different errors for local fs (RuntimeError) and GCS 
(IOError) when reading failed. 

We want to return consistent error for both.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1263) Add Jenkins Suites for Performance Tests

[Jason Kuster (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816664#comment-15816664
 ] 

Jason Kuster commented on BEAM-1263:


https://github.com/apache/beam/pull/1751 is the PR currently working on this 
issue. A couple things yet to add.
- Refactor common items out into common_job_properties.groovy
- Set up "Restrict which machines these jobs can execute on"

> Add Jenkins Suites for Performance Tests
> 
>
> Key: BEAM-1263
> URL: https://issues.apache.org/jira/browse/BEAM-1263
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Jason Kuster
>Assignee: Jason Kuster
>
> Add Groovy files describing performance testing suites for Beam (for now just 
> Spark and Dataflow, given that those are the two we are currently able to 
> support).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1263) Add Jenkins Suites for Performance Tests

[Jason Kuster (JIRA)]

Jason Kuster created BEAM-1263:
--

 Summary: Add Jenkins Suites for Performance Tests
 Key: BEAM-1263
 URL: https://issues.apache.org/jira/browse/BEAM-1263
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Jason Kuster
Assignee: Jason Kuster


Add Groovy files describing performance testing suites for Beam (for now just 
Spark and Dataflow, given that those are the two we are currently able to 
support).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1763: [BEAM-147] Adding sum of squares to distributions i...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/1763

[BEAM-147] Adding sum of squares to distributions in metrics

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam sum-of-squares-distros

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1763.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1763


commit 933c3baca7997495b373829b341bdfb9b7fbe12f
Author: Pablo 
Date:   2017-01-10T23:24:25Z

Adding sum of squares to distributions in metrics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816494#comment-15816494
 ] 

ASF GitHub Bot commented on BEAM-759:
-

GitHub user aaltay reopened a pull request:

https://github.com/apache/beam/pull/1762

[BEAM-759] Implement wait_until_finish method for existing runners.

Implement wait_until_finish method for existing runners. 

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.

Main changes are in the runners/ folder
runner.py - has the APIs
dataflow_runner.py, direct_runner.py modified to implement the API (moving 
the existing blocking code around.)

Rest of the changes are mechanical to mainly convert
p.run() to p.run().wait_until_finish() in tests and examples. Changed tests 
because they run validation after the run and need to block until completion. 
We may revert the changes in examples. I converted the because in the 
instructions we directed users to blocking runners before and this change keeps 
the behavior same.

I have started a local post commit run (not completed yet) and it was 
successful with the first few tests so far and the changes are same for all 
tests.

Remaining work after this PR:
- Removing BlockingDataflowRunner. It is not used after this change with 
the SDK code/examples/tests.
- Support for duration argument in wait_until_finish is missing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1762.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1762


commit c6c78a3fe15910acf2d1cc0bf244af06794dea1f
Author: Ahmet Altay 
Date:   2017-01-10T02:23:20Z

Implement wait_until_finish method for existing runners.

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.

commit a27773088d1ecad371725fdd5d27c5a1dd58a5cb
Author: Ahmet Altay 
Date:   2017-01-10T22:15:36Z

Fix wrong use of wait_until_complete in examples.




> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-759) PipelineResult needs waitToFinish() and cancel()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816493#comment-15816493
 ] 

ASF GitHub Bot commented on BEAM-759:
-

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1762


> PipelineResult needs waitToFinish() and cancel()
> 
>
> Key: BEAM-759
> URL: https://issues.apache.org/jira/browse/BEAM-759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: sdk-consistency
>
> Java SDK, added waitToFinish() and cancel() to the PipelineResults, and as a 
> result were able to remove BlockingDataflowRunner.
> (See: https://issues.apache.org/jira/browse/BEAM-443)
> The same changes needs to happen in python sdk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1762: [BEAM-759] Implement wait_until_finish method for e...

[aaltay]

Github user aaltay closed the pull request at:

https://github.com/apache/beam/pull/1762


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1762: [BEAM-759] Implement wait_until_finish method for e...

[aaltay]

GitHub user aaltay reopened a pull request:

https://github.com/apache/beam/pull/1762

[BEAM-759] Implement wait_until_finish method for existing runners.

Implement wait_until_finish method for existing runners. 

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.

Main changes are in the runners/ folder
runner.py - has the APIs
dataflow_runner.py, direct_runner.py modified to implement the API (moving 
the existing blocking code around.)

Rest of the changes are mechanical to mainly convert
p.run() to p.run().wait_until_finish() in tests and examples. Changed tests 
because they run validation after the run and need to block until completion. 
We may revert the changes in examples. I converted the because in the 
instructions we directed users to blocking runners before and this change keeps 
the behavior same.

I have started a local post commit run (not completed yet) and it was 
successful with the first few tests so far and the changes are same for all 
tests.

Remaining work after this PR:
- Removing BlockingDataflowRunner. It is not used after this change with 
the SDK code/examples/tests.
- Support for duration argument in wait_until_finish is missing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1762.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1762


commit c6c78a3fe15910acf2d1cc0bf244af06794dea1f
Author: Ahmet Altay 
Date:   2017-01-10T02:23:20Z

Implement wait_until_finish method for existing runners.

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.

commit a27773088d1ecad371725fdd5d27c5a1dd58a5cb
Author: Ahmet Altay 
Date:   2017-01-10T22:15:36Z

Fix wrong use of wait_until_complete in examples.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1260) PAssert should capture the assertion site

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816390#comment-15816390
 ] 

Eugene Kirpichov commented on BEAM-1260:


PR https://github.com/apache/beam/pull/1753 does this

> PAssert should capture the assertion site
> -
>
> Key: BEAM-1260
> URL: https://issues.apache.org/jira/browse/BEAM-1260
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>
> When a PAssert assertion fails, it doesn't tell where the assertion 
> (PAssert.that(blah)) call was made in code - only when the failure was 
> detected, i.e. somewhere deep in worker code usually.
> It also doesn't allow specifying a message (unlike JUnit).
> This issue is about improving the way PAssert failures are reported.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1760: [BEAM-1258] BigQueryServicesImpl.getTable() returns...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1760


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #1762: Implement wait_until_finish method for existing run...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/1762

Implement wait_until_finish method for existing runners.

Implement wait_until_finish method for existing runners. 

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.

Main changes are in the runners/ folder
runner.py - has the APIs
dataflow_runner.py, direct_runner.py modified to implement the API (moving 
the existing blocking code around.)

Rest of the changes are mechanical to mainly convert
p.run() to p.run().wait_until_finish() in tests and examples. Changed tests 
because they run validation after the run and need to block until completion. 
We may revert the changes in examples. I converted the because in the 
instructions we directed users to blocking runners before and this change keeps 
the behavior same.

I have started a local post commit run (not completed yet) and it was 
successful with the first few tests so far and the changes are same for all 
tests.

Remaining work after this PR:
- Removing BlockingDataflowRunner. It is not used after this change with 
the SDK code/examples/tests.
- Support for duration argument in wait_until_finish is missing.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/incubator-beam expand

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1762.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1762


commit c6c78a3fe15910acf2d1cc0bf244af06794dea1f
Author: Ahmet Altay 
Date:   2017-01-10T02:23:20Z

Implement wait_until_finish method for existing runners.

Also defines the not implemented cancel() method and updates existing
usages to use wait_until_finish() instead of blocking runners.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1258) BigQuery CREATE_IF_NEEDED not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816302#comment-15816302
 ] 

ASF GitHub Bot commented on BEAM-1258:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1760


> BigQuery CREATE_IF_NEEDED not working
> -
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table myproject:mytable",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table myproject:mytable"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:299)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> at 
> com.google.cloud.training.dataanalyst.javahelp.StreamDemoConsumer.main(StreamDemoConsumer.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (BEAM-1259) Suppress fallback coder for typehint warnings

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-1259:
--
Description: There are lots of warnings of the form 'WARNING:root:Using 
fallback coder for typehint: Any.'. Converting these log statements to warnings 
would allow users to filter them as (all/once/none) as they prefer.

> Suppress fallback coder for typehint warnings
> -
>
> Key: BEAM-1259
> URL: https://issues.apache.org/jira/browse/BEAM-1259
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Neda Mirian
>Assignee: Ahmet Altay
>Priority: Minor
>
> There are lots of warnings of the form 'WARNING:root:Using fallback coder for 
> typehint: Any.'. Converting these log statements to warnings would allow 
> users to filter them as (all/once/none) as they prefer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[2/2] beam git commit: This closes #1760

[dhalperi]

This closes #1760


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/68b4c34a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/68b4c34a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/68b4c34a

Branch: refs/heads/master
Commit: 68b4c34a413fe94080aa7fa3fd8c5934c5752d19
Parents: c1b7f86 e670e7e
Author: Dan Halperin 
Authored: Tue Jan 10 13:51:14 2017 -0800
Committer: Dan Halperin 
Committed: Tue Jan 10 13:51:14 2017 -0800

--
 .../sdk/io/gcp/bigquery/BigQueryServices.java   |  5 +-
 .../io/gcp/bigquery/BigQueryServicesImpl.java   | 32 +++---
 .../gcp/bigquery/BigQueryServicesImplTest.java  | 65 
 3 files changed, 94 insertions(+), 8 deletions(-)
--

[jira] [Commented] (BEAM-147) Introduce an easy API for pipeline metrics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816272#comment-15816272
 ] 

ASF GitHub Bot commented on BEAM-147:
-

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/1761

[BEAM-147] Adding proto buff matchers for dataflow client

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Adding these matchers to test changes to dataflow client - service 
communication in metrics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam proto-matchers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1761.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1761


commit e8be8fa28a1095c8e881a1d48bb4a0265ab42ed4
Author: Pablo 
Date:   2017-01-10T21:38:48Z

Adding proto buff matchers for dataflow client




> Introduce an easy API for pipeline metrics
> --
>
> Key: BEAM-147
> URL: https://issues.apache.org/jira/browse/BEAM-147
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core, sdk-py
>Reporter: Robert Bradshaw
>Assignee: Ben Chambers
>
> The existing Aggregators are confusing both because of their name and because 
> they serve multiple purposes.
> Previous discussions around Aggregators/metrics/etc.
> See discussion at 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201603.mbox/browser
>  and 
> http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201603.mbox/browser
>  . Exact name still being bikeshedded.
> Design document: http://s.apache.org/beam-metrics-api



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1761: [BEAM-147] Adding proto buff matchers for dataflow ...

[pabloem]

GitHub user pabloem opened a pull request:

https://github.com/apache/beam/pull/1761

[BEAM-147] Adding proto buff matchers for dataflow client

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---
Adding these matchers to test changes to dataflow client - service 
communication in metrics.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam proto-matchers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1761.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1761


commit e8be8fa28a1095c8e881a1d48bb4a0265ab42ed4
Author: Pablo 
Date:   2017-01-10T21:38:48Z

Adding proto buff matchers for dataflow client




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1260) PAssert should capture the assertion site

[Eugene Kirpichov (JIRA)]

Eugene Kirpichov created BEAM-1260:
--

 Summary: PAssert should capture the assertion site
 Key: BEAM-1260
 URL: https://issues.apache.org/jira/browse/BEAM-1260
 Project: Beam
  Issue Type: Improvement
  Components: sdk-java-core
Reporter: Eugene Kirpichov
Assignee: Eugene Kirpichov


When a PAssert assertion fails, it doesn't tell where the assertion 
(PAssert.that(blah)) call was made in code - only when the failure was 
detected, i.e. somewhere deep in worker code usually.

It also doesn't allow specifying a message (unlike JUnit).

This issue is about improving the way PAssert failures are reported.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1259) Suppress fallback coder for typehint warnings

[Neda Mirian (JIRA)]

Neda Mirian created BEAM-1259:
-

 Summary: Suppress fallback coder for typehint warnings
 Key: BEAM-1259
 URL: https://issues.apache.org/jira/browse/BEAM-1259
 Project: Beam
  Issue Type: Bug
  Components: sdk-py
Reporter: Neda Mirian
Assignee: Ahmet Altay
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1258) BigQuery CREATE_IF_NEEDED not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815989#comment-15815989
 ] 

ASF GitHub Bot commented on BEAM-1258:
--

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1760

[BEAM-1258] BigQueryServicesImpl.getTable() returns null when tables not 
found.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bq-create-if-needed-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1760.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1760


commit 52c0df03f4001a39e26894c91bcb0e6b8ff8b5e1
Author: Pei He 
Date:   2017-01-10T18:57:38Z

[BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().

commit e8d6c8fd3e072763c9eafb1dc007bace4ac02de4
Author: Pei He 
Date:   2017-01-10T19:49:37Z

[BEAM-1258] BigQueryServicesImpl.getTable() returns null when tables not 
found.




> BigQuery CREATE_IF_NEEDED not working
> -
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table cloud-training-demos:demos.streamdemo",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table cloud-training-demos:demos.streamdemo"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at 

[GitHub] beam pull request #1760: [BEAM-1258] BigQueryServicesImpl.getTable() returns...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1760

[BEAM-1258] BigQueryServicesImpl.getTable() returns null when tables not 
found.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bq-create-if-needed-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1760.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1760


commit 52c0df03f4001a39e26894c91bcb0e6b8ff8b5e1
Author: Pei He 
Date:   2017-01-10T18:57:38Z

[BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().

commit e8d6c8fd3e072763c9eafb1dc007bace4ac02de4
Author: Pei He 
Date:   2017-01-10T19:49:37Z

[BEAM-1258] BigQueryServicesImpl.getTable() returns null when tables not 
found.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1227) Release 0.4.0

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815982#comment-15815982
 ] 

Daniel Halperin commented on BEAM-1227:
---

+1, thanks.

> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-1227) Release 0.4.0

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-1227.
-

> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
> Fix For: Not applicable
>
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1227) Release 0.4.0

[Alexey Diomin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815919#comment-15815919
 ] 

Alexey Diomin commented on BEAM-1227:
-

Maybe we can close this task ;)

> Release 0.4.0
> -
>
> Key: BEAM-1227
> URL: https://issues.apache.org/jira/browse/BEAM-1227
> Project: Beam
>  Issue Type: Task
>  Components: project-management
>Affects Versions: 0.4.0
>Reporter: Daniel Halperin
>Assignee: Jean-Baptiste Onofré
>
> Umbrella bug for the 0.4.0 (incubating or not) release.
> JB is the release manager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1258) BigQuery CREATE_IF_NEEDED not working

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815856#comment-15815856
 ] 

ASF GitHub Bot commented on BEAM-1258:
--

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1759

[BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bq-create-if-needed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1759.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1759






> BigQuery CREATE_IF_NEEDED not working
> -
>
> Key: BEAM-1258
> URL: https://issues.apache.org/jira/browse/BEAM-1258
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Pei He
>Assignee: Pei He
>
> User reported BigQuery error in beam 2.0:
> """
> Updating my streaming pipeline to 2.0 (org.beam).  This is my code:
> .apply(BigQueryIO.Write.to(output)//
> .withSchema(schema)//
> 
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED));
> The error when pipeline is being deployed is below.  After several such 
> errors, the pipeline is deployed, but no table is ever created, so I don't 
> get any output.
> Jan 09, 2017 3:44:13 PM 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl executeWithRetries
> WARNING: Ignore the error and retry the request.
> com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not 
> Found
> {
>   "code" : 404,
>   "errors" : [ {
> "domain" : "global",
> "message" : "Not found: Table cloud-training-demos:demos.streamdemo",
> "reason" : "notFound"
>   } ],
>   "message" : "Not found: Table cloud-training-demos:demos.streamdemo"
> }
> at 
> com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
> at 
> com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
> at 
> com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
> at 
> com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl.executeWithRetries(BigQueryServicesImpl.java:860)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.isTableEmpty(BigQueryServicesImpl.java:512)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.verifyTableEmpty(BigQueryIO.java:1836)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1886)
> at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$Write$Bound.validate(BigQueryIO.java:1623)
> at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:384)
> at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:299)
> at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:154)
> at 
> com.google.cloud.training.dataanalyst.javahelp.StreamDemoConsumer.main(StreamDemoConsumer.java:102)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> 

[GitHub] beam pull request #1759: [BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryService...

[peihe]

GitHub user peihe opened a pull request:

https://github.com/apache/beam/pull/1759

[BEAM-1258] DONT_RETRY_NOT_FOUND in BigQueryServicesImpl.isTableEmpty().

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/peihe/incubator-beam bq-create-if-needed

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1759.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1759






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1084) Update apitools to version 0.5.6

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1084.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Update apitools to version 0.5.6
> 
>
> Key: BEAM-1084
> URL: https://issues.apache.org/jira/browse/BEAM-1084
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
> Fix For: Not applicable
>
>
> There are some fixes to JsonValue that should be included in beam 
> (https://github.com/google/apitools/pull/136) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (BEAM-1087) Pickling error in save main session

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1087.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Pickling error in save main session
> ---
>
> Key: BEAM-1087
> URL: https://issues.apache.org/jira/browse/BEAM-1087
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>Priority: Minor
> Fix For: Not applicable
>
>
> {code}
>   File "/usr/local/lib/python2.7/pickle.py", line 286, in save
> f(self, obj) # Call unbound method with explicit self
>   File "/usr/local/lib/python2.7/site-packages/dill/dill.py", line 1231, in 
> save_type
> StockPickler.save_global(pickler, obj)
>   File "/usr/local/lib/python2.7/pickle.py", line 754, in save_global
> (obj, module, name))
> pickle.PicklingError: Can't pickle  'apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum'>:
>  it's not found as 
> apache_beam.internal.clients.dataflow.dataflow_v1b3_messages.TypeValueValuesEnum
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-519) fileio.CompressionType requires a __ne__ method

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815627#comment-15815627
 ] 

Ahmet Altay commented on BEAM-519:
--

Thank you, resolving.

>  fileio.CompressionType requires a __ne__ method
> 
>
> Key: BEAM-519
> URL: https://issues.apache.org/jira/browse/BEAM-519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Ahmet Altay
>Priority: Minor
>  Labels: starter
> Fix For: Not applicable
>
>
> This code: 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L279
> Without the __ne__ operator instances of this class cannot be used in != 
> expressions (only ==).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-1256) SparkPipelineResult.StreamingMode should not return null on awaitTermination()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815614#comment-15815614
 ] 

ASF GitHub Bot commented on BEAM-1256:
--

GitHub user amitsela opened a pull request:

https://github.com/apache/beam/pull/1758

[BEAM-1256] SparkPipelineResult.StreamingMode should not return null …

…on awaitTermination()

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/beam BEAM-1256

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1758


commit 7af52a1a60e37dbf52cf1d78dbc9ede0cb7eba84
Author: Sela 
Date:   2017-01-10T17:37:20Z

[BEAM-1256] SparkPipelineResult.StreamingMode should not return null on 
awaitTermination()




> SparkPipelineResult.StreamingMode should not return null on awaitTermination()
> --
>
> Key: BEAM-1256
> URL: https://issues.apache.org/jira/browse/BEAM-1256
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
>Priority: Minor
>
> Currently, {{SparkPipelineResult.StreamingMode}} will return {{null}} if 
> {{JavaStreamingContext#awaitTerminationOrTimeout(Long)}} returns {{false}}.
> This is wrong because this will always return {{false}} and so {{null}} 
> because Spark's {{ContextWaiter}} will only return {{true}} if the context 
> was stopped and this only happens after calling "awaitTermination".
> Following Spark code:
> awaitTermination returns the [state of 
> stopped|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/ContextWaiter.scala#L74]
> But can only stopped via 
> [notifyStopped|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/ContextWaiter.scala#L44]
> Which is only called by 
> [StreamingContext#stop()|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L714]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam pull request #1758: [BEAM-1256] SparkPipelineResult.StreamingMode shoul...

[amitsela]

GitHub user amitsela opened a pull request:

https://github.com/apache/beam/pull/1758

[BEAM-1256] SparkPipelineResult.StreamingMode should not return null …

…on awaitTermination()

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amitsela/beam BEAM-1256

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1758


commit 7af52a1a60e37dbf52cf1d78dbc9ede0cb7eba84
Author: Sela 
Date:   2017-01-10T17:37:20Z

[BEAM-1256] SparkPipelineResult.StreamingMode should not return null on 
awaitTermination()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-1256) SparkPipelineResult.StreamingMode should not return null on awaitTermination()

[Amit Sela (JIRA)]

Amit Sela created BEAM-1256:
---

 Summary: SparkPipelineResult.StreamingMode should not return null 
on awaitTermination()
 Key: BEAM-1256
 URL: https://issues.apache.org/jira/browse/BEAM-1256
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Amit Sela
Assignee: Amit Sela
Priority: Minor


Currently, {{SparkPipelineResult.StreamingMode}} will return {{null}} if 
{{JavaStreamingContext#awaitTerminationOrTimeout(Long)}} returns {{false}}.
This is wrong because this will always return {{false}} and so {{null}} because 
Spark's {{ContextWaiter}} will only return {{true}} if the context was stopped 
and this only happens after calling "awaitTermination".

Following Spark code:
awaitTermination returns the [state of 
stopped|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/ContextWaiter.scala#L74]
But can only stopped via 
[notifyStopped|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/ContextWaiter.scala#L44]
Which is only called by 
[StreamingContext#stop()|https://github.com/apache/spark/blob/branch-1.6/streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala#L714]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-1253) Replace Twitter header image

[James Malone (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Malone closed BEAM-1253.
--
   Resolution: Fixed
Fix Version/s: Not applicable

All done!

> Replace Twitter header image
> 
>
> Key: BEAM-1253
> URL: https://issues.apache.org/jira/browse/BEAM-1253
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: James Malone
>Assignee: James Malone
>Priority: Minor
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1255) java.io.NotSerializableException in flink on UnboundedSource

[Alexey Diomin (JIRA)]

Alexey Diomin created BEAM-1255:
---

 Summary: java.io.NotSerializableException in flink on 
UnboundedSource
 Key: BEAM-1255
 URL: https://issues.apache.org/jira/browse/BEAM-1255
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Affects Versions: 0.5.0
Reporter: Alexey Diomin
Assignee: Maximilian Michels


After introduce new Coders with TypeDescriptor on flink runner we have issue:

{code}
Caused by: java.io.NotSerializableException: 
sun.reflect.generics.reflectiveObjects.TypeVariableImpl
- element of array (index: 0)
- array (class "[Ljava.lang.Object;", size: 2)
- field (class 
"com.google.common.collect.ImmutableList$SerializedForm", name: "elements", 
type: "class [Ljava.lang.Object;")
- object (class 
"com.google.common.collect.ImmutableList$SerializedForm", 
com.google.common.collect.ImmutableList$SerializedForm@30af5b6b)
- field (class "com.google.common.reflect.Types$ParameterizedTypeImpl", 
name: "argumentsList", type: "class com.google.common.collect.ImmutableList")
- object (class 
"com.google.common.reflect.Types$ParameterizedTypeImpl", 
org.apache.beam.sdk.io.UnboundedSource)
- field (class "com.google.common.reflect.TypeToken", name: 
"runtimeType", type: "interface java.lang.reflect.Type")
- object (class "com.google.common.reflect.TypeToken$SimpleTypeToken", 
org.apache.beam.sdk.io.UnboundedSource)
- field (class "org.apache.beam.sdk.values.TypeDescriptor", name: 
"token", type: "class com.google.common.reflect.TypeToken")
- object (class 
"org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper$1",
 org.apache.beam.sdk.io.UnboundedSource)
- field (class "org.apache.beam.sdk.coders.SerializableCoder", name: 
"typeDescriptor", type: "class org.apache.beam.sdk.values.TypeDescriptor")
- object (class "org.apache.beam.sdk.coders.SerializableCoder", 
SerializableCoder)
- field (class "org.apache.beam.sdk.coders.KvCoder", name: "keyCoder", 
type: "interface org.apache.beam.sdk.coders.Coder")
- object (class "org.apache.beam.sdk.coders.KvCoder", 
KvCoder(SerializableCoder,AvroCoder))
- field (class "org.apache.beam.sdk.coders.IterableLikeCoder", name: 
"elementCoder", type: "interface org.apache.beam.sdk.coders.Coder")
- object (class "org.apache.beam.sdk.coders.ListCoder", 
ListCoder(KvCoder(SerializableCoder,AvroCoder)))
- field (class 
"org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper",
 name: "checkpointCoder", type: "class org.apache.beam.sdk.coders.ListCoder")
- root object (class 
"org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper",
 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper@b2c5e07)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1182)
{code}

bug introduced after commit:
7b98fa08d14e8121e8885f00a9a9a878b73f81a6

pull request:
https://github.com/apache/beam/pull/1537

Code for reproduce error
{code}
import com.google.common.collect.ImmutableList;
import org.apache.beam.runners.flink.FlinkPipelineOptions;
import org.apache.beam.runners.flink.FlinkRunner;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.io.kafka.KafkaIO;
import org.apache.beam.sdk.options.PipelineOptionsFactory;

public class FlinkSerialisationError {

public static void main(String[] args) {
FlinkPipelineOptions options = 
PipelineOptionsFactory.as(FlinkPipelineOptions.class);
options.setRunner(FlinkRunner.class);
options.setStreaming(true);


Pipeline pipeline = Pipeline.create(options);

pipeline.apply(
KafkaIO.read()
.withBootstrapServers("localhost:9092")
.withTopics(ImmutableList.of("test"))
// set ConsumerGroup
.withoutMetadata());

pipeline.run();
}
}
{code}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (BEAM-1254) SuppressWarnings("serial") for starter and example archetypes

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-1254:
-

 Summary: SuppressWarnings("serial") for starter and example 
archetypes
 Key: BEAM-1254
 URL: https://issues.apache.org/jira/browse/BEAM-1254
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Kenneth Knowles
Priority: Minor


The usual case for using Beam is that every user `DoFn`, etc, is serialized 
with autogenerated `serialVersionUID`. Our starter and example archetypes 
should reflect this, so when someone loads them up in an IDE they are not 
presented with warning-throwing code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[3/3] beam git commit: This closes #1624

[amitsela]

This closes #1624


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/c1b7f869
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/c1b7f869
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/c1b7f869

Branch: refs/heads/master
Commit: c1b7f8695a599981e549fad598f36a1559971859
Parents: fe7fc29 662934b
Author: Sela 
Authored: Tue Jan 10 19:13:37 2017 +0200
Committer: Sela 
Committed: Tue Jan 10 19:13:37 2017 +0200

--
 .../runners/spark/translation/DoFnFunction.java |   6 +-
 .../spark/translation/EvaluationContext.java|  34 +++--
 .../translation/GroupCombineFunctions.java  |   8 +-
 .../spark/translation/MultiDoFnFunction.java|   7 +-
 .../translation/SparkAbstractCombineFn.java |  12 +-
 .../spark/translation/SparkGlobalCombineFn.java |  13 +-
 .../spark/translation/SparkKeyedCombineFn.java  |  13 +-
 .../spark/translation/SparkPCollectionView.java |  99 +++
 .../spark/translation/TransformTranslator.java  |  33 -
 .../spark/translation/TranslationUtils.java |  37 --
 .../streaming/StreamingTransformTranslator.java |  75 +++
 .../runners/spark/util/BroadcastHelper.java | 127 ---
 .../runners/spark/util/SideInputBroadcast.java  |  77 +++
 .../spark/util/SparkSideInputReader.java|   8 +-
 .../ResumeFromCheckpointStreamingTest.java  |  18 +++
 .../src/main/resources/beam/findbugs-filter.xml |  26 
 16 files changed, 353 insertions(+), 240 deletions(-)
--

[2/3] beam git commit: Create broadcast lazily

[amitsela]

Create broadcast lazily

Fix Amit's comments + rename BroadcastHelper to SideInputBroadcast


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/662934b1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/662934b1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/662934b1

Branch: refs/heads/master
Commit: 662934b1e88c9697585b05e29d9e7b4a34fc6943
Parents: 130c113
Author: ksalant 
Authored: Thu Jan 5 09:40:50 2017 +0200
Committer: Sela 
Committed: Tue Jan 10 19:12:25 2017 +0200

--
 .../runners/spark/translation/DoFnFunction.java |  6 +-
 .../spark/translation/EvaluationContext.java| 11 ++-
 .../translation/GroupCombineFunctions.java  |  8 +-
 .../spark/translation/MultiDoFnFunction.java|  7 +-
 .../translation/SparkAbstractCombineFn.java | 12 +--
 .../spark/translation/SparkGlobalCombineFn.java | 13 ++--
 .../spark/translation/SparkKeyedCombineFn.java  | 13 ++--
 .../spark/translation/SparkPCollectionView.java | 42 +-
 .../spark/translation/TransformTranslator.java  | 20 ++---
 .../spark/translation/TranslationUtils.java | 25 +++---
 .../streaming/StreamingTransformTranslator.java | 20 ++---
 .../runners/spark/util/BroadcastHelper.java | 82 
 .../runners/spark/util/SideInputBroadcast.java  | 77 ++
 .../spark/util/SparkSideInputReader.java|  9 +--
 .../ResumeFromCheckpointStreamingTest.java  |  3 +-
 .../src/main/resources/beam/findbugs-filter.xml | 26 ---
 16 files changed, 166 insertions(+), 208 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/662934b1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
index 6a641b5..af8e089 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/DoFnFunction.java
@@ -27,7 +27,7 @@ import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.spark.aggregators.NamedAggregators;
 import org.apache.beam.runners.spark.aggregators.SparkAggregators;
-import org.apache.beam.runners.spark.util.BroadcastHelper;
+import org.apache.beam.runners.spark.util.SideInputBroadcast;
 import org.apache.beam.runners.spark.util.SparkSideInputReader;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.util.WindowedValue;
@@ -50,7 +50,7 @@ public class DoFnFunction
   private final Accumulator accumulator;
   private final DoFn doFn;
   private final SparkRuntimeContext runtimeContext;
-  private final Map> sideInputs;
+  private final Map> sideInputs;
   private final WindowingStrategy windowingStrategy;
 
   /**
@@ -64,7 +64,7 @@ public class DoFnFunction
   Accumulator accumulator,
   DoFn doFn,
   SparkRuntimeContext runtimeContext,
-  Map> 
sideInputs,
+  Map> 
sideInputs,
   WindowingStrategy windowingStrategy) {
 
 this.accumulator = accumulator;

http://git-wip-us.apache.org/repos/asf/beam/blob/662934b1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
--
diff --git 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
index b1a1142..0ad862d 100644
--- 
a/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
+++ 
b/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/EvaluationContext.java
@@ -192,23 +192,26 @@ public class EvaluationContext {
   }
 
   /**
-   * Retruns the current views creates in the pipepline.
+   * Retrun the current views creates in the pipepline.
+   *
* @return SparkPCollectionView
*/
-  public SparkPCollectionView getPviews() {
+  public SparkPCollectionView getPViews() {
 return pviews;
   }
 
   /**
* Adds/Replaces a view to the current views creates in the pipepline.
+   *
* @param view - Identifier of the view
* @param value - Actual value of the view
* @param coder - Coder of the 

[GitHub] beam pull request #1755: [BEAM-1248] Combine with side inputs API should mat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1755


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (BEAM-1223) Add static factory methods for Sum.[*]Fn classes

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815469#comment-15815469
 ] 

Luke Cwik edited comment on BEAM-1223 at 1/10/17 4:47 PM:
--

Interfaces and base classes can't be used to enforce the presence of static 
methods in java.

I do believe this pattern could be extended to the other PTransforms.


was (Author: lcwik):
Interfaces and base classes can't be used to enforce the presence of static 
methods in java.

I do believe should a pattern could be extended to the other PTransforms.

> Add static factory methods for Sum.[*]Fn classes
> 
>
> Key: BEAM-1223
> URL: https://issues.apache.org/jira/browse/BEAM-1223
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Stas Levin
>Assignee: Stas Levin
>Priority: Minor
>
> {{Sum.SumDoubleFn}}, {{SumIntegerFn}} and {{SumLongFn}} are not using the 
> {{X.of()}} or {{X.from()}} or other instance creation via static method 
> patterns that are ubiquitous in Beam.
> Following a discussion on the dev list, it would be great to preserve a 
> consistent look and feel and change the creation patterns for these classes 
> to something like {{SumFn.ofLong()}} etc.
> See also the corresponding [dev list 
> thread|https://lists.apache.org/thread.html/5d8e905ee49b116d13811c2a96da65eeb44ab7c002870f50936ee1ad@%3Cdev.beam.apache.org%3E].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (BEAM-1177) Input DStream "bundles" should be in serialized form and include relevant metadata.

[Amit Sela (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Amit Sela closed BEAM-1177.
---

> Input DStream "bundles" should be in serialized form and include relevant 
> metadata.
> ---
>
> Key: BEAM-1177
> URL: https://issues.apache.org/jira/browse/BEAM-1177
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Amit Sela
>Assignee: Amit Sela
> Fix For: 0.5.0
>
>
> Currently, the input partitions hold "bundles" of read elements within the 
> {{mapWithStateDStream}} used for the read.
> Since this is automatically shuffled, user-data (the read elements) should be 
> serialized using coders to avoid breaking (if user-data is not {{Kryo}} 
> serializable).
> Even after BEAM-848 would complete, the resulting {{MapWithStateDStream}} 
> would be checkpointed periodically and so it would still have to remain in 
> serialized form.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] beam-site pull request #124: Blog post on Beam graduation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/124


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #124

[davor]

This closes #124


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/c9379f5e
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/c9379f5e
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/c9379f5e

Branch: refs/heads/asf-site
Commit: c9379f5e3ddc8e356c96312e012e248fa95f81d2
Parents: e41c4b4 eaca9f5
Author: Davor Bonaci 
Authored: Tue Jan 10 03:03:10 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 03:03:10 2017 -0800

--
 content/blog/2017/01/10/beam-graduates.html  | 242 ++
 content/blog/index.html  |  20 +
 content/contribute/maturity-model/index.html |   4 +-
 content/feed.xml | 864 ++
 content/index.html   |   4 +-
 src/_posts/2017-01-10-beam-graduates.md  |  65 ++
 6 files changed, 399 insertions(+), 800 deletions(-)
--

[1/3] beam-site git commit: Blog post on Beam graduation

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site e41c4b4f7 -> c9379f5e3


Blog post on Beam graduation


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/df96c7d4
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/df96c7d4
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/df96c7d4

Branch: refs/heads/asf-site
Commit: df96c7d4a08aff9aaa7e2b5da41ad19d51cda2dd
Parents: e41c4b4
Author: Davor Bonaci 
Authored: Mon Jan 9 13:12:54 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 03:02:02 2017 -0800

--
 src/_posts/2017-01-10-beam-graduates.md | 65 
 1 file changed, 65 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/df96c7d4/src/_posts/2017-01-10-beam-graduates.md
--
diff --git a/src/_posts/2017-01-10-beam-graduates.md 
b/src/_posts/2017-01-10-beam-graduates.md
new file mode 100644
index 000..f17eebd
--- /dev/null
+++ b/src/_posts/2017-01-10-beam-graduates.md
@@ -0,0 +1,65 @@
+---
+layout: post
+title:  "Apache Beam established as a new top-level project"
+date:   2017-01-10 00:00:01 -0800
+excerpt_separator: 
+categories: blog
+authors:
+  - davor
+---
+
+Today, the Apache Software Foundation 
[announced](https://blogs.apache.org/foundation/entry/the-apache-software-foundation-announces)
+that Apache Beam has successfully graduated from incubation, becoming a new
+Top-Level Project at the foundation and signifying that its "community and
+products have been well-governed under the foundation’s meritocratic process
+and principles".
+
+
+
+Graduation is an exciting milestone for Apache Beam. Becoming a top-level
+project is a recognition of the amazing growth of the Apache Beam community,
+both in terms of size and diversity. It is fantastic to see ever-increasing
+participation on our development and user mailing lists, in pull requests,
+and at conferences and meetups. Contributions of additional runners and IO
+connectors further show the significant interest from other projects and
+organizations in being part of our growing community.
+
+Although graduation is a statement about community, our technical progress has
+also been impressive. We started with code donations from several companies
+that were tailored with a specific vendor and scenario in mind and had fragile
+component boundaries. The whole community worked extremely hard over the past
+year to refactor the codebase and documentation into a truly vendor-neutral and
+extensible framework. We have established an impressive engineering system and
+processes that promote test-driven development. This concerted focus on quality
+and execution excellence enabled us to publish four releases, each 
significantly
+better than the previous.
+
+Going forward, we will continue to extend the core abstractions to distill
+additional complex data processing patterns into intuitive APIs, and, at the
+same time, enhance the ability to interconnect additional storage/messaging
+systems and execution engines. Together, we are excited to push forward the
+state of the art in distributed data processing.
+
+This is also an opportunity to thank those who helped and supported us reaching
+this milestone. None of this would be possible without the Apache Software
+Foundation and its amazing volunteers who have wholeheartedly welcomed us into
+the family. In particular, we’d like to thank the Apache Incubator community
+for their guidance producing releases and following processes. We are 
especially
+thankful to the Apache Infra team for their assistance establishing Beam’s
+engineering system, particularly in the areas of testing infrastructure and
+development productivity.
+
+On behalf of the whole Apache Beam community, thank you to our incubation
+mentors Ted Dunning, Venkatesh Seetharam, and, previously, Bertrand Delacretaz
+and Jim Jagielski, for teaching us the Apache Way and guiding us through
+incubation. In particular, special thanks goes to our incubation champion and
+mentor Jean-Baptiste Onofré, whose extraordinary help, day after day, was
+instrumental in getting us here, and who continues as an active code 
contributor
+and a member of the project management committee.
+
+Please consider joining us, whether as a user or a contributor, as we work
+towards our first release with API stability. If you’d like to try out Apache
+Beam today, check out the latest
+[0.4.0 release]({{ site.baseurl }}/get-started/downloads/). We welcome
+contribution and participation from anyone through our mailing lists, issue
+tracker, pull requests, and events.

[GitHub] beam-site pull request #125: Update maturity model to reflect graduation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/125


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam-site git commit: Update maturity model to reflect graduation

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 49534e9f9 -> e41c4b4f7


Update maturity model to reflect graduation


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/3c859318
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/3c859318
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/3c859318

Branch: refs/heads/asf-site
Commit: 3c8593183d31474d277fdf2429895071cff63e76
Parents: 49534e9
Author: Davor Bonaci 
Authored: Mon Jan 9 18:26:12 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 02:58:50 2017 -0800

--
 src/contribute/maturity-model.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/3c859318/src/contribute/maturity-model.md
--
diff --git a/src/contribute/maturity-model.md b/src/contribute/maturity-model.md
index 3ae90de..6fe272b 100644
--- a/src/contribute/maturity-model.md
+++ b/src/contribute/maturity-model.md
@@ -6,7 +6,9 @@ permalink: /contribute/maturity-model/
 
 # Apache Maturity Model Assessment for Apache Beam
 
-*Last updated: November 22, 2016*
+*Apache Beam has graduated from incubation as a top-level project at the
+Apache Software Foundation. This page was last updated as a part of the
+graduation process and is no longer being maintained.*
 
 * TOC
 {:toc}

[2/2] beam-site git commit: This closes #125

[davor]

This closes #125


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/e41c4b4f
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/e41c4b4f
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/e41c4b4f

Branch: refs/heads/asf-site
Commit: e41c4b4f7ce977bb255d041201c7bc3c2347e804
Parents: 49534e9 3c85931
Author: Davor Bonaci 
Authored: Tue Jan 10 02:59:21 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 02:59:21 2017 -0800

--
 src/contribute/maturity-model.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
--

[1/3] beam-site git commit: Update copyright on each page

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 98820c672 -> 49534e9f9


Update copyright on each page

Update year, fix ordering, make it automatically update.

Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/61d1fd6c
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/61d1fd6c
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/61d1fd6c

Branch: refs/heads/asf-site
Commit: 61d1fd6ceee9ac698f777e973d5117d06ef51646
Parents: 98820c6
Author: Davor Bonaci 
Authored: Tue Jan 10 01:49:18 2017 -0800
Committer: GitHub 
Committed: Tue Jan 10 01:49:18 2017 -0800

--
 src/_includes/footer.html | 13 +
 1 file changed, 9 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/61d1fd6c/src/_includes/footer.html
--
diff --git a/src/_includes/footer.html b/src/_includes/footer.html
index 85830ef..8f7798f 100644
--- a/src/_includes/footer.html
+++ b/src/_includes/footer.html
@@ -2,10 +2,15 @@
   
   
   
-   Copyright 2016
-http://www.apache.org;>The Apache Software 
Foundation. All Rights Reserved.
-Privacy Policy |
-RSS 
Feed
+  
+ Copyright
+http://www.apache.org;>The Apache Software 
Foundation,
+{{ 'now' | date: "%Y" }}. All Rights Reserved.
+  
+  
+Privacy Policy 
|
+RSS 
Feed
+  
   
   
   

[3/3] beam-site git commit: This closes #127

[davor]

This closes #127


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/49534e9f
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/49534e9f
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/49534e9f

Branch: refs/heads/asf-site
Commit: 49534e9f9449a4a87ef400359eb06a99570f601d
Parents: 98820c6 ac3b3d8
Author: Davor Bonaci 
Authored: Tue Jan 10 01:57:16 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 01:57:16 2017 -0800

--
 .../beam/capability/2016/03/17/capability-matrix.html  | 13 +
 .../capability/2016/04/03/presentation-materials.html  | 13 +
 .../python/sdk/2016/02/25/python-sdk-now-public.html   | 13 +
 content/beam/release/2016/06/15/first-release.html | 13 +
 .../2016/10/11/strata-hadoop-world-and-beam.html   | 13 +
 .../update/website/2016/02/22/beam-has-a-logo.html | 13 +
 content/blog/2016/05/18/splitAtFraction-method.html| 13 +
 .../2016/05/27/where-is-my-pcollection-dot-map.html| 13 +
 .../blog/2016/06/13/flink-batch-runner-milestone.html  | 13 +
 content/blog/2016/08/03/six-months.html| 13 +
 content/blog/2016/10/20/test-stream.html   | 13 +
 content/blog/2017/01/09/added-apex-runner.html | 13 +
 content/blog/index.html| 13 +
 content/coming-soon.html   | 13 +
 content/contribute/contribution-guide/index.html   | 13 +
 content/contribute/design-principles/index.html| 13 +
 content/contribute/index.html  | 13 +
 content/contribute/logos/index.html| 13 +
 content/contribute/maturity-model/index.html   | 13 +
 content/contribute/presentation-materials/index.html   | 13 +
 content/contribute/release-guide/index.html| 13 +
 content/contribute/source-repository/index.html| 13 +
 content/contribute/team/index.html | 13 +
 content/contribute/testing/index.html  | 13 +
 content/contribute/work-in-progress/index.html | 13 +
 content/documentation/index.html   | 13 +
 .../pipelines/create-your-pipeline/index.html  | 13 +
 .../pipelines/design-your-pipeline/index.html  | 13 +
 .../pipelines/test-your-pipeline/index.html| 13 +
 content/documentation/programming-guide/index.html | 13 +
 content/documentation/resources/index.html | 13 +
 content/documentation/runners/apex/index.html  | 13 +
 .../documentation/runners/capability-matrix/index.html | 13 +
 content/documentation/runners/dataflow/index.html  | 13 +
 content/documentation/runners/direct/index.html| 13 +
 content/documentation/runners/flink/index.html | 13 +
 content/documentation/runners/spark/index.html | 13 +
 content/documentation/sdks/java/index.html | 13 +
 content/documentation/sdks/python/index.html   | 13 +
 content/get-started/beam-overview/index.html   | 13 +
 content/get-started/downloads/index.html   | 13 +
 content/get-started/index.html | 13 +
 content/get-started/mobile-gaming-example/index.html   | 13 +
 content/get-started/quickstart/index.html  | 13 +
 content/get-started/support/index.html | 13 +
 content/get-started/wordcount-example/index.html   | 13 +
 content/index.html | 13 +
 content/privacy_policy/index.html  | 13 +
 src/_includes/footer.html  | 13 +
 49 files changed, 441 insertions(+), 196 deletions(-)
--

[GitHub] beam-site pull request #127: Update copyright on each page

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/127


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/ac3b3d8d
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/ac3b3d8d
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/ac3b3d8d

Branch: refs/heads/asf-site
Commit: ac3b3d8d3ac20a20a33e568c885bae43bb406925
Parents: 61d1fd6
Author: Davor Bonaci 
Authored: Tue Jan 10 01:57:15 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 01:57:15 2017 -0800

--
 .../beam/capability/2016/03/17/capability-matrix.html  | 13 +
 .../capability/2016/04/03/presentation-materials.html  | 13 +
 .../python/sdk/2016/02/25/python-sdk-now-public.html   | 13 +
 content/beam/release/2016/06/15/first-release.html | 13 +
 .../2016/10/11/strata-hadoop-world-and-beam.html   | 13 +
 .../update/website/2016/02/22/beam-has-a-logo.html | 13 +
 content/blog/2016/05/18/splitAtFraction-method.html| 13 +
 .../2016/05/27/where-is-my-pcollection-dot-map.html| 13 +
 .../blog/2016/06/13/flink-batch-runner-milestone.html  | 13 +
 content/blog/2016/08/03/six-months.html| 13 +
 content/blog/2016/10/20/test-stream.html   | 13 +
 content/blog/2017/01/09/added-apex-runner.html | 13 +
 content/blog/index.html| 13 +
 content/coming-soon.html   | 13 +
 content/contribute/contribution-guide/index.html   | 13 +
 content/contribute/design-principles/index.html| 13 +
 content/contribute/index.html  | 13 +
 content/contribute/logos/index.html| 13 +
 content/contribute/maturity-model/index.html   | 13 +
 content/contribute/presentation-materials/index.html   | 13 +
 content/contribute/release-guide/index.html| 13 +
 content/contribute/source-repository/index.html| 13 +
 content/contribute/team/index.html | 13 +
 content/contribute/testing/index.html  | 13 +
 content/contribute/work-in-progress/index.html | 13 +
 content/documentation/index.html   | 13 +
 .../pipelines/create-your-pipeline/index.html  | 13 +
 .../pipelines/design-your-pipeline/index.html  | 13 +
 .../pipelines/test-your-pipeline/index.html| 13 +
 content/documentation/programming-guide/index.html | 13 +
 content/documentation/resources/index.html | 13 +
 content/documentation/runners/apex/index.html  | 13 +
 .../documentation/runners/capability-matrix/index.html | 13 +
 content/documentation/runners/dataflow/index.html  | 13 +
 content/documentation/runners/direct/index.html| 13 +
 content/documentation/runners/flink/index.html | 13 +
 content/documentation/runners/spark/index.html | 13 +
 content/documentation/sdks/java/index.html | 13 +
 content/documentation/sdks/python/index.html   | 13 +
 content/get-started/beam-overview/index.html   | 13 +
 content/get-started/downloads/index.html   | 13 +
 content/get-started/index.html | 13 +
 content/get-started/mobile-gaming-example/index.html   | 13 +
 content/get-started/quickstart/index.html  | 13 +
 content/get-started/support/index.html | 13 +
 content/get-started/wordcount-example/index.html   | 13 +
 content/index.html | 13 +
 content/privacy_policy/index.html  | 13 +
 48 files changed, 432 insertions(+), 192 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/ac3b3d8d/content/beam/capability/2016/03/17/capability-matrix.html
--
diff --git a/content/beam/capability/2016/03/17/capability-matrix.html 
b/content/beam/capability/2016/03/17/capability-matrix.html
index 7c19748..285b4b4 100644
--- a/content/beam/capability/2016/03/17/capability-matrix.html
+++ b/content/beam/capability/2016/03/17/capability-matrix.html
@@ -949,10 +949,15 @@
   
   
   
-   Copyright 2016
-http://www.apache.org;>The Apache Software 
Foundation. All Rights 

[GitHub] beam-site pull request #127: Update copyright on each page

[davorbonaci]

GitHub user davorbonaci opened a pull request:

https://github.com/apache/beam-site/pull/127

Update copyright on each page

Update year, fix ordering, make it automatically update.

To be reviewed later by R: @francesperry.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/davorbonaci/beam-site patch-4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #127


commit 61d1fd6ceee9ac698f777e973d5117d06ef51646
Author: Davor Bonaci 
Date:   2017-01-10T09:49:18Z

Update copyright on each page

Update year, fix ordering, make it automatically update.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam-site pull request #126: Limit the number of blog posts listed on the ma...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/3] beam-site git commit: Limit the number of blog posts listed on the main page

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site a46c512c3 -> 98820c672


Limit the number of blog posts listed on the main page

Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/add05e1e
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/add05e1e
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/add05e1e

Branch: refs/heads/asf-site
Commit: add05e1e00031134afa538c7b8259fa47ffb9f7c
Parents: a46c512
Author: Davor Bonaci 
Authored: Tue Jan 10 00:31:03 2017 -0800
Committer: GitHub 
Committed: Tue Jan 10 00:31:03 2017 -0800

--
 src/index.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/add05e1e/src/index.md
--
diff --git a/src/index.md b/src/index.md
index bb51dfd..535051f 100644
--- a/src/index.md
+++ b/src/index.md
@@ -24,7 +24,7 @@ Beam is an [Apache Software 
Foundation](http://www.apache.org) project, availabl
   
 Blog
 
-{% for post in site.posts %}
+{% for post in site.posts limit:7 %}
 {{ post.date | date: "%b %-d, %Y" }} - {{ post.title }}
 {% endfor %}
 

[3/3] beam-site git commit: This closes #126

[davor]

This closes #126


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/98820c67
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/98820c67
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/98820c67

Branch: refs/heads/asf-site
Commit: 98820c6726b91986e767be170a8a9ecef004f593
Parents: a46c512 8e9900d
Author: Davor Bonaci 
Authored: Tue Jan 10 01:05:30 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 01:05:30 2017 -0800

--
 content/index.html | 10 --
 src/index.md   |  2 +-
 2 files changed, 1 insertion(+), 11 deletions(-)
--

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8e9900da
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8e9900da
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8e9900da

Branch: refs/heads/asf-site
Commit: 8e9900dab746c0ce7ffc490452b5d1effa855fcd
Parents: add05e1
Author: Davor Bonaci 
Authored: Tue Jan 10 01:05:30 2017 -0800
Committer: Davor Bonaci 
Committed: Tue Jan 10 01:05:30 2017 -0800

--
 content/index.html | 10 --
 1 file changed, 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/8e9900da/content/index.html
--
diff --git a/content/index.html b/content/index.html
index 54682f7..1dc94cd 100644
--- a/content/index.html
+++ b/content/index.html
@@ -184,16 +184,6 @@
 
 May 27, 2016 - 
Where's my PCollection.map()?
 
-May 18, 2016 - Dynamic work 
rebalancing for Beam
-
-Apr 3, 2016 - 
Apache Beam Presentation Materials
-
-Mar 17, 2016 - 
Clarifying  Formalizing Runner Capabilities
-
-Feb 25, 2016 - 
Dataflow Python SDK is now public!
-
-Feb 22, 2016 - 
Apache Beam has a logo!
-
 
   
   

[GitHub] beam pull request #1757: Compressed file with missing last EOF create a fake...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1757

Compressed file with missing last EOF create a fake element

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/incubator-beam 
BEAM-compressed-file-testing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1757


commit 9e8d59822edaf86f83b9ca89d5f3cf374bcddc14
Author: Sourabh Bajaj 
Date:   2017-01-10T01:18:16Z

Compressed file with missing last EOF create a fake element




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1188) More Verifiers For Python E2E Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814249#comment-15814249
 ] 

ASF GitHub Bot commented on BEAM-1188:
--

GitHub user markflyhigh opened a pull request:

https://github.com/apache/beam/pull/1756

[BEAM-1188] Python File Verifer For E2E Tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Add FileChecksumVerifier to verify E2E test output file(s) locally or on 
GCS:
 - Refactor TestPipeline to be clean and able to get option value by name
 - Create FileChecksumVerifier with retry when IO failed.
 - Add FileChecksumVerifier to wordcount e2e test
 - Create test_utils to hold utility method for testing.

Test is done by running wordcount_it against Dataflow service.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
file-checksum-verifier

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1756.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1756


commit 16109f1651dd9537464687129d8d0c0f3a6a4a91
Author: Mark Liu 
Date:   2017-01-10T07:48:42Z

[BEAM-1188] Python File Verifer For E2E Tests




> More Verifiers For Python E2E Tests
> ---
>
> Key: BEAM-1188
> URL: https://issues.apache.org/jira/browse/BEAM-1188
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py, testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>
> Add more basic verifiers in e2e test to verify output data in different 
> storage/fs:
> 1. File verifier: compute and verify checksum of file(s) that’s stored on a 
> filesystem (GCS / local fs). 
> 2. Bigquery verifier: query from Bigquery table and verify response content. 
> ...
> Also update TestOptions.on_success_matcher to accept a list of matchers 
> instead of single one.
> Note: Have retry when doing IO to avoid test flacky that may come from 
> inconsistency of the filesystem. This problem happened in Java integration 
> tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)