[jira] [Commented] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040198#comment-16040198
 ] 

ASF GitHub Bot commented on BEAM-2405:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3306


> Create a BigQuery sink for streaming using PTransform
> -
>
> Key: BEAM-2405
> URL: https://issues.apache.org/jira/browse/BEAM-2405
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3306: [BEAM-2405] Override to sink interface in batch dat...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3306


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3306

[chamikara]

This closes #3306


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3cc4ff6d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3cc4ff6d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3cc4ff6d

Branch: refs/heads/master
Commit: 3cc4ff6d72cf74e91b0a0d9cdd4277288958c242
Parents: b6347d0 e641997
Author: chamik...@google.com 
Authored: Tue Jun 6 22:17:18 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Jun 6 22:17:18 2017 -0700

--
 .../examples/cookbook/bigquery_tornadoes.py   | 11 +--
 sdks/python/apache_beam/io/gcp/bigquery.py|  2 +-
 .../runners/dataflow/dataflow_runner.py   | 18 ++
 3 files changed, 24 insertions(+), 7 deletions(-)
--

[1/2] beam git commit: [BEAM-2405] Override to sink interface in the batch dataflow BQ

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master b6347d02c -> 3cc4ff6d7


[BEAM-2405] Override to sink interface in the batch dataflow BQ


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e641997a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e641997a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e641997a

Branch: refs/heads/master
Commit: e641997affc378ec0337d5ac19d8677cba0d0933
Parents: b6347d0
Author: Sourabh Bajaj 
Authored: Tue Jun 6 19:49:54 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Jun 6 22:17:05 2017 -0700

--
 .../examples/cookbook/bigquery_tornadoes.py   | 11 +--
 sdks/python/apache_beam/io/gcp/bigquery.py|  2 +-
 .../runners/dataflow/dataflow_runner.py   | 18 ++
 3 files changed, 24 insertions(+), 7 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/e641997a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py 
b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
index d3b216e..1ca49c5 100644
--- a/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
+++ b/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
@@ -83,12 +83,11 @@ def run(argv=None):
 
 # Write the output using a "Write" transform that has side effects.
 # pylint: disable=expression-not-assigned
-counts | 'write' >> beam.io.Write(
-beam.io.BigQuerySink(
-known_args.output,
-schema='month:INTEGER, tornado_count:INTEGER',
-create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
-write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE))
+counts | 'Write' >> beam.io.WriteToBigQuery(
+known_args.output,
+schema='month:INTEGER, tornado_count:INTEGER',
+create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE)
 
 # Run the pipeline (all operations are deferred until run() is called).
 

http://git-wip-us.apache.org/repos/asf/beam/blob/e641997a/sdks/python/apache_beam/io/gcp/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py 
b/sdks/python/apache_beam/io/gcp/bigquery.py
index 9069f73..da8be68 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -1299,7 +1299,7 @@ class WriteToBigQuery(PTransform):
 create_disposition=self.create_disposition,
 write_disposition=self.write_disposition,
 client=self.test_client)
-return pcoll | 'Write to BigQuery' >> ParDo(bigquery_write_fn)
+return pcoll | 'WriteToBigQuery' >> ParDo(bigquery_write_fn)
 
   def display_data(self):
 res = {}

http://git-wip-us.apache.org/repos/asf/beam/blob/e641997a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 62cea33..3fc8983 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -27,6 +27,7 @@ import time
 import traceback
 import urllib
 
+import apache_beam as beam
 from apache_beam import error
 from apache_beam import coders
 from apache_beam import pvalue
@@ -378,6 +379,23 @@ class DataflowRunner(PipelineRunner):
   PropertyNames.ENCODING: step.encoding,
   PropertyNames.OUTPUT_NAME: PropertyNames.OUT}])
 
+  def apply_WriteToBigQuery(self, transform, pcoll):
+standard_options = pcoll.pipeline._options.view_as(StandardOptions)
+if standard_options.streaming:
+  if (transform.write_disposition ==
+  beam.io.BigQueryDisposition.WRITE_TRUNCATE):
+raise RuntimeError('Can not use write truncation mode in streaming')
+  return self.apply_PTransform(transform, pcoll)
+else:
+  return pcoll  | 'WriteToBigQuery' >> beam.io.Write(
+  beam.io.BigQuerySink(
+  transform.table_reference.tableId,
+  transform.table_reference.datasetId,
+  transform.table_reference.projectId,
+  transform.schema,
+  transform.create_disposition,
+  transform.write_disposition))
+
   def apply_GroupByKey(self, transform, pcoll):
 # Infer coder of parent.
 #

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Colin Bookman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040093#comment-16040093
 ] 

Colin Bookman commented on BEAM-2418:
-

[~lcwik], I the ProtobufCoderProviderRegistar is included in the jar file. 
Here's the jar file in question: 
https://storage.googleapis.com/beam-dataflowio-bucket/dataflow-teleport-1.0-Alpha.jar
 

Here's the entire code for the Beam pipeline: 
https://github.com/cobookman/DatastoreToGCS/tree/beam
Here's the script / build I'm running and getting the error for: 
https://github.com/cobookman/DatastoreToGCS/blob/beam/scripts/datastore_to_gcs.sh



> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Vikas Kedigehalli
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4038

[Apache Jenkins Server]

See 


Changes:

[bchambers] Increase visibility of some Metrics methods

--
[...truncated 271.54 KB...]
2017-06-07T03:10:32.749 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jdt-core/2.0.1/jdt-core-2.0.1.pom
 (2 KB at 42.0 KB/sec)
2017-06-07T03:10:32.751 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/formatters/2.0.1/formatters-2.0.1.pom
2017-06-07T03:10:32.780 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/formatters/2.0.1/formatters-2.0.1.pom
 (4 KB at 128.6 KB/sec)
2017-06-07T03:10:32.783 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jsdt-core/2.0.1/jsdt-core-2.0.1.pom
2017-06-07T03:10:32.811 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jsdt-core/2.0.1/jsdt-core-2.0.1.pom
 (2 KB at 43.5 KB/sec)
2017-06-07T03:10:32.814 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-digester3/3.2/commons-digester3-3.2.pom
2017-06-07T03:10:32.843 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-digester3/3.2/commons-digester3-3.2.pom
 (18 KB at 592.3 KB/sec)
2017-06-07T03:10:32.847 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.pom
2017-06-07T03:10:32.876 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.pom (2 KB at 
51.4 KB/sec)
2017-06-07T03:10:32.877 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/asm/asm/3.3.1/asm-3.3.1.pom
2017-06-07T03:10:32.903 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/asm/asm/3.3.1/asm-3.3.1.pom (266 B at 10.0 
KB/sec)
2017-06-07T03:10:32.905 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/asm/asm-parent/3.3.1/asm-parent-3.3.1.pom
2017-06-07T03:10:32.932 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/asm/asm-parent/3.3.1/asm-parent-3.3.1.pom 
(5 KB at 156.7 KB/sec)
2017-06-07T03:10:32.934 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.pom
2017-06-07T03:10:32.963 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/commons-beanutils/commons-beanutils/1.8.3/commons-beanutils-1.8.3.pom
 (11 KB at 377.1 KB/sec)
2017-06-07T03:10:32.965 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom
2017-06-07T03:10:32.994 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/14/commons-parent-14.pom
 (31 KB at 1057.0 KB/sec)
2017-06-07T03:10:32.997 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.pom
2017-06-07T03:10:33.026 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-annotations/3.5/maven-plugin-annotations-3.5.pom
 (2 KB at 56.2 KB/sec)
2017-06-07T03:10:33.027 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-tools/3.5/maven-plugin-tools-3.5.pom
2017-06-07T03:10:33.057 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugin-tools/maven-plugin-tools/3.5/maven-plugin-tools-3.5.pom
 (16 KB at 527.0 KB/sec)
2017-06-07T03:10:33.061 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
2017-06-07T03:10:33.088 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-resources/1.0.1/plexus-resources-1.0.1.pom
 (2 KB at 47.7 KB/sec)
2017-06-07T03:10:33.090 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.8/plexus-utils-3.0.8.pom
2017-06-07T03:10:33.119 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-utils/3.0.8/plexus-utils-3.0.8.pom
 (4 KB at 105.8 KB/sec)
2017-06-07T03:10:33.123 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/20.0/guava-20.0.jar
2017-06-07T03:10:33.123 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jdt-core/2.0.1/jdt-core-2.0.1.jar
2017-06-07T03:10:33.123 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/net/revelc/code/formatter/jsdt-core/2.0.1/jsdt-core-2.0.1.jar
2017-06-07T03:10:33.124 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-digester3/3.2/commons-digester3-3.2.jar
2017-06-07T03:10:33.125 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.jar
2017-06-07T03:10:33.171 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/cglib/cglib/2.2.2/cglib-2.2.2.jar (281 KB 
at 5967.3 KB/sec)
2017-06-07T03:10:33.172 [INFO] Downloading: 

[GitHub] beam pull request #3307: Increase visibility of some Metrics methods

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3307


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3307: Increase visibility of some Metrics methods

[kenn]

This closes #3307: Increase visibility of some Metrics methods


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b6347d02
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b6347d02
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b6347d02

Branch: refs/heads/master
Commit: b6347d02ce3086a01e8dd59f80103e45f6af1b5c
Parents: 1d2000d 39674ca
Author: Kenneth Knowles 
Authored: Tue Jun 6 19:43:53 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue Jun 6 19:43:53 2017 -0700

--
 .../apache/beam/runners/core/metrics/CounterCell.java| 10 +++---
 .../org/apache/beam/runners/core/metrics/DirtyState.java |  4 +++-
 .../beam/runners/core/metrics/DistributionCell.java  | 10 +++---
 .../org/apache/beam/runners/core/metrics/GaugeCell.java  | 11 +++
 .../beam/runners/core/metrics/MetricsContainerImpl.java  |  4 +++-
 .../org/apache/beam/sdk/metrics/MetricsContainer.java|  3 ++-
 6 files changed, 29 insertions(+), 13 deletions(-)
--

[1/2] beam git commit: Increase visibility of some Metrics methods

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 1d2000d8c -> b6347d02c


Increase visibility of some Metrics methods

Also revise the Javadoc on MetricsContainers.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/39674ca8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/39674ca8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/39674ca8

Branch: refs/heads/master
Commit: 39674ca8d0ae1d82bfb5da6a81f26843613d3cd7
Parents: 6853d8e
Author: bchambers 
Authored: Tue Jun 6 15:08:46 2017 -0700
Committer: bchambers 
Committed: Tue Jun 6 15:08:55 2017 -0700

--
 .../apache/beam/runners/core/metrics/CounterCell.java| 10 +++---
 .../org/apache/beam/runners/core/metrics/DirtyState.java |  4 +++-
 .../beam/runners/core/metrics/DistributionCell.java  | 10 +++---
 .../org/apache/beam/runners/core/metrics/GaugeCell.java  | 11 +++
 .../beam/runners/core/metrics/MetricsContainerImpl.java  |  4 +++-
 .../org/apache/beam/sdk/metrics/MetricsContainer.java|  3 ++-
 6 files changed, 29 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/39674ca8/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/CounterCell.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/CounterCell.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/CounterCell.java
index 4378bb9..886d681 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/CounterCell.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/CounterCell.java
@@ -21,8 +21,10 @@ package org.apache.beam.runners.core.metrics;
 import java.util.concurrent.atomic.AtomicLong;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.annotations.Internal;
 import org.apache.beam.sdk.metrics.Counter;
 import org.apache.beam.sdk.metrics.MetricName;
+import org.apache.beam.sdk.metrics.MetricsContainer;
 
 /**
  * Tracks the current value (and delta) for a Counter metric for a specific 
context and bundle.
@@ -40,10 +42,12 @@ public class CounterCell implements Counter, 
MetricCell {
   private final MetricName name;
 
   /**
-   * Package-visibility because all {@link CounterCell CounterCells} should be 
created by
-   * {@link MetricsContainerImpl#getCounter(MetricName)}.
+   * Generally, runners should construct instances using the methods in
+   * {@link MetricsContainerImpl}, unless they need to define their own 
version of
+   * {@link MetricsContainer}. These constructors are *only* public so runners 
can instantiate.
*/
-  CounterCell(MetricName name) {
+  @Internal
+  public CounterCell(MetricName name) {
 this.name = name;
   }
 

http://git-wip-us.apache.org/repos/asf/beam/blob/39674ca8/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DirtyState.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DirtyState.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DirtyState.java
index 532fc2a..1976049 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DirtyState.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DirtyState.java
@@ -22,6 +22,7 @@ import java.io.Serializable;
 import java.util.concurrent.atomic.AtomicReference;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.annotations.Internal;
 
 /**
  * Atomically tracks the dirty-state of a metric.
@@ -42,7 +43,8 @@ import org.apache.beam.sdk.annotations.Experimental.Kind;
  * completed.
  */
 @Experimental(Kind.METRICS)
-class DirtyState implements Serializable {
+@Internal
+public class DirtyState implements Serializable {
   private enum State {
 /** Indicates that there have been changes to the MetricCell since last 
commit. */
 DIRTY,

http://git-wip-us.apache.org/repos/asf/beam/blob/39674ca8/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
--
diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
index 5a5099a..8713ec4 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/metrics/DistributionCell.java
+++ 

[jira] [Commented] (BEAM-2375) upgrade beam-version to 2.0.0

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040056#comment-16040056
 ] 

Kenneth Knowles commented on BEAM-2375:
---

I think during development, you don't need to use versions to avoid the impact 
of the changes in the master branch. Instead, merges from `master` to `DSL_SQL` 
are the natural place to handle those impacts. They should be very minor, for a 
DSL, now that we have a stable release.

In all cases, you need to inherit from the parent pom and have dependencies for 
the same version or you can have confusing breakage. You can do this easily by 
only setting the parent pom version and otherwise leaving all dependency 
versions to be managed by the root.

> upgrade beam-version to 2.0.0
> -
>
> Key: BEAM-2375
> URL: https://issues.apache.org/jira/browse/BEAM-2375
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: James Xu
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1476) Support MapState in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1476:
-

Assignee: Pei He  (was: Jingsong Lee)

> Support MapState in Flink runner
> 
>
> Key: BEAM-1476
> URL: https://issues.apache.org/jira/browse/BEAM-1476
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Pei He
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1476) Support MapState in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles resolved BEAM-1476.
---
   Resolution: Fixed
Fix Version/s: 2.1.0

> Support MapState in Flink runner
> 
>
> Key: BEAM-1476
> URL: https://issues.apache.org/jira/browse/BEAM-1476
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1476) Support MapState in Flink runner

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-1476:
-

Assignee: Jingsong Lee  (was: Pei He)

> Support MapState in Flink runner
> 
>
> Key: BEAM-1476
> URL: https://issues.apache.org/jira/browse/BEAM-1476
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1476) Support MapState in Flink runner

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040044#comment-16040044
 ] 

Jingsong Lee commented on BEAM-1476:


solved in https://github.com/apache/beam/pull/3289

> Support MapState in Flink runner
> 
>
> Key: BEAM-1476
> URL: https://issues.apache.org/jira/browse/BEAM-1476
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Pei He
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1476) Support MapState in Flink runner

[Jingsong Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee reassigned BEAM-1476:
--

Assignee: Pei He  (was: Jingsong Lee)

> Support MapState in Flink runner
> 
>
> Key: BEAM-1476
> URL: https://issues.apache.org/jira/browse/BEAM-1476
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Pei He
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2416) Add AkkaIO

[Manu Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039955#comment-16039955
 ] 

Manu Zhang commented on BEAM-2416:
--

[~jbonofre], do you mean an akka-stream connector ? Could you give more 
descriptions ?

> Add AkkaIO
> --
>
> Key: BEAM-2416
> URL: https://issues.apache.org/jira/browse/BEAM-2416
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4037

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #102

[Apache Jenkins Server]

See 


Changes:

[altay] Do not fail when gen_protos cannot be imported

[klk] Fix RawPTransform translation

[klk] Improve Splittable ParDo translation

[altay] Migrate Python tests to not depend on fix sharding for file output

[altay] soft-enable the use of streaming flag

[chamikara] [BEAM-2405] Write to BQ using the streaming API

[altay] Whitelist find for tox environments

[davor] Generate a random table name. Assume Spanner database exists.

[altay] Fix imports in sdk_worker.

[altay] Fix the staging directory path in copying from GCS

--
[...truncated 2.53 MB...]
2017-06-07T00:35:37.683 [INFO] 
2017-06-07T00:35:37.683 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-07T00:35:37.688 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-07T00:35:37.698 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 144.7 KB/sec)
2017-06-07T00:35:37.701 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-07T00:35:37.714 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 865.8 KB/sec)
2017-06-07T00:35:37.718 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-07T00:35:37.726 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1604.2 KB/sec)
2017-06-07T00:35:37.730 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-07T00:35:37.740 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 217.3 KB/sec)
2017-06-07T00:35:37.744 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-07T00:35:37.754 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 157.3 KB/sec)
2017-06-07T00:35:37.758 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-07T00:35:37.767 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 583.3 KB/sec)
2017-06-07T00:35:37.770 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-07T00:35:37.778 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 311.9 KB/sec)
2017-06-07T00:35:37.782 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-07T00:35:37.792 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 342.7 KB/sec)
2017-06-07T00:35:37.797 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-07T00:35:37.805 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 125.4 KB/sec)
2017-06-07T00:35:37.810 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-07T00:35:37.819 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 188.5 KB/sec)
2017-06-07T00:35:37.823 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-07T00:35:37.842 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 92.8 KB/sec)
2017-06-07T00:35:37.846 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-07T00:35:37.856 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 265.4 KB/sec)
2017-06-07T00:35:37.880 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-07T00:35:37.892 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1468.0 KB/sec)
2017-06-07T00:35:37.896 [INFO] Downloading: 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #2424

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » JDK 1.7 (latest),beam #53

[Apache Jenkins Server]

See 


Changes:

[altay] Do not fail when gen_protos cannot be imported

[klk] Fix RawPTransform translation

[klk] Improve Splittable ParDo translation

[altay] Migrate Python tests to not depend on fix sharding for file output

[altay] soft-enable the use of streaming flag

[chamikara] [BEAM-2405] Write to BQ using the streaming API

[altay] Whitelist find for tox environments

[davor] Generate a random table name. Assume Spanner database exists.

[altay] Fix imports in sdk_worker.

[altay] Fix the staging directory path in copying from GCS

--
[...truncated 1.23 MB...]
2017-06-07\T\00:34:02.741 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-07\T\00:34:02.770 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 68.8 KB/sec)
2017-06-07\T\00:34:02.771 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-07\T\00:34:02.798 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 60.6 KB/sec)
2017-06-07\T\00:34:02.799 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-07\T\00:34:02.825 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 89.1 KB/sec)
2017-06-07\T\00:34:02.827 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-07\T\00:34:02.853 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 76.8 KB/sec)
2017-06-07\T\00:34:02.855 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-07\T\00:34:02.881 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-07\T\00:34:02.883 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-07\T\00:34:02.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 156.2 KB/sec)
2017-06-07\T\00:34:02.914 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-07\T\00:34:02.943 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 207.9 KB/sec)
2017-06-07\T\00:34:02.947 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-07\T\00:34:02.974 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 117.2 KB/sec)
2017-06-07\T\00:34:02.975 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-07\T\00:34:03.004 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 864.3 KB/sec)
2017-06-07\T\00:34:03.006 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-07\T\00:34:03.034 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
473.3 KB/sec)
2017-06-07\T\00:34:03.050 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-07\T\00:34:03.078 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 165.4 KB/sec)
2017-06-07\T\00:34:03.080 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-07\T\00:34:03.107 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 52.1 KB/sec)
2017-06-07\T\00:34:03.111 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
2017-06-07\T\00:34:03.153 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom
 (4 KB at 76.4 KB/sec)
2017-06-07\T\00:34:03.159 [INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #53

[Apache Jenkins Server]

See 


Changes:

[altay] Do not fail when gen_protos cannot be imported

[klk] Fix RawPTransform translation

[klk] Improve Splittable ParDo translation

[altay] Migrate Python tests to not depend on fix sharding for file output

[altay] soft-enable the use of streaming flag

[chamikara] [BEAM-2405] Write to BQ using the streaming API

[altay] Whitelist find for tox environments

[davor] Generate a random table name. Assume Spanner database exists.

[altay] Fix imports in sdk_worker.

[altay] Fix the staging directory path in copying from GCS

--
[...truncated 1.24 MB...]
2017-06-07\T\00:33:12.372 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
2017-06-07\T\00:33:12.401 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
 (2 KB at 64.5 KB/sec)
2017-06-07\T\00:33:12.402 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-07\T\00:33:12.533 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 15.4 KB/sec)
2017-06-07\T\00:33:12.534 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-07\T\00:33:12.564 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 54.5 KB/sec)
2017-06-07\T\00:33:12.566 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-07\T\00:33:12.593 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 85.8 KB/sec)
2017-06-07\T\00:33:12.595 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-07\T\00:33:12.622 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 73.9 KB/sec)
2017-06-07\T\00:33:12.623 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-07\T\00:33:12.649 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 76.4 KB/sec)
2017-06-07\T\00:33:12.651 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-07\T\00:33:12.678 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 156.2 KB/sec)
2017-06-07\T\00:33:12.682 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-07\T\00:33:12.708 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 231.9 KB/sec)
2017-06-07\T\00:33:12.714 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-07\T\00:33:12.743 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 109.1 KB/sec)
2017-06-07\T\00:33:12.745 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-07\T\00:33:12.773 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 895.2 KB/sec)
2017-06-07\T\00:33:12.774 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-07\T\00:33:12.801 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
508.3 KB/sec)
2017-06-07\T\00:33:12.815 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-07\T\00:33:12.874 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 78.5 KB/sec)
2017-06-07\T\00:33:12.876 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-07\T\00:33:12.910 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 41.4 KB/sec)
2017-06-07\T\00:33:12.914 [INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #53

[Apache Jenkins Server]

See 


Changes:

[altay] Do not fail when gen_protos cannot be imported

[klk] Fix RawPTransform translation

[klk] Improve Splittable ParDo translation

[altay] Migrate Python tests to not depend on fix sharding for file output

[altay] soft-enable the use of streaming flag

[chamikara] [BEAM-2405] Write to BQ using the streaming API

[altay] Whitelist find for tox environments

[davor] Generate a random table name. Assume Spanner database exists.

[altay] Fix imports in sdk_worker.

[altay] Fix the staging directory path in copying from GCS

--
[...truncated 1.24 MB...]
2017-06-07\T\00:29:51.779 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
2017-06-07\T\00:29:51.804 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
 (2 KB at 74.8 KB/sec)
2017-06-07\T\00:29:51.805 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-07\T\00:29:51.832 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 73.9 KB/sec)
2017-06-07\T\00:29:51.833 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-07\T\00:29:51.858 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 65.4 KB/sec)
2017-06-07\T\00:29:51.876 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-07\T\00:29:51.902 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 89.1 KB/sec)
2017-06-07\T\00:29:51.904 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-07\T\00:29:51.931 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 73.9 KB/sec)
2017-06-07\T\00:29:51.932 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-07\T\00:29:51.959 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 73.6 KB/sec)
2017-06-07\T\00:29:51.960 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-07\T\00:29:51.986 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 162.2 KB/sec)
2017-06-07\T\00:29:51.990 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-07\T\00:29:52.017 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 223.3 KB/sec)
2017-06-07\T\00:29:52.021 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-07\T\00:29:52.063 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 75.3 KB/sec)
2017-06-07\T\00:29:52.064 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-07\T\00:29:52.091 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 928.3 KB/sec)
2017-06-07\T\00:29:52.094 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-07\T\00:29:52.120 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
527.9 KB/sec)
2017-06-07\T\00:29:52.132 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-07\T\00:29:52.158 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 185.2 KB/sec)
2017-06-07\T\00:29:52.159 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-07\T\00:29:52.183 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 58.6 KB/sec)
2017-06-07\T\00:29:52.187 [INFO] Downloading: 

[GitHub] beam pull request #3300: Fix the staging directory path in copying from GCS

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3300


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Fix the staging directory path in copying from GCS

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 6fed1779d -> 1d2000d8c


Fix the staging directory path in copying from GCS


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/171a9930
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/171a9930
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/171a9930

Branch: refs/heads/master
Commit: 171a993044d97c42f027e1ec44436a3b8af7c32f
Parents: 6fed177
Author: Sourabh Bajaj 
Authored: Tue Jun 6 12:55:42 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 16:43:58 2017 -0700

--
 .../runners/dataflow/internal/dependency.py   |  7 ++-
 .../runners/dataflow/internal/dependency_test.py  | 14 +-
 2 files changed, 19 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/171a9930/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
index 3a0ff46..e656600 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency.py
@@ -181,7 +181,12 @@ def _stage_extra_packages(extra_packages, 
staging_location, temp_dir,
   staging_temp_dir = tempfile.mkdtemp(dir=temp_dir)
 logging.info('Downloading extra package: %s locally before staging',
  package)
-_dependency_file_copy(package, staging_temp_dir)
+if os.path.isfile(staging_temp_dir):
+  local_file_path = staging_temp_dir
+else:
+  _, last_component = FileSystems.split(package)
+  local_file_path = FileSystems.join(staging_temp_dir, last_component)
+_dependency_file_copy(package, local_file_path)
   else:
 raise RuntimeError(
 'The file %s cannot be found. It was specified in the '

http://git-wip-us.apache.org/repos/asf/beam/blob/171a9930/sdks/python/apache_beam/runners/dataflow/internal/dependency_test.py
--
diff --git 
a/sdks/python/apache_beam/runners/dataflow/internal/dependency_test.py 
b/sdks/python/apache_beam/runners/dataflow/internal/dependency_test.py
index 5eac7d6..e555b69 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/dependency_test.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/dependency_test.py
@@ -31,6 +31,16 @@ from apache_beam.options.pipeline_options import 
PipelineOptions
 from apache_beam.options.pipeline_options import SetupOptions
 
 
+# Protect against environments where GCS library is not available.
+# pylint: disable=wrong-import-order, wrong-import-position
+try:
+  from apitools.base.py.exceptions import HttpError
+except ImportError:
+  HttpError = None
+# pylint: enable=wrong-import-order, wrong-import-position
+
+
+@unittest.skipIf(HttpError is None, 'GCP dependencies are not installed')
 class SetupTest(unittest.TestCase):
 
   def update_options(self, options):
@@ -369,7 +379,9 @@ class SetupTest(unittest.TestCase):
   if from_path.startswith('gs://'):
 gcs_copied_files.append(from_path)
 _, from_name = os.path.split(from_path)
-self.create_temp_file(os.path.join(to_path, from_name), 'nothing')
+if os.path.isdir(to_path):
+  to_path = os.path.join(to_path, from_name)
+self.create_temp_file(to_path, 'nothing')
 logging.info('Fake copied GCS file: %s to %s', from_path, to_path)
   elif to_path.startswith('gs://'):
 logging.info('Faking file_copy(%s, %s)', from_path, to_path)

[2/2] beam git commit: This closes #3300

[altay]

This closes #3300


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1d2000d8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1d2000d8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1d2000d8

Branch: refs/heads/master
Commit: 1d2000d8c570ce2a2cdb63b5a208201526c394d8
Parents: 6fed177 171a993
Author: Ahmet Altay 
Authored: Tue Jun 6 16:44:03 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 16:44:03 2017 -0700

--
 .../runners/dataflow/internal/dependency.py   |  7 ++-
 .../runners/dataflow/internal/dependency_test.py  | 14 +-
 2 files changed, 19 insertions(+), 2 deletions(-)
--

Build failed in Jenkins: beam_PostCommit_Python_Verify #2423

[Apache Jenkins Server]

See 


--
Started by GitHub push by asfgit
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/beam.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:809)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1076)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1107)
at hudson.scm.SCM.checkout(SCM.java:496)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1281)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1728)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags 
--progress https://github.com/apache/beam.git 
+refs/heads/*:refs/remotes/origin/* 
+refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*" returned 
status code 128:
stdout: 
stderr: remote: GitHub is offline for maintenance. See http://status.github.com 
for more info.
fatal: unable to access 'https://github.com/apache/beam.git/': The requested 
URL returned error: 503

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1877)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1596)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:71)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:348)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:153)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:336)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to beam3(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1545)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:830)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor679.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy103.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:807)
... 11 more
ERROR: null
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
https://github.com/apache/beam.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:809)

[jira] [Updated] (BEAM-2418) Datastore IO does not work out of the box

[Colin Bookman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Bookman updated BEAM-2418:


Yep. It's a flat jar file of a compressed binary.




> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Vikas Kedigehalli
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Fix imports in sdk_worker.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master be26dd301 -> 6fed1779d


Fix imports in sdk_worker.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/efe8e1f4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/efe8e1f4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/efe8e1f4

Branch: refs/heads/master
Commit: efe8e1f41d78e06bef7ab5c9d72a7c65f553c5a3
Parents: be26dd3
Author: Robert Bradshaw 
Authored: Tue Jun 6 15:14:12 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 16:12:39 2017 -0700

--
 sdks/python/apache_beam/runners/worker/sdk_worker.py | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/efe8e1f4/sdks/python/apache_beam/runners/worker/sdk_worker.py
--
diff --git a/sdks/python/apache_beam/runners/worker/sdk_worker.py 
b/sdks/python/apache_beam/runners/worker/sdk_worker.py
index 596bb90..33c50ad 100644
--- a/sdks/python/apache_beam/runners/worker/sdk_worker.py
+++ b/sdks/python/apache_beam/runners/worker/sdk_worker.py
@@ -35,7 +35,8 @@ from google.protobuf import wrappers_pb2
 from apache_beam.coders import coder_impl
 from apache_beam.coders import WindowedValueCoder
 from apache_beam.internal import pickler
-from apache_beam.runners.dataflow.native_io import iobase
+from apache_beam.io import iobase
+from apache_beam.runners.dataflow.native_io import iobase as native_iobase
 from apache_beam.utils import counters
 from apache_beam.runners.api import beam_fn_api_pb2
 from apache_beam.runners.worker import operation_specs
@@ -126,7 +127,8 @@ class DataInputOperation(RunnerIOOperation):
 # custom sources without forcing intermediate materialization.  This seems very
 # related to the desire to inject key and window preserving [Splittable]DoFns
 # into the view computation.
-class SideInputSource(iobase.NativeSource, iobase.NativeSourceReader):
+class SideInputSource(native_iobase.NativeSource,
+  native_iobase.NativeSourceReader):
   """A 'source' for reading side inputs via state API calls.
   """
 

[GitHub] beam pull request #3308: Fix imports in sdk_worker.

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3308


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3308

[altay]

This closes #3308


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6fed1779
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6fed1779
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6fed1779

Branch: refs/heads/master
Commit: 6fed1779dbd78dddfe69bab46571f74101329504
Parents: be26dd3 efe8e1f
Author: Ahmet Altay 
Authored: Tue Jun 6 16:12:44 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 16:12:44 2017 -0700

--
 sdks/python/apache_beam/runners/worker/sdk_worker.py | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)
--

[jira] [Commented] (BEAM-2414) Add TwitterIO

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039843#comment-16039843
 ] 

Davor Bonaci commented on BEAM-2414:


May or may not be relevant -- there's elsewhere work on the Apache Streams 
connector, which would presumably include Twitter as well.

> Add TwitterIO
> -
>
> Key: BEAM-2414
> URL: https://issues.apache.org/jira/browse/BEAM-2414
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039838#comment-16039838
 ] 

Luke Cwik commented on BEAM-2418:
-

[~bookman_google] Does build/libs/*.jar contain a jar representing 
beam-sdks-java-extensions-protobuf?

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Vikas Kedigehalli
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2418) Datastore IO does not work out of the box

[Davor Bonaci (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Davor Bonaci reassigned BEAM-2418:
--

Assignee: Vikas Kedigehalli  (was: Davor Bonaci)

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Vikas Kedigehalli
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2394) Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039815#comment-16039815
 ] 

ASF GitHub Bot commented on BEAM-2394:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3298


> Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed
> --
>
> Key: BEAM-2394
> URL: https://issues.apache.org/jira/browse/BEAM-2394
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Mark Liu
>Assignee: Mairbek Khadikov
>
> SpannerWriteIT.testWrite failed in Postcommit_Java_JDK_Version_Test since 
> database didn't setup successfully.
> Error logs:
> {code}
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT
> 2017-05-31\T\12:21:30.032 [ERROR] 
> testWrite(org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT)  Time elapsed: 
> 283.011 s  <<< ERROR!
> java.lang.RuntimeException: 
> (b2cfd106d806288f): com.google.cloud.spanner.SpannerException: NOT_FOUND: 
> io.grpc.StatusRuntimeException: NOT_FOUND: Database not found: 
> projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
> resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
> resource_name: 
> "projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
> description: "Database does not exist."
>   at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
>   at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
>   at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
>   at 
> com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
>   at 
> com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
>   at 
> com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
>   at 
> com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
>   at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
> ...
> {code}
> Jenkins link:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%207%20(on%20Ubuntu%20only),label=beam/
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%208%20(on%20Ubuntu%20only),label=beam/
> Note: the root directory of JDK version test contains space, which is the 
> main difference with Postcommit_Java_MavenInstall. It can be like: 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/OpenJDK
>  7 (on Ubuntu only)/label/beam/..."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #3298

[davor]

This closes #3298


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/be26dd30
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/be26dd30
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/be26dd30

Branch: refs/heads/master
Commit: be26dd301a0da28451dbebe8d846e0265311d3fe
Parents: 6853d8e 3a9c00a
Author: Davor Bonaci 
Authored: Tue Jun 6 15:44:54 2017 -0700
Committer: Davor Bonaci 
Committed: Tue Jun 6 15:44:54 2017 -0700

--
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 32 ++--
 1 file changed, 23 insertions(+), 9 deletions(-)
--

[GitHub] beam pull request #3298: [BEAM-2394] Support concurrent execution of Spanner...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Generate a random table name. Assume Spanner database exists.

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 6853d8ef6 -> be26dd301


Generate a random table name.
Assume Spanner database exists.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3a9c00ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3a9c00ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3a9c00ac

Branch: refs/heads/master
Commit: 3a9c00ac7303823490d97f2f0adb5469700687ac
Parents: 6853d8e
Author: Mairbek Khadikov 
Authored: Mon Jun 5 12:29:02 2017 -0700
Committer: Davor Bonaci 
Committed: Tue Jun 6 15:44:44 2017 -0700

--
 .../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 32 ++--
 1 file changed, 23 insertions(+), 9 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/3a9c00ac/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
index 064c65e..8df224b 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/SpannerWriteIT.java
@@ -33,6 +33,7 @@ import com.google.cloud.spanner.SpannerOptions;
 import com.google.cloud.spanner.Statement;
 import com.google.spanner.admin.database.v1.CreateDatabaseMetadata;
 import java.util.Collections;
+
 import org.apache.beam.sdk.io.GenerateSequence;
 import org.apache.beam.sdk.options.Default;
 import org.apache.beam.sdk.options.Description;
@@ -52,6 +53,9 @@ import org.junit.runners.JUnit4;
 /** End-to-end test of Cloud Spanner Sink. */
 @RunWith(JUnit4.class)
 public class SpannerWriteIT {
+
+  private static final int MAX_DB_NAME_LENGTH = 30;
+
   @Rule public final transient TestPipeline p = TestPipeline.create();
 
   /** Pipeline options for this test. */
@@ -66,10 +70,10 @@ public class SpannerWriteIT {
 String getInstanceId();
 void setInstanceId(String value);
 
-@Description("Database ID to write to in Spanner")
+@Description("Database ID prefix to write to in Spanner")
 @Default.String("beam-testdb")
-String getDatabaseId();
-void setDatabaseId(String value);
+String getDatabaseIdPrefix();
+void setDatabaseIdPrefix(String value);
 
 @Description("Table name")
 @Default.String("users")
@@ -80,6 +84,7 @@ public class SpannerWriteIT {
   private Spanner spanner;
   private DatabaseAdminClient databaseAdminClient;
   private SpannerTestPipelineOptions options;
+  private String databaseName;
 
   @Before
   public void setUp() throws Exception {
@@ -88,15 +93,17 @@ public class SpannerWriteIT {
 
 spanner = 
SpannerOptions.newBuilder().setProjectId(options.getProjectId()).build().getService();
 
+databaseName = generateDatabaseName();
+
 databaseAdminClient = spanner.getDatabaseAdminClient();
 
 // Delete database if exists.
-databaseAdminClient.dropDatabase(options.getInstanceId(), 
options.getDatabaseId());
+databaseAdminClient.dropDatabase(options.getInstanceId(), databaseName);
 
 Operation op =
 databaseAdminClient.createDatabase(
 options.getInstanceId(),
-options.getDatabaseId(),
+databaseName,
 Collections.singleton(
 "CREATE TABLE "
 + options.getTable()
@@ -107,6 +114,13 @@ public class SpannerWriteIT {
 op.waitFor();
   }
 
+  private String generateDatabaseName() {
+String random = RandomStringUtils
+.randomAlphanumeric(MAX_DB_NAME_LENGTH - 1 - 
options.getDatabaseIdPrefix().length())
+.toLowerCase();
+return options.getDatabaseIdPrefix() + "-" + random;
+  }
+
   @Test
   public void testWrite() throws Exception {
 p.apply(GenerateSequence.from(0).to(100))
@@ -115,13 +129,13 @@ public class SpannerWriteIT {
 SpannerIO.write()
 .withProjectId(options.getProjectId())
 .withInstanceId(options.getInstanceId())
-.withDatabaseId(options.getDatabaseId()));
+.withDatabaseId(databaseName));
 
 p.run();
 DatabaseClient databaseClient =
 spanner.getDatabaseClient(
 DatabaseId.of(
-options.getProjectId(), options.getInstanceId(), 
options.getDatabaseId()));
+options.getProjectId(), options.getInstanceId(), 
databaseName));
 
 ResultSet resultSet =
 databaseClient
@@ -134,7 +148,7 @@ public class 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4032

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3308: Fix imports in sdk_worker.

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/3308

Fix imports in sdk_worker.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam sdk-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3308


commit b151914e0277088c04ceba02e18d649da19ed22e
Author: Robert Bradshaw 
Date:   2017-06-06T22:14:12Z

Fix imports in sdk_worker.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3307: Increase visibility of some Metrics methods

[bjchambers]

GitHub user bjchambers opened a pull request:

https://github.com/apache/beam/pull/3307

Increase visibility of some Metrics methods

Also revise the Javadoc on MetricsContainers.

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [*] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [*] Make sure tests pass via `mvn clean verify`.
 - [*] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [*] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bjchambers/beam visibility

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3307.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3307


commit 39674ca8d0ae1d82bfb5da6a81f26843613d3cd7
Author: bchambers 
Date:   2017-06-06T22:08:46Z

Increase visibility of some Metrics methods

Also revise the Javadoc on MetricsContainers.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4031

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Colin Bookman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039733#comment-16039733
 ] 

Colin Bookman commented on BEAM-2418:
-

Same issue. Tried with the following arguments.


java -jar build/libs/*.jar \
  --runner=DataflowRunner \
  --project=my-project \
  --stagingLocation=gs://my-project.appspot.com/staging/ \
  --tempLocation=gs://my-project.appspot.com/temp/


```
Jun 06, 2017 2:57:37 PM org.apache.beam.runners.dataflow.DataflowRunner 
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from 
the classpath: will stage 1 files. Enable logging at DEBUG level to see which 
files will be staged.
Exception in thread "main" java.lang.IllegalStateException: Unable to return a 
default Coder for 
IngestEntities/ParDo(GqlQueryTranslate)/ParMultiDo(GqlQueryTranslate).out0 
[PCollection]. Correct one of the following root causes:
  No Coder has been manually specified;  you may do so using .setCoder().
  Inferring a Coder from the CoderRegistry failed: Unable to provide a Coder 
for com.google.datastore.v1.Query.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
  Using the default output Coder from the producing PTransform failed: Unable 
to provide a Coder for com.google.datastore.v1.Query.
  Building a Coder using a registered CoderProvider failed.
  See suppressed exceptions for detailed failures.
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
at org.apache.beam.sdk.values.PCollection.getCoder(PCollection.java:250)
at 
org.apache.beam.sdk.values.PCollection.finishSpecifying(PCollection.java:104)
at 
org.apache.beam.sdk.runners.TransformHierarchy.finishSpecifyingInput(TransformHierarchy.java:147)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:481)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:422)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:277)
at 
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.expand(DatastoreV1.java:581)
at 
org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$Read.expand(DatastoreV1.java:226)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:482)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:441)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:179)
at 
com.google.cloud.dataflow.teleport.DatastoreToGcs.main(DatastoreToGcs.java:50)
at com.google.cloud.dataflow.teleport.Main.main(Main.java:50)
```

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039724#comment-16039724
 ] 

ASF GitHub Bot commented on BEAM-2405:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3306

[BEAM-2405] Override to sink interface in batch dataflow

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @chamikaramj  PTAL

We can deprecate the sink interface to not be user facing now as this is a 
unified model.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-BQ-SINK-MERGER

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3306.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3306


commit 4c23cde32528b93cd8da4b8c9b83b8970075c01c
Author: Sourabh Bajaj 
Date:   2017-06-06T21:51:42Z

[BEAM-2405] Override to sink interface in the batch dataflow worker




> Create a BigQuery sink for streaming using PTransform
> -
>
> Key: BEAM-2405
> URL: https://issues.apache.org/jira/browse/BEAM-2405
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3306: [BEAM-2405] Override to sink interface in batch dat...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3306

[BEAM-2405] Override to sink interface in batch dataflow

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @chamikaramj  PTAL

We can deprecate the sink interface to not be user facing now as this is a 
unified model.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-BQ-SINK-MERGER

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3306.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3306


commit 4c23cde32528b93cd8da4b8c9b83b8970075c01c
Author: Sourabh Bajaj 
Date:   2017-06-06T21:51:42Z

[BEAM-2405] Override to sink interface in the batch dataflow worker




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4029

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3304: Whitelist find for tox environments

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3304


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3304

[altay]

This closes #3304


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6853d8ef
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6853d8ef
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6853d8ef

Branch: refs/heads/master
Commit: 6853d8ef6532f0897ad075bda2e47c38a7aa6214
Parents: fdfd775 dcfb31f
Author: Ahmet Altay 
Authored: Tue Jun 6 14:13:40 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 14:13:40 2017 -0700

--
 sdks/python/tox.ini | 2 ++
 1 file changed, 2 insertions(+)
--

[1/2] beam git commit: Whitelist find for tox environments

[altay]

Repository: beam
Updated Branches:
  refs/heads/master fdfd77510 -> 6853d8ef6


Whitelist find for tox environments


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dcfb31ff
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dcfb31ff
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dcfb31ff

Branch: refs/heads/master
Commit: dcfb31ffe46a037e69967c8f6c054562d4d9b3b9
Parents: fdfd775
Author: Sourabh Bajaj 
Authored: Tue Jun 6 11:58:53 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 14:13:36 2017 -0700

--
 sdks/python/tox.ini | 2 ++
 1 file changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dcfb31ff/sdks/python/tox.ini
--
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 2166f6a..eff91fe 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -29,6 +29,7 @@ select = E3
 deps =
   nose==1.3.7
   grpcio-tools==1.3.5
+whitelist_externals=find
 commands =
   python --version
   # Clean up all previous python generated files.
@@ -73,6 +74,7 @@ passenv = TRAVIS*
 # autocomplete_test depends on nose when invoked directly.
 deps =
   nose==1.3.7
+whitelist_externals=find
 commands =
   pip install -e .[test,gcp]
   python --version

[jira] [Commented] (BEAM-2405) Create a BigQuery sink for streaming using PTransform

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039639#comment-16039639
 ] 

ASF GitHub Bot commented on BEAM-2405:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3288


> Create a BigQuery sink for streaming using PTransform
> -
>
> Key: BEAM-2405
> URL: https://issues.apache.org/jira/browse/BEAM-2405
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3288: [BEAM-2405] Write to BQ using the streaming API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3288


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-2405] Write to BQ using the streaming API

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master fa3922b57 -> fdfd77510


[BEAM-2405] Write to BQ using the streaming API


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1498684d
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1498684d
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1498684d

Branch: refs/heads/master
Commit: 1498684dfea31594a236edd7fde5d299e4b0aa1e
Parents: fa3922b
Author: Sourabh Bajaj 
Authored: Fri Jun 2 20:32:48 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Jun 6 14:04:33 2017 -0700

--
 sdks/python/apache_beam/io/gcp/bigquery.py  | 177 +++
 sdks/python/apache_beam/io/gcp/bigquery_test.py | 172 ++
 2 files changed, 349 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1498684d/sdks/python/apache_beam/io/gcp/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py 
b/sdks/python/apache_beam/io/gcp/bigquery.py
index 201c798..9069f73 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -115,6 +115,9 @@ from apache_beam.internal.gcp import auth
 from apache_beam.internal.gcp.json_value import from_json_value
 from apache_beam.internal.gcp.json_value import to_json_value
 from apache_beam.runners.dataflow.native_io import iobase as dataflow_io
+from apache_beam.transforms import DoFn
+from apache_beam.transforms import ParDo
+from apache_beam.transforms import PTransform
 from apache_beam.transforms.display import DisplayDataItem
 from apache_beam.utils import retry
 from apache_beam.options.pipeline_options import GoogleCloudOptions
@@ -134,6 +137,7 @@ __all__ = [
 'BigQueryDisposition',
 'BigQuerySource',
 'BigQuerySink',
+'WriteToBigQuery',
 ]
 
 JSON_COMPLIANCE_ERROR = 'NAN, INF and -INF values are not JSON compliant.'
@@ -813,6 +817,7 @@ class BigQueryWrapper(object):
 request = bigquery.BigqueryTablesInsertRequest(
 projectId=project_id, datasetId=dataset_id, table=table)
 response = self.client.tables.Insert(request)
+logging.debug("Created the table with id %s", table_id)
 # The response is a bigquery.Table instance.
 return response
 
@@ -1134,3 +1139,175 @@ class BigQueryWrapper(object):
   else:
 result[field.name] = self._convert_cell_value_to_dict(value, field)
 return result
+
+
+class BigQueryWriteFn(DoFn):
+  """A ``DoFn`` that streams writes to BigQuery once the table is created.
+  """
+
+  def __init__(self, table_id, dataset_id, project_id, batch_size, schema,
+   create_disposition, write_disposition, client):
+"""Initialize a WriteToBigQuery transform.
+
+Args:
+  table_id: The ID of the table. The ID must contain only letters
+(a-z, A-Z), numbers (0-9), or underscores (_). If dataset argument is
+None then the table argument must contain the entire table reference
+specified as: 'DATASET.TABLE' or 'PROJECT:DATASET.TABLE'.
+  dataset_id: The ID of the dataset containing this table or null if the
+table reference is specified entirely by the table argument.
+  project_id: The ID of the project containing this table or null if the
+table reference is specified entirely by the table argument.
+  batch_size: Number of rows to be written to BQ per streaming API insert.
+  schema: The schema to be used if the BigQuery table to write has to be
+created. This can be either specified as a 'bigquery.TableSchema' 
object
+or a single string  of the form 
'field1:type1,field2:type2,field3:type3'
+that defines a comma separated list of fields. Here 'type' should
+specify the BigQuery type of the field. Single string based schemas do
+not support nested fields, repeated fields, or specifying a BigQuery
+mode for fields (mode will always be set to 'NULLABLE').
+  create_disposition: A string describing what happens if the table does 
not
+exist. Possible values are:
+- BigQueryDisposition.CREATE_IF_NEEDED: create if does not exist.
+- BigQueryDisposition.CREATE_NEVER: fail the write if does not exist.
+  write_disposition: A string describing what happens if the table has
+already some data. Possible values are:
+-  BigQueryDisposition.WRITE_TRUNCATE: delete existing rows.
+-  BigQueryDisposition.WRITE_APPEND: add to existing rows.
+-  BigQueryDisposition.WRITE_EMPTY: fail the write if table not empty.
+For streaming pipelines WriteTruncate can not be used.
+  test_client: Override the default bigquery client used for testing.
+"""
+

[2/2] beam git commit: This closes #3288

[chamikara]

This closes #3288


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fdfd7751
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fdfd7751
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fdfd7751

Branch: refs/heads/master
Commit: fdfd775101f0e24f7cb3dce1894dafb10b39fb2b
Parents: fa3922b 1498684
Author: chamik...@google.com 
Authored: Tue Jun 6 14:05:03 2017 -0700
Committer: chamik...@google.com 
Committed: Tue Jun 6 14:05:03 2017 -0700

--
 sdks/python/apache_beam/io/gcp/bigquery.py  | 177 +++
 sdks/python/apache_beam/io/gcp/bigquery_test.py | 172 ++
 2 files changed, 349 insertions(+)
--

[jira] [Commented] (BEAM-1265) Add streaming support to Python DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039625#comment-16039625
 ] 

ASF GitHub Bot commented on BEAM-1265:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3305


> Add streaming support to Python DirectRunner
> 
>
> Key: BEAM-1265
> URL: https://issues.apache.org/jira/browse/BEAM-1265
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>
> Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3305: [BEAM-1265] soft-enable the use of streaming flag

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3305


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: soft-enable the use of streaming flag

[altay]

Repository: beam
Updated Branches:
  refs/heads/master e3139a38f -> fa3922b57


soft-enable the use of streaming flag


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/51139509
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/51139509
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/51139509

Branch: refs/heads/master
Commit: 51139509b65b5fa04a39c31f584a02f1a29170dc
Parents: e3139a3
Author: Ahmet Altay 
Authored: Tue Jun 6 13:34:09 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 13:56:57 2017 -0700

--
 sdks/python/apache_beam/options/pipeline_options.py  | 5 +++--
 .../apache_beam/options/pipeline_options_validator_test.py   | 8 
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py  | 2 +-
 .../apache_beam/runners/dataflow/internal/apiclient.py   | 2 +-
 4 files changed, 5 insertions(+), 12 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/51139509/sdks/python/apache_beam/options/pipeline_options.py
--
diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index 777926a..daef3a7 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -18,6 +18,7 @@
 """Pipeline options obtained from command line parsing."""
 
 import argparse
+import warnings
 
 from apache_beam.transforms.display import HasDisplayData
 from apache_beam.options.value_provider import StaticValueProvider
@@ -278,12 +279,12 @@ class StandardOptions(PipelineOptions):
 action='store_true',
 help='Whether to enable streaming mode.')
 
-  # TODO(BEAM-1265): Remove this error, once at least one runner supports
+  # TODO(BEAM-1265): Remove this warning, once at least one runner supports
   # streaming pipelines.
   def validate(self, validator):
 errors = []
 if self.view_as(StandardOptions).streaming:
-  errors.append('Streaming pipelines are not supported.')
+  warnings.warn('Streaming pipelines are not supported.')
 return errors
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/51139509/sdks/python/apache_beam/options/pipeline_options_validator_test.py
--
diff --git a/sdks/python/apache_beam/options/pipeline_options_validator_test.py 
b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
index 28fcbe3..97834cc 100644
--- a/sdks/python/apache_beam/options/pipeline_options_validator_test.py
+++ b/sdks/python/apache_beam/options/pipeline_options_validator_test.py
@@ -300,14 +300,6 @@ class SetupTest(unittest.TestCase):
 errors = validator.validate()
 self.assertFalse(errors)
 
-  def test_streaming(self):
-pipeline_options = PipelineOptions(['--streaming'])
-runner = MockRunners.TestDataflowRunner()
-validator = PipelineOptionsValidator(pipeline_options, runner)
-errors = validator.validate()
-
-self.assertIn('Streaming pipelines are not supported.', errors)
-
   def test_test_matcher(self):
 def get_validator(matcher):
   options = ['--project=example:example',

http://git-wip-us.apache.org/repos/asf/beam/blob/51139509/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py 
b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
index 3e0e268..62cea33 100644
--- a/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
+++ b/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
@@ -64,7 +64,7 @@ class DataflowRunner(PipelineRunner):
   # a job submission and is used by the service to establish what features
   # are expected by the workers.
   BATCH_ENVIRONMENT_MAJOR_VERSION = '6'
-  STREAMING_ENVIRONMENT_MAJOR_VERSION = '0'
+  STREAMING_ENVIRONMENT_MAJOR_VERSION = '1'
 
   def __init__(self, cache=None):
 # Cache of CloudWorkflowStep protos generated while the runner

http://git-wip-us.apache.org/repos/asf/beam/blob/51139509/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
--
diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py 
b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
index bfdd5e4..df1a3f2 100644
--- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
+++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
@@ -145,7 +145,7 @@ class Environment(object):
 # Version information.
 self.proto.version = 

[2/2] beam git commit: This closes #3305

[altay]

This closes #3305


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fa3922b5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fa3922b5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fa3922b5

Branch: refs/heads/master
Commit: fa3922b57b86beb163efb892d1f3c699402d684a
Parents: e3139a3 5113950
Author: Ahmet Altay 
Authored: Tue Jun 6 13:57:01 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 13:57:01 2017 -0700

--
 sdks/python/apache_beam/options/pipeline_options.py  | 5 +++--
 .../apache_beam/options/pipeline_options_validator_test.py   | 8 
 sdks/python/apache_beam/runners/dataflow/dataflow_runner.py  | 2 +-
 .../apache_beam/runners/dataflow/internal/apiclient.py   | 2 +-
 4 files changed, 5 insertions(+), 12 deletions(-)
--

[2/2] beam git commit: This closes #3299

[altay]

This closes #3299


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/e3139a38
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/e3139a38
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/e3139a38

Branch: refs/heads/master
Commit: e3139a38ff533c21fbc70f85eafcb1d68b52a4b0
Parents: 513c952 b5c257d
Author: Ahmet Altay 
Authored: Tue Jun 6 13:55:25 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 13:55:25 2017 -0700

--
 .../complete/juliaset/juliaset/juliaset_test.py |  5 +++--
 .../apache_beam/examples/complete/tfidf_test.py |  5 +++--
 .../examples/cookbook/group_with_coder_test.py  |  5 +++--
 .../examples/cookbook/mergecontacts_test.py |  3 ++-
 .../examples/cookbook/multiple_output_pardo_test.py | 11 ++-
 .../examples/wordcount_debugging_test.py|  3 ++-
 .../apache_beam/examples/wordcount_minimal_test.py  |  3 ++-
 sdks/python/apache_beam/examples/wordcount_test.py  |  3 ++-
 sdks/python/apache_beam/testing/util.py | 16 
 9 files changed, 39 insertions(+), 15 deletions(-)
--

[jira] [Commented] (BEAM-1265) Add streaming support to Python DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039590#comment-16039590
 ] 

ASF GitHub Bot commented on BEAM-1265:
--

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3305

[BEAM-1265] soft-enable the use of streaming flag

R: @tvalentyn 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam stflag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3305.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3305


commit a5b6d3fdd4a93350f97a257aaa8a96dec9319d7c
Author: Ahmet Altay 
Date:   2017-06-06T20:34:09Z

soft-enable the use of streaming flag




> Add streaming support to Python DirectRunner
> 
>
> Key: BEAM-1265
> URL: https://issues.apache.org/jira/browse/BEAM-1265
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Charles Chen
>
> Continue the work started in https://issues.apache.org/jira/browse/BEAM-428



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3305: [BEAM-1265] soft-enable the use of streaming flag

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/3305

[BEAM-1265] soft-enable the use of streaming flag

R: @tvalentyn 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam stflag

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3305.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3305


commit a5b6d3fdd4a93350f97a257aaa8a96dec9319d7c
Author: Ahmet Altay 
Date:   2017-06-06T20:34:09Z

soft-enable the use of streaming flag




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3282: [BEAM-3271] Improve Splittable ParDo translation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3282


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam git commit: This closes #3282: [BEAM-3271] Improve Splittable ParDo translation

[kenn]

This closes #3282: [BEAM-3271] Improve Splittable ParDo translation

  Improve Splittable ParDo translation
  Fix RawPTransform translation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/513c952f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/513c952f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/513c952f

Branch: refs/heads/master
Commit: 513c952fa65dbddee167160f6efabb634133425d
Parents: 7c608c3 1b00d95
Author: Kenneth Knowles 
Authored: Tue Jun 6 13:13:40 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue Jun 6 13:13:40 2017 -0700

--
 .../construction/PTransformTranslation.java | 57 
 .../core/construction/ParDoTranslation.java | 20 +++
 .../core/construction/SplittableParDo.java  | 18 +--
 .../core/construction/ParDoTranslationTest.java | 35 +++-
 .../core/SplittableParDoViaKeyedWorkItems.java  | 10 +++-
 runners/direct-java/pom.xml |  5 --
 .../beam/runners/direct/DirectGroupByKey.java   |  5 +-
 .../direct/KeyedPValueTrackingVisitor.java  |  2 +-
 .../direct/ParDoMultiOverrideFactory.java   |  3 +-
 .../direct/TestStreamEvaluatorFactory.java  |  3 +-
 .../direct/TransformEvaluatorRegistry.java  |  8 +--
 .../runners/direct/ViewOverrideFactory.java |  3 +-
 .../src/main/proto/beam_runner_api.proto|  3 ++
 13 files changed, 138 insertions(+), 34 deletions(-)
--

[1/3] beam git commit: Fix RawPTransform translation

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 7c608c32a -> 513c952fa


Fix RawPTransform translation


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/840492d9
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/840492d9
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/840492d9

Branch: refs/heads/master
Commit: 840492d9d8fb3b08cfe70a525655759fc1a31fdf
Parents: 7c608c3
Author: Kenneth Knowles 
Authored: Fri May 26 14:18:03 2017 -0700
Committer: Kenneth Knowles 
Committed: Tue Jun 6 13:10:33 2017 -0700

--
 .../construction/PTransformTranslation.java | 57 
 runners/direct-java/pom.xml |  5 --
 .../beam/runners/direct/DirectGroupByKey.java   |  5 +-
 .../direct/ParDoMultiOverrideFactory.java   |  3 +-
 .../direct/TestStreamEvaluatorFactory.java  |  3 +-
 .../direct/TransformEvaluatorRegistry.java  |  8 +--
 .../runners/direct/ViewOverrideFactory.java |  3 +-
 7 files changed, 56 insertions(+), 28 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/840492d9/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
index 7c5c593..32ecf43 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java
@@ -22,7 +22,6 @@ import static 
com.google.common.base.Preconditions.checkArgument;
 
 import com.google.common.collect.ImmutableMap;
 import com.google.protobuf.Any;
-import com.google.protobuf.Message;
 import java.io.IOException;
 import java.util.Collections;
 import java.util.List;
@@ -115,7 +114,20 @@ public class PTransformTranslation {
 // TODO: Display Data
 
 PTransform transform = appliedPTransform.getTransform();
-if (KNOWN_PAYLOAD_TRANSLATORS.containsKey(transform.getClass())) {
+// A RawPTransform directly vends its payload. Because it will generally be
+// a subclass, we cannot do dictionary lookup in KNOWN_PAYLOAD_TRANSLATORS.
+if (transform instanceof RawPTransform) {
+  RawPTransform rawPTransform = (RawPTransform) transform;
+
+  if (rawPTransform.getUrn() != null) {
+FunctionSpec.Builder payload = 
FunctionSpec.newBuilder().setUrn(rawPTransform.getUrn());
+@Nullable Any parameter = rawPTransform.getPayload();
+if (parameter != null) {
+  payload.setParameter(parameter);
+}
+transformBuilder.setSpec(payload);
+  }
+} else if (KNOWN_PAYLOAD_TRANSLATORS.containsKey(transform.getClass())) {
   FunctionSpec payload =
   KNOWN_PAYLOAD_TRANSLATORS
   .get(transform.getClass())
@@ -145,6 +157,25 @@ public class PTransformTranslation {
   }
 
   /**
+   * Returns the URN for the transform if it is known, otherwise {@code null}.
+   */
+  @Nullable
+  public static String urnForTransformOrNull(PTransform transform) {
+
+// A RawPTransform directly vends its URN. Because it will generally be
+// a subclass, we cannot do dictionary lookup in KNOWN_PAYLOAD_TRANSLATORS.
+if (transform instanceof RawPTransform) {
+  return ((RawPTransform) transform).getUrn();
+}
+
+TransformPayloadTranslator translator = 
KNOWN_PAYLOAD_TRANSLATORS.get(transform.getClass());
+if (translator == null) {
+  return null;
+}
+return translator.getUrn(transform);
+  }
+
+  /**
* Returns the URN for the transform if it is known, otherwise throws.
*/
   public static String urnForTransform(PTransform transform) {
@@ -176,13 +207,14 @@ public class PTransformTranslation {
* fully expanded in the pipeline proto.
*/
   public abstract static class RawPTransform<
-  InputT extends PInput, OutputT extends POutput, PayloadT extends 
Message>
+  InputT extends PInput, OutputT extends POutput>
   extends PTransform {
 
+@Nullable
 public abstract String getUrn();
 
 @Nullable
-PayloadT getPayload() {
+public Any getPayload() {
   return null;
 }
   }
@@ -190,24 +222,29 @@ public class PTransformTranslation {
   /**
* A translator that uses the explicit URN and payload from a {@link 
RawPTransform}.
*/
-  public static class RawPTransformTranslator
-  implements TransformPayloadTranslator {
+  public static class 

[jira] [Created] (BEAM-2419) Refactor dependency.py as the types are inconsistently passed in the filecopy code

[Sourabh Bajaj (JIRA)]

Sourabh Bajaj created BEAM-2419:
---

 Summary: Refactor dependency.py as the types are inconsistently 
passed in the filecopy code
 Key: BEAM-2419
 URL: https://issues.apache.org/jira/browse/BEAM-2419
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Sourabh Bajaj






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #3279

[altay]

This closes #3279


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7c608c32
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7c608c32
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7c608c32

Branch: refs/heads/master
Commit: 7c608c32a871124e2dcb8533fee2d354229283e9
Parents: a054550 ae3dc5f
Author: Ahmet Altay 
Authored: Tue Jun 6 12:58:31 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 12:58:31 2017 -0700

--
 sdks/python/setup.py | 20 
 1 file changed, 12 insertions(+), 8 deletions(-)
--

[GitHub] beam pull request #3279: Do not fail when gen_protos cannot be imported

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3279


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Do not fail when gen_protos cannot be imported

[altay]

Repository: beam
Updated Branches:
  refs/heads/master a05455088 -> 7c608c32a


Do not fail when gen_protos cannot be imported


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ae3dc5f3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ae3dc5f3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ae3dc5f3

Branch: refs/heads/master
Commit: ae3dc5f313ad55f3f86805b9f220bd1cdf1c902b
Parents: a054550
Author: Ahmet Altay 
Authored: Thu Jun 1 11:29:42 2017 -0700
Committer: Ahmet Altay 
Committed: Tue Jun 6 12:58:26 2017 -0700

--
 sdks/python/setup.py | 20 
 1 file changed, 12 insertions(+), 8 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ae3dc5f3/sdks/python/setup.py
--
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 596c8c5..051043b 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -125,14 +125,18 @@ GCP_REQUIREMENTS = [
 
 # We must generate protos after setup_requires are installed.
 def generate_protos_first(original_cmd):
-  # See https://issues.apache.org/jira/browse/BEAM-2366
-  # pylint: disable=wrong-import-position
-  import gen_protos
-  class cmd(original_cmd, object):
-def run(self):
-  gen_protos.generate_proto_files()
-  super(cmd, self).run()
-  return cmd
+  try:
+# See https://issues.apache.org/jira/browse/BEAM-2366
+# pylint: disable=wrong-import-position
+import gen_protos
+class cmd(original_cmd, object):
+  def run(self):
+gen_protos.generate_proto_files()
+super(cmd, self).run()
+return cmd
+  except ImportError:
+warnings.warn("Could not import gen_protos, skipping proto generation.")
+return original_cmd
 
 
 setuptools.setup(

[jira] [Commented] (BEAM-2415) Add FacebookIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039492#comment-16039492
 ] 

Jean-Baptiste Onofré commented on BEAM-2415:


I started an IO leveraging {{org.facebook4j}} which can {{read}} posts and 
provides a {{PCollection}} or create posts ({{write}}) from a {{PCollection}}.

> Add FacebookIO
> --
>
> Key: BEAM-2415
> URL: https://issues.apache.org/jira/browse/BEAM-2415
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2414) Add TwitterIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039487#comment-16039487
 ] 

Jean-Baptiste Onofré commented on BEAM-2414:


I started a IO using {{org.twitter4j}} that can search and retrieves tweets as 
{{PCollection}} ({{read}}) and create tweets from a {{PCollection}} ({{write}}).

> Add TwitterIO
> -
>
> Key: BEAM-2414
> URL: https://issues.apache.org/jira/browse/BEAM-2414
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4027

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Vikas Kedigehalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039467#comment-16039467
 ] 

Vikas Kedigehalli commented on BEAM-2418:
-

[~bookman_google] could you try running it without templates (by passing query 
and other options via command line arguments) and see if it works? 

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3304: Whitelist find for tox environments

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/3304

Whitelist find for tox environments

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam fix-to

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3304.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3304


commit a5d9f6ba4b29ae2b1df358193cdf3e182496914f
Author: Sourabh Bajaj 
Date:   2017-06-06T18:58:53Z

Whitelist find for tox environments




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2414) Add TwitterIO

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039463#comment-16039463
 ] 

Eugene Kirpichov commented on BEAM-2414:


Please clarify what you mean by this.

> Add TwitterIO
> -
>
> Key: BEAM-2414
> URL: https://issues.apache.org/jira/browse/BEAM-2414
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2415) Add FacebookIO

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039460#comment-16039460
 ] 

Eugene Kirpichov commented on BEAM-2415:


Please clarify what you mean by this?

> Add FacebookIO
> --
>
> Key: BEAM-2415
> URL: https://issues.apache.org/jira/browse/BEAM-2415
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall_Windows #101

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2276] Cleanups on the windowed DefaultFilenamePolicy

[aljoscha.krettek] [BEAM-1498] Use Flink-native side outputs

[aljoscha.krettek] Remove the FnOutputT parameter from DoFnOperator

[pei] Flink runner: support MapState in FlinkStateInternals.

--
[...truncated 2.53 MB...]
2017-06-06T18:31:24.222 [INFO] 
2017-06-06T18:31:24.222 [INFO] --- groovy-maven-plugin:2.0:execute 
(find-supported-python-for-clean) @ beam-sdks-python ---
2017-06-06T18:31:24.226 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
2017-06-06T18:31:24.235 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-slf4j/1.8/gossip-slf4j-1.8.pom
 (2 KB at 160.8 KB/sec)
2017-06-06T18:31:24.238 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
2017-06-06T18:31:24.248 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip/1.8/gossip-1.8.pom
 (12 KB at 1125.6 KB/sec)
2017-06-06T18:31:24.251 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
2017-06-06T18:31:24.260 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/forge/forge-parent/9/forge-parent-9.pom
 (13 KB at 1426.0 KB/sec)
2017-06-06T18:31:24.265 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
2017-06-06T18:31:24.273 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-core/1.8/gossip-core-1.8.pom
 (3 KB at 271.6 KB/sec)
2017-06-06T18:31:24.278 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
2017-06-06T18:31:24.284 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/sonatype/gossip/gossip-bootstrap/1.8/gossip-bootstrap-1.8.pom
 (2 KB at 262.2 KB/sec)
2017-06-06T18:31:24.288 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
2017-06-06T18:31:24.296 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava/14.0.1/guava-14.0.1.pom
 (6 KB at 656.2 KB/sec)
2017-06-06T18:31:24.299 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
2017-06-06T18:31:24.312 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/14.0.1/guava-parent-14.0.1.pom
 (3 KB at 191.9 KB/sec)
2017-06-06T18:31:24.316 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
2017-06-06T18:31:24.328 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-classworlds/2.4.2/plexus-classworlds-2.4.2.pom
 (4 KB at 285.6 KB/sec)
2017-06-06T18:31:24.333 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
2017-06-06T18:31:24.346 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-interpolation/1.16/plexus-interpolation-1.16.pom
 (2 KB at 77.1 KB/sec)
2017-06-06T18:31:24.350 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
2017-06-06T18:31:24.359 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.pom
 (2 KB at 188.5 KB/sec)
2017-06-06T18:31:24.365 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
2017-06-06T18:31:24.384 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-api/2.0/gmaven-adapter-api-2.0.pom
 (2 KB at 92.8 KB/sec)
2017-06-06T18:31:24.389 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
2017-06-06T18:31:24.398 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/gmaven/gmaven-adapter-impl/2.0/gmaven-adapter-impl-2.0.pom
 (3 KB at 294.9 KB/sec)
2017-06-06T18:31:24.403 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
2017-06-06T18:31:24.415 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/codehaus/groovy/groovy-all/2.1.5/groovy-all-2.1.5.pom
 (18 KB at 1468.0 KB/sec)
2017-06-06T18:31:24.419 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom
2017-06-06T18:31:24.431 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/ant/ant/1.8.4/ant-1.8.4.pom (10 
KB at 857.1 KB/sec)
2017-06-06T18:31:24.434 [INFO] Downloading: 

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 8 (on Ubuntu only),beam #52

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2276] Cleanups on the windowed DefaultFilenamePolicy

[aljoscha.krettek] [BEAM-1498] Use Flink-native side outputs

[aljoscha.krettek] Remove the FnOutputT parameter from DoFnOperator

[pei] Flink runner: support MapState in FlinkStateInternals.

--
[...truncated 1.23 MB...]
2017-06-06\T\18:31:35.288 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/data-artisans/flakka-slf4j_2.10/2.3-custom/flakka-slf4j_2.10-2.3-custom.pom
 (3 KB at 92.6 KB/sec)
2017-06-06\T\18:31:35.289 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
2017-06-06\T\18:31:35.318 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/clapper/grizzled-slf4j_2.10/1.0.2/grizzled-slf4j_2.10-1.0.2.pom
 (2 KB at 64.5 KB/sec)
2017-06-06\T\18:31:35.320 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
2017-06-06\T\18:31:35.348 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.3/scala-library-2.10.3.pom
 (2 KB at 71.3 KB/sec)
2017-06-06\T\18:31:35.350 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
2017-06-06\T\18:31:35.378 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/scopt/scopt_2.10/3.5.0/scopt_2.10-3.5.0.pom
 (2 KB at 58.4 KB/sec)
2017-06-06\T\18:31:35.379 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
2017-06-06\T\18:31:35.407 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill_2.10/0.7.4/chill_2.10-0.7.4.pom
 (3 KB at 82.8 KB/sec)
2017-06-06\T\18:31:35.409 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
2017-06-06\T\18:31:35.437 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.10.5/scala-library-2.10.5.pom
 (2 KB at 71.3 KB/sec)
2017-06-06\T\18:31:35.439 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
2017-06-06\T\18:31:35.468 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/twitter/chill-java/0.7.4/chill-java-0.7.4.pom
 (2 KB at 68.5 KB/sec)
2017-06-06\T\18:31:35.470 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
2017-06-06\T\18:31:35.507 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-optimizer_2.10/1.3.0/flink-optimizer_2.10-1.3.0.pom
 (5 KB at 114.0 KB/sec)
2017-06-06\T\18:31:35.513 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
2017-06-06\T\18:31:35.544 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-streaming-java_2.10/1.3.0/flink-streaming-java_2.10-1.3.0.pom
 (7 KB at 194.5 KB/sec)
2017-06-06\T\18:31:35.551 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
2017-06-06\T\18:31:35.642 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/org.apache.sling.commons.json/2.0.6/org.apache.sling.commons.json-2.0.6.pom
 (4 KB at 34.8 KB/sec)
2017-06-06\T\18:31:35.644 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom
2017-06-06\T\18:31:35.677 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/sling/sling/10/sling-10.pom (26 
KB at 759.6 KB/sec)
2017-06-06\T\18:31:35.680 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom
2017-06-06\T\18:31:35.708 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/8/apache-8.pom (14 KB at 
490.2 KB/sec)
2017-06-06\T\18:31:35.723 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
2017-06-06\T\18:31:35.764 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils_2.10/1.3.0/flink-test-utils_2.10-1.3.0.pom
 (5 KB at 112.9 KB/sec)
2017-06-06\T\18:31:35.765 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
2017-06-06\T\18:31:35.852 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-parent/1.3.0/flink-test-utils-parent-1.3.0.pom
 (2 KB at 16.2 KB/sec)
2017-06-06\T\18:31:35.856 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/flink/flink-test-utils-junit/1.3.0/flink-test-utils-junit-1.3.0.pom

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Colin Bookman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039392#comment-16039392
 ] 

Colin Bookman commented on BEAM-2418:
-

If it helps I'm trying to build this as a template. Here's my CLI 

java -jar build/libs/*.jar \
  --runner=DataflowRunner \
  --project=my-project \
  --stagingLocation=gs://my-project.appspot.com/staging/ \
  --tempLocation=gs://my-project.appspot.com/temp/ \
  --templateLocation=gs://my-project.appspot.com/templates/


> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_JDK_Versions_Test » OpenJDK 7 (on Ubuntu only),beam #52

[Apache Jenkins Server]

See 


Changes:

[jbonofre] [BEAM-2276] Cleanups on the windowed DefaultFilenamePolicy

[aljoscha.krettek] [BEAM-1498] Use Flink-native side outputs

[aljoscha.krettek] Remove the FnOutputT parameter from DoFnOperator

[pei] Flink runner: support MapState in FlinkStateInternals.

--
[...truncated 878.64 KB...]
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.api-client:google-api-client-java6:jar:1.22.0 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.api-client:google-api-client-jackson2:jar:1.22.0 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 from the shaded 
jar.
2017-06-06\T\18:16:22.942 [INFO] Including com.google.guava:guava:jar:20.0 in 
the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding org.apache.avro:avro:jar:1.8.2 from 
the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding org.tukaani:xz:jar:1.5 from the 
shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.errorprone:error_prone_annotations:jar:2.0.15 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.code.findbugs:jsr305:jar:3.0.1 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-core:jar:2.8.8 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.8 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.8 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding org.slf4j:slf4j-api:jar:1.7.14 from 
the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from the shaded jar.
2017-06-06\T\18:16:22.942 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from 
the shaded jar.
2017-06-06\T\18:16:27.271 [INFO] Replacing original artifact with shaded 
artifact.
2017-06-06\T\18:16:27.271 [INFO] Replacing 

 with 

2017-06-06\T\18:16:27.271 [INFO] Replacing original test artifact with shaded 
test artifact.
2017-06-06\T\18:16:27.271 [INFO] Replacing 

 with 

2017-06-06\T\18:16:27.271 [INFO] Dependency-reduced POM written at: 

2017-06-06\T\18:16:27.374 [INFO] 
2017-06-06\T\18:16:27.374 [INFO] --- 
maven-failsafe-plugin:2.20:integration-test (integration-test) @ 
beam-runners-google-cloud-dataflow-java ---
2017-06-06\T\18:16:27.447 [INFO] Failsafe report directory: 

2017-06-06\T\18:16:27.451 [INFO] parallel='all', perCoreThreadCount=true, 
threadCount=4, useUnlimitedThreads=false, threadCountSuites=0, 
threadCountClasses=0, threadCountMethods=0, parallelOptimized=true
2017-06-06\T\18:16:27.467 [INFO] 
2017-06-06\T\18:16:27.467 [INFO] 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #3051

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Vikas Kedigehalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039324#comment-16039324
 ] 

Vikas Kedigehalli commented on BEAM-2418:
-

Looks like we do include 'beam-sdks-java-extensions-protobuf" 
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L76,
 and we also have integration tests that pass 
(https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/datastore/V1ReadIT.java#L105)

Taking a look.

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2364) PCollectionView should not be a PValue, it should expand to its underlying PCollection

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039301#comment-16039301
 ] 

ASF GitHub Bot commented on BEAM-2364:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3303

[BEAM-2364] Expand all PValues to component PCollections always

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
direct_write_view_as_primitive_again

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3303.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3303


commit 812f83e676a7d67e06829784c396911935b233ff
Author: Thomas Groh 
Date:   2017-06-02T01:39:58Z

Expand all PValues to component PCollections always

Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.




> PCollectionView should not be a PValue, it should expand to its underlying 
> PCollection
> --
>
> Key: BEAM-2364
> URL: https://issues.apache.org/jira/browse/BEAM-2364
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Thomas Groh
>  Labels: backward-incompatible
>
> This bug has been present for a very long time.
> It is a change to {{@Internal}} details but is going to be 
> backwards-incompatible in a couple ways there, because backwards compatible 
> behavior is incorrect. But we need to be very careful with the surrounding 
> logic.
> The particular motivating need is that as long as a PCollectionView expands 
> to only itself, the outputs for surgery and serialization are not correct. 
> There may be a solution involving hardcoded logic, TBD.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #3303: [BEAM-2364] Expand all PValues to component PCollec...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3303

[BEAM-2364] Expand all PValues to component PCollections always

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`.
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---
Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam 
direct_write_view_as_primitive_again

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3303.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3303


commit 812f83e676a7d67e06829784c396911935b233ff
Author: Thomas Groh 
Date:   2017-06-02T01:39:58Z

Expand all PValues to component PCollections always

Update the implementation of WriteView

The PCollectionView is constructed within the composite override, but
WriteView just produces a primitive PCollection which has no consumers.
Track the ViewWriter within the Direct Runner, and utilize that
transform rather than the producer to update PCollection Watermarks.

Remove most Flink View overrides. All of the overrides are materially
identical within the flink runner, so use a single override to replace
all of them.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-2418) Datastore IO does not work out of the box

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2418:
--
Fix Version/s: 2.1.0

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2418) Datastore IO does not work out of the box

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-2418:
--
Priority: Blocker  (was: Major)

> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>Priority: Blocker
> Fix For: 2.1.0
>
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2418) Datastore IO does not work out of the box

[Colin Bookman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039282#comment-16039282
 ] 

Colin Bookman commented on BEAM-2418:
-

Tried adding `compile group: 'org.apache.beam', name: 
'beam-sdks-java-extensions-protobuf', version: '2.0.0'` to my build. Still did 
not solve issue.

Here's a gist that shows the stack trace, java code, and gradle build file: 
https://gist.github.com/cobookman/e4d2f2b89b4c3cadae9cd83892162758



> Datastore IO does not work out of the box
> -
>
> Key: BEAM-2418
> URL: https://issues.apache.org/jira/browse/BEAM-2418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions, sdk-java-gcp
>Affects Versions: 2.0.0
>Reporter: Stephen Sisk
>Assignee: Davor Bonaci
>
> We have user reports that DatastoreIO does not work when they try to use it.
> We believe this is a result of our effort to minimize our dependencies in the 
> core SDK (protobuf in this case). ProtoCoder is not registered by default, so 
> a user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
> their maven dependencies to get it. 
> We  need to confirm it, but if so, we will probably need to fix this in the 
> next release to have ProtoCoder when using DatastoreIO.
> cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1679) create gRPC IO

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk updated BEAM-1679:
---
Component/s: (was: sdk-java-core)
 sdk-java-extensions

> create gRPC IO
> --
>
> Key: BEAM-1679
> URL: https://issues.apache.org/jira/browse/BEAM-1679
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Borisa Zivkovic
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-2418) Datastore IO does not work out of the box

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-2418:
--

 Summary: Datastore IO does not work out of the box
 Key: BEAM-2418
 URL: https://issues.apache.org/jira/browse/BEAM-2418
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions, sdk-java-gcp
Affects Versions: 2.0.0
Reporter: Stephen Sisk
Assignee: Davor Bonaci


We have user reports that DatastoreIO does not work when they try to use it.

We believe this is a result of our effort to minimize our dependencies in the 
core SDK (protobuf in this case). ProtoCoder is not registered by default, so a 
user would need explicitly include 'beam-sdks-java-extensions-protobuf' in 
their maven dependencies to get it. 

We  need to confirm it, but if so, we will probably need to fix this in the 
next release to have ProtoCoder when using DatastoreIO.

cc [~vikasrk]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3050

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3289: Flink runner: support MapState in FlinkStateInterna...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3289


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3289

[pei]

This closes #3289


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a0545508
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a0545508
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a0545508

Branch: refs/heads/master
Commit: a05455088de338b46582a55df974865561dc70e7
Parents: aebd3a4 dbab052
Author: Pei He 
Authored: Tue Jun 6 23:19:25 2017 +0800
Committer: Pei He 
Committed: Tue Jun 6 23:19:25 2017 +0800

--
 runners/flink/pom.xml   |   1 -
 .../streaming/state/FlinkStateInternals.java| 205 ++-
 2 files changed, 202 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: Flink runner: support MapState in FlinkStateInternals.

[pei]

Repository: beam
Updated Branches:
  refs/heads/master aebd3a4c5 -> a05455088


Flink runner: support MapState in FlinkStateInternals.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/dbab052c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/dbab052c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/dbab052c

Branch: refs/heads/master
Commit: dbab052c4456ff51dd4ce44979c77a508acc17e9
Parents: aebd3a4
Author: 波特 
Authored: Thu May 18 12:23:20 2017 +0800
Committer: Pei He 
Committed: Tue Jun 6 23:18:33 2017 +0800

--
 runners/flink/pom.xml   |   1 -
 .../streaming/state/FlinkStateInternals.java| 205 ++-
 2 files changed, 202 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/dbab052c/runners/flink/pom.xml
--
diff --git a/runners/flink/pom.xml b/runners/flink/pom.xml
index 92f95a0..c4c6b55 100644
--- a/runners/flink/pom.xml
+++ b/runners/flink/pom.xml
@@ -92,7 +92,6 @@
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
 org.apache.beam.sdk.testing.LargeKeys$Above100MB,
 org.apache.beam.sdk.testing.UsesSetState,
-org.apache.beam.sdk.testing.UsesMapState,
 org.apache.beam.sdk.testing.UsesCommittedMetrics,
 org.apache.beam.sdk.testing.UsesTestStream,
 org.apache.beam.sdk.testing.UsesSplittableParDo

http://git-wip-us.apache.org/repos/asf/beam/blob/dbab052c/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
--
diff --git 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
index b73abe9..f0d3278 100644
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/FlinkStateInternals.java
@@ -26,6 +26,7 @@ import org.apache.beam.runners.core.StateInternals;
 import org.apache.beam.runners.core.StateNamespace;
 import org.apache.beam.runners.core.StateTag;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
+import org.apache.beam.runners.flink.translation.types.CoderTypeSerializer;
 import org.apache.beam.sdk.coders.Coder;
 import org.apache.beam.sdk.coders.CoderException;
 import org.apache.beam.sdk.coders.InstantCoder;
@@ -46,6 +47,7 @@ import 
org.apache.beam.sdk.transforms.windowing.TimestampCombiner;
 import org.apache.beam.sdk.util.CoderUtils;
 import org.apache.beam.sdk.util.CombineContextFactory;
 import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.state.MapStateDescriptor;
 import org.apache.flink.api.common.state.ValueStateDescriptor;
 import org.apache.flink.api.common.typeutils.base.StringSerializer;
 import org.apache.flink.runtime.state.KeyedStateBackend;
@@ -132,11 +134,11 @@ public class FlinkStateInternals implements 
StateInternals {
 
   @Override
   public  MapState bindMap(
-  StateTag> spec,
+  StateTag> address,
   Coder mapKeyCoder,
   Coder mapValueCoder) {
-throw new UnsupportedOperationException(
-String.format("%s is not supported", 
MapState.class.getSimpleName()));
+return new FlinkMapState<>(
+flinkStateBackend, address, namespace, mapKeyCoder, 
mapValueCoder);
   }
 
   @Override
@@ -1029,4 +1031,201 @@ public class FlinkStateInternals implements 
StateInternals {
   return result;
 }
   }
+
+  private static class FlinkMapState implements MapState {
+
+private final StateNamespace namespace;
+private final StateTag> address;
+private final MapStateDescriptor flinkStateDescriptor;
+private final KeyedStateBackend flinkStateBackend;
+
+FlinkMapState(
+KeyedStateBackend flinkStateBackend,
+StateTag> address,
+StateNamespace namespace,
+Coder mapKeyCoder, Coder mapValueCoder) {
+  this.namespace = namespace;
+  this.address = address;
+  this.flinkStateBackend = flinkStateBackend;
+  this.flinkStateDescriptor 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #4025

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1679) create gRPC IO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039006#comment-16039006
 ] 

Jean-Baptiste Onofré commented on BEAM-1679:


BEAM-2088

> create gRPC IO
> --
>
> Key: BEAM-1679
> URL: https://issues.apache.org/jira/browse/BEAM-1679
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Borisa Zivkovic
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2414) Add TwitterIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-2414:
--

Assignee: Jean-Baptiste Onofré  (was: Davor Bonaci)

> Add TwitterIO
> -
>
> Key: BEAM-2414
> URL: https://issues.apache.org/jira/browse/BEAM-2414
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-2415) Add FacebookIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-2415:
---
Summary: Add FacebookIO  (was: Add facebookIO)

> Add FacebookIO
> --
>
> Key: BEAM-2415
> URL: https://issues.apache.org/jira/browse/BEAM-2415
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-2415) Add facebookIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-2415:
--

Assignee: Jean-Baptiste Onofré

> Add facebookIO
> --
>
> Key: BEAM-2415
> URL: https://issues.apache.org/jira/browse/BEAM-2415
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2331) SQRT Function

[Tarush Grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarush Grover closed BEAM-2331.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> SQRT Function
> -
>
> Key: BEAM-2331
> URL: https://issues.apache.org/jira/browse/BEAM-2331
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-2330) MOD Function

[Tarush Grover (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16038957#comment-16038957
 ] 

Tarush Grover commented on BEAM-2330:
-

This is already being implemented in arithmetic functions.

> MOD Function
> 
>
> Key: BEAM-2330
> URL: https://issues.apache.org/jira/browse/BEAM-2330
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-2330) MOD Function

[Tarush Grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarush Grover resolved BEAM-2330.
-
   Resolution: Fixed
Fix Version/s: 2.0.0

> MOD Function
> 
>
> Key: BEAM-2330
> URL: https://issues.apache.org/jira/browse/BEAM-2330
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-2329) ABS Function

[Tarush Grover (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarush Grover closed BEAM-2329.
---
   Resolution: Fixed
Fix Version/s: 2.0.0

> ABS Function
> 
>
> Key: BEAM-2329
> URL: https://issues.apache.org/jira/browse/BEAM-2329
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Tarush Grover
>Assignee: Tarush Grover
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-1498) Use Flink-native side outputs

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed BEAM-1498.
--
   Resolution: Fixed
Fix Version/s: 2.1.0

> Use Flink-native side outputs
> -
>
> Key: BEAM-1498
> URL: https://issues.apache.org/jira/browse/BEAM-1498
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Jingsong Lee
> Fix For: 2.1.0
>
>
> Once Flink has support for side outputs we should use them instead of 
> manually dealing with the {{RawUnionValues}}.
> Side outputs for Flink is being tracked in 
> https://issues.apache.org/jira/browse/FLINK-4460.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)