date:20180731

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1137

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=129570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129570
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 01/Aug/18 04:04
Start Date: 01/Aug/18 04:04
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6110: 
[BEAM-4176] Tests for running Python on Flink.
URL: https://github.com/apache/beam/pull/6110#discussion_r206749979
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py
 ##
 @@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.runners.portability import portable_runner
+from apache_beam.runners.portability import portable_runner_test
+from apache_beam.testing.util import assert_that
+
+if __name__ == '__main__':
+  # Run as
+  #
+  # python -m apache_beam.runners.portability.flink_runner_test \
+  # /path/to/job_server.jar \
+  # [FlinkRunnerTest.test_method, ...]
+  flinkJobServerJar = sys.argv.pop(1)
+
+  # This is defined here to only be run when we invoke this file explicitly.
+  class FlinkRunnerTest(portable_runner_test.PortableRunnerTest):
+_use_grpc = True
+_use_subprocesses = True
+
+@classmethod
+def _subprocess_command(cls, port):
+  return [
+  'java',
+  '-jar', flinkJobServerJar,
+  '--artifacts-dir', '/tmp/flink',
+  '--job-host', 'localhost:%s' % port,
+  ]
+
+@classmethod
+def get_runner(cls):
+  return portable_runner.PortableRunner()
+
+def create_options(self):
+  options = super(FlinkRunnerTest, self).create_options()
 
 Review comment:
   We should support the `streaming` option here and cover the streaming mode.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129570)
Time Spent: 8h 40m  (was: 8.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=129569=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129569
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 01/Aug/18 04:01
Start Date: 01/Aug/18 04:01
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6110: 
[BEAM-4176] Tests for running Python on Flink.
URL: https://github.com/apache/beam/pull/6110#discussion_r206749666
 
 

 ##
 File path: sdks/python/apache_beam/runners/portability/flink_runner_test.py
 ##
 @@ -0,0 +1,76 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from __future__ import absolute_import
+from __future__ import print_function
+
+import logging
+import sys
+import unittest
+
+import apache_beam as beam
+from apache_beam.options.pipeline_options import DebugOptions
+from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.runners.portability import portable_runner
+from apache_beam.runners.portability import portable_runner_test
+from apache_beam.testing.util import assert_that
+
+if __name__ == '__main__':
+  # Run as
+  #
+  # python -m apache_beam.runners.portability.flink_runner_test \
+  # /path/to/job_server.jar \
+  # [FlinkRunnerTest.test_method, ...]
+  flinkJobServerJar = sys.argv.pop(1)
+
+  # This is defined here to only be run when we invoke this file explicitly.
+  class FlinkRunnerTest(portable_runner_test.PortableRunnerTest):
+_use_grpc = True
+_use_subprocesses = True
+
+@classmethod
+def _subprocess_command(cls, port):
+  return [
+  'java',
+  '-jar', flinkJobServerJar,
+  '--artifacts-dir', '/tmp/flink',
 
 Review comment:
   should probably use the system tmp dir here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129569)
Time Spent: 8.5h  (was: 8h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4936) Beam Dependency Update Request: org.codehaus.groovy

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4936?focusedWorklogId=129568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129568
 ]

ASF GitHub Bot logged work on BEAM-4936:


Author: ASF GitHub Bot
Created on: 01/Aug/18 04:00
Start Date: 01/Aug/18 04:00
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6115: DO NOT MERGE, 
[BEAM-4936] update org.apache.httpcomponents
URL: https://github.com/apache/beam/pull/6115#issuecomment-409441890
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129568)
Time Spent: 10m
Remaining Estimate: 0h

> Beam Dependency Update Request: org.codehaus.groovy
> ---
>
> Key: BEAM-4936
> URL: https://issues.apache.org/jira/browse/BEAM-4936
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:26:44.528984
> Please review and upgrade the org.codehaus.groovy to the latest 
> version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (a704a98 -> ddef7de)

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a704a98  Merge pull request #6107: Spelling mistakes
 add 2a851ad  Add convenience methods for pojo and javabean schema 
registration.
 add a1a07a5  Address code-review comments.
 new ddef7de  Merge pull request #6114: [BEAM-4453] Add convenience methods 
for pojo and javabean schema registration.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/beam/sdk/schemas/SchemaRegistry.java| 89 +-
 .../beam/sdk/schemas/SchemaRegistryTest.java   | 21 +
 2 files changed, 108 insertions(+), 2 deletions(-)

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #318

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129564
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 03:35
Start Date: 01/Aug/18 03:35
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #6114: [BEAM-4453] Add 
convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
index 895d8abb787..ec26c421937 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
@@ -28,6 +28,7 @@
 import java.util.ServiceLoader;
 import java.util.Set;
 import java.util.function.Function;
+import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
 import org.apache.beam.sdk.annotations.Experimental.Kind;
 import org.apache.beam.sdk.transforms.SerializableFunction;
@@ -67,8 +68,41 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  private static class PerTypeRegisteredProvider implements SchemaProvider {
+private final Map providers = 
Maps.newHashMap();
+
+void registerProvider(TypeDescriptor typeDescriptor, SchemaProvider 
schemaProvider) {
+  providers.put(typeDescriptor, schemaProvider);
+}
+
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.schemaFor(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction toRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.toRowFunction(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction fromRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.fromRowFunction(typeDescriptor) : null;
+}
+  }
+
+  private final PerTypeRegisteredProvider perTypeRegisteredProviders =
+  new PerTypeRegisteredProvider();
+
   private SchemaRegistry() {
 providers = new ArrayDeque<>(REGISTERED_SCHEMA_PROVIDERS);
+providers.addFirst(perTypeRegisteredProviders);
   }
 
   public static SchemaRegistry createDefault() {
@@ -104,6 +138,57 @@ public void registerSchemaProvider(SchemaProvider 
schemaProvider) {
 providers.addFirst(schemaProvider);
   }
 
+  /** Register a {@link SchemaProvider} to be used for a specific type. * */
+  public  void registerSchemaProvider(Class clazz, SchemaProvider 
schemaProvider) {
+registerSchemaProvider(TypeDescriptor.of(clazz), schemaProvider);
+  }
+
+  /** Register a {@link SchemaProvider} to be used for a specific type. * */
+  public  void registerSchemaProvider(
+  TypeDescriptor typeDescriptor, SchemaProvider schemaProvider) {
+perTypeRegisteredProviders.registerProvider(typeDescriptor, 
schemaProvider);
+  }
+
+  /**
+   * Register a POJO type for automatic schema inference.
+   *
+   * Currently schema field names will match field names in the POJO, and 
all fields must be
+   * mutable (i.e. no final fields).
+   */
+  public  void registerPOJO(Class clazz) {
+registerPOJO(TypeDescriptor.of(clazz));
+  }
+
+  /**
+   * Register a POJO type for automatic schema inference.
+   *
+   * Currently schema field names will match field names in the POJO, and 
all fields must be
+   * mutable (i.e. no final fields).
+   */
+  public  void registerPOJO(TypeDescriptor typeDescriptor) {
+registerSchemaProvider(typeDescriptor, new JavaFieldSchema());
+  }
+
+  /**
+   * Register a JavaBean type for automatic schema inference.
+   *
+   * Currently schema field names will match getter names in the bean, and 
all getters must have
+   * matching setters.
+   */
+  public  void registerJavaBean(Class clazz) {
+registerJavaBean(TypeDescriptor.of(clazz));
+  }
+
+  /**
+   * Register a JavaBean type for automatic schema inference.
+   *
+   * Currently schema field names will match getter names in the bean, and 
all getters must have
+   * matching setters.
+   */
+  public  void

[beam] 01/01: Merge pull request #6114: [BEAM-4453] Add convenience methods for pojo and javabean schema registration.

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ddef7de5f0f232d70e77482163852ea1a06e8220
Merge: a704a98 a1a07a5
Author: reuvenlax 
AuthorDate: Tue Jul 31 20:35:09 2018 -0700

Merge pull request #6114: [BEAM-4453] Add convenience methods for pojo and 
javabean schema registration.

 .../apache/beam/sdk/schemas/SchemaRegistry.java| 89 +-
 .../beam/sdk/schemas/SchemaRegistryTest.java   | 21 +
 2 files changed, 108 insertions(+), 2 deletions(-)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1136

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[devinduan] Spelling mistakes

--
[...truncated 19.77 MB...]
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:18.574Z: Autoscaling was automatically enabled for 
job 2018-07-31_20_13_18-5778142971044246673.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:21.084Z: Checking required Cloud APIs are enabled.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:21.394Z: Checking permissions granted to controller 
Service Account.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:25.192Z: Worker configuration: n1-standard-1 in 
us-central1-f.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:25.652Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:25.878Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:25.922Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.183Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.228Z: Elided trivial flatten 
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.276Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.303Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.335Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.374Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.412Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.455Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 01, 2018 3:13:28 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T03:13:26.523Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129563=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129563
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 03:03
Start Date: 01/Aug/18 03:03
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6114: [BEAM-4453] Add 
convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#issuecomment-409434139
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129563)
Time Spent: 10h 50m  (was: 10h 40m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6107: Spelling mistakes

2018-07-31 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a704a988037c40e28f436b04e9fd983c4bf41aee
Merge: fa41bac 2e8dc17
Author: Kenn Knowles 
AuthorDate: Tue Jul 31 19:25:44 2018 -0700

Merge pull request #6107: Spelling mistakes

 .../org/apache/beam/runners/spark/translation/EvaluationContext.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

[beam] branch master updated (fa41bac -> a704a98)

2018-07-31 Thread kenn

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fa41bac  [BEAM-4852] Only read symbol table when required.
 add 2e8dc17  Spelling mistakes
 new a704a98  Merge pull request #6107: Spelling mistakes

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/runners/spark/translation/EvaluationContext.java  | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1135

2018-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 19.55 MB...]
INFO: 2018-08-01T00:42:01.041Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.097Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.145Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.191Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.231Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.270Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.318Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.358Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.409Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Aug 01, 2018 12:42:03 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:42:01.458Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129553=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129553
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:40
Start Date: 01/Aug/18 00:40
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206725173
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
+private Map providers = Maps.newHashMap();
+
+void registerProvider(TypeDescriptor typeDescriptor, SchemaProvider 
schemaProvider) {
+  providers.put(typeDescriptor, schemaProvider);
+}
+
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.schemaFor(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction toRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.toRowFunction(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction fromRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.fromRowFunction(typeDescriptor) : null;
+}
+  }
+
+  private PerTypeRegisteredProvider perTypeRegisteredProviders = new 
PerTypeRegisteredProvider();
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129553)
Time Spent: 10.5h  (was: 10h 20m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129554
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:40
Start Date: 01/Aug/18 00:40
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206725206
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129554)
Time Spent: 10h 40m  (was: 10.5h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129552
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:40
Start Date: 01/Aug/18 00:40
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206725153
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
+private Map providers = Maps.newHashMap();
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129552)
Time Spent: 10h 20m  (was: 10h 10m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1134

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[github] [BEAM-4852] Only read symbol table when required.

[github] Update symbols.go

[rober] Avoid overwritting user changes to Resolver

[rober] Clean up deferedResolver

--
[...truncated 19.95 MB...]
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-07-31_17_32_16-14424834426018011843?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-07-31_17_32_16-14424834426018011843

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Aug 01, 2018 12:32:17 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-07-31_17_32_16-14424834426018011843
Aug 01, 2018 12:32:17 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-07-31_17_32_16-14424834426018011843 with 0 
expected assertions.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:16.766Z: Autoscaling is enabled for job 
2018-07-31_17_32_16-14424834426018011843. The number of workers will be between 
1 and 1000.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:16.811Z: Autoscaling was automatically enabled for 
job 2018-07-31_17_32_16-14424834426018011843.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:19.484Z: Checking required Cloud APIs are enabled.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:19.794Z: Checking permissions granted to controller 
Service Account.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:23.155Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:23.635Z: Expanding CoGroupByKey operations into 
optimizable parts.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:23.924Z: Expanding GroupByKey operations into 
optimizable parts.
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:23.990Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.318Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.375Z: Elided trivial flatten 
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.411Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.457Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.503Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 01, 2018 12:32:27 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-01T00:32:24.554Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #583

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Closed] (BEAM-5055) TEST: Precommit slowness

2018-07-31 Thread Udi Meiri (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri closed BEAM-5055.
---
   Resolution: Won't Fix
Fix Version/s: Not applicable

> TEST: Precommit slowness 
> -
>
> Key: BEAM-5055
> URL: https://issues.apache.org/jira/browse/BEAM-5055
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Labels: precommit
> Fix For: Not applicable
>
>
> please ignore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5055) TEST: Precommit slowness

2018-07-31 Thread Udi Meiri (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-5055:
---

Assignee: Udi Meiri  (was: Jason Kuster)

> TEST: Precommit slowness 
> -
>
> Key: BEAM-5055
> URL: https://issues.apache.org/jira/browse/BEAM-5055
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Labels: precommit
> Fix For: Not applicable
>
>
> please ignore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Java_Cron #171

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[timrobertson100] [BEAM-2661] Adds KuduIO

[github] [BEAM-4852] Only read symbol table when required.

[github] Update symbols.go

[timrobertson100] [BEAM-2661] KuduIO: Add missing licenses

[github] Don't rely on order of elements in a PCollection after GBK in

[relax] Convert BeamSQL to use Schemas.

[relax] Deprecate getRowCoder.

[relax] Add setSchema to remaining Table objects.

[relax] Delete a bunch of code that is no longer used.

[relax] Move utilities into schemas.utils package.

[relax] Convert Nexmark to use schemas.

[relax] remove deprecated Schema.getRowCoder

[relax] Apply spotless.

[relax] Plumb schema through DoFn.

[relax] Add helper setRowSchema to eliminatre boilerplate.

[rober] Avoid overwritting user changes to Resolver

[rober] Clean up deferedResolver

[pablo] Fix scheduling for jobs

--
[...truncated 16.08 MB...]
BeamCalcRel(expr#0..17=[{inputs}], name=[$t11], city=[$t14], state=[$t15], 
id=[$t0])
  BeamJoinRel(condition=[=($7, $10)], joinType=[inner])
BeamCalcRel(expr#0..9=[{inputs}], expr#10=[10], expr#11=[=($t8, $t10)], 
proj#0..9=[{exprs}], $condition=[$t11])
  BeamIOSourceRel(table=[[beam, Auction]])
BeamCalcRel(expr#0..7=[{inputs}], expr#8=['OR'], expr#9=[=($t5, $t8)], 
expr#10=['ID'], expr#11=[=($t5, $t10)], expr#12=['CA'], expr#13=[=($t5, $t12)], 
expr#14=[OR($t9, $t11, $t13)], proj#0..7=[{exprs}], $condition=[$t14])
  BeamIOSourceRel(table=[[beam, Person]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery0Test > testPassesBidsThrough 
STANDARD_ERROR
Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `PCOLLECTION`.`auction`, `PCOLLECTION`.`bidder`, 
`PCOLLECTION`.`price`, `PCOLLECTION`.`dateTime`, `PCOLLECTION`.`extra`
FROM `beam`.`PCOLLECTION` AS `PCOLLECTION`
Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], bidder=[$1], price=[$2], dateTime=[$3], 
extra=[$4])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])

Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], proj#0..4=[{exprs}])
  BeamIOSourceRel(table=[[beam, PCOLLECTION]])


org.apache.beam.sdk.nexmark.queries.sql.SqlQuery5Test > testBids STANDARD_ERROR
Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQL:
SELECT `AuctionBids`.`auction`, `AuctionBids`.`num`
FROM (SELECT `B1`.`auction`, COUNT(*) AS `num`, HOP_START(`B1`.`dateTime`, 
INTERVAL '5' SECOND, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B1`
GROUP BY `B1`.`auction`, HOP(`B1`.`dateTime`, INTERVAL '5' SECOND, INTERVAL 
'10' SECOND)) AS `AuctionBids`
INNER JOIN (SELECT MAX(`CountBids`.`num`) AS `maxnum`, 
`CountBids`.`starttime`
FROM (SELECT COUNT(*) AS `num`, HOP_START(`B2`.`dateTime`, INTERVAL '5' 
SECOND, INTERVAL '10' SECOND) AS `starttime`
FROM `beam`.`Bid` AS `B2`
GROUP BY `B2`.`auction`, HOP(`B2`.`dateTime`, INTERVAL '5' SECOND, INTERVAL 
'10' SECOND)) AS `CountBids`
GROUP BY `CountBids`.`starttime`) AS `MaxBids` ON `AuctionBids`.`starttime` 
= `MaxBids`.`starttime` AND `AuctionBids`.`num` >= `MaxBids`.`maxnum`
Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: SQLPlan>
LogicalProject(auction=[$0], num=[$1])
  LogicalJoin(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
LogicalProject(auction=[$0], num=[$2], starttime=[$1])
  LogicalAggregate(group=[{0, 1}], num=[COUNT()])
LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 1)])
  BeamIOSourceRel(table=[[beam, Bid]])
LogicalProject(maxnum=[$1], starttime=[$0])
  LogicalAggregate(group=[{0}], maxnum=[MAX($1)])
LogicalProject(starttime=[$1], num=[$0])
  LogicalProject(num=[$2], starttime=[$1])
LogicalAggregate(group=[{0, 1}], num=[COUNT()])
  LogicalProject(auction=[$0], $f1=[HOP($3, 5000, 1)])
BeamIOSourceRel(table=[[beam, Bid]])

Aug 01, 2018 12:26:15 AM 
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner convertToBeamRel
INFO: BEAMPlan>
BeamCalcRel(expr#0..4=[{inputs}], proj#0..1=[{exprs}])
  BeamJoinRel(condition=[AND(=($2, $4), >=($1, $3))], joinType=[inner])
BeamCalcRel(expr#0..2=[{inputs}], auction=[$t0], num=[$t2], 
starttime=[$t1])
  BeamAggregationRel(group=[{0, 1}], num=[COUNT()])
BeamCalcRel(expr#0..4=[{inputs}], expr#5=[5000], expr#6=[1], 
expr#7=[HOP($t3, $t5, $t6)], auction=[$t0], $f1=[$t7])

[jira] [Created] (BEAM-5055) TEST: Precommit slowness

2018-07-31 Thread Udi Meiri (JIRA)

Udi Meiri created BEAM-5055:
---

 Summary: TEST: Precommit slowness 
 Key: BEAM-5055
 URL: https://issues.apache.org/jira/browse/BEAM-5055
 Project: Beam
  Issue Type: Task
  Components: testing
Reporter: Udi Meiri
Assignee: Jason Kuster


please ignore



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129548
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:09
Start Date: 01/Aug/18 00:09
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206721142
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
 
 Review comment:
   `private`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129548)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129549
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:09
Start Date: 01/Aug/18 00:09
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206720643
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
+private Map providers = Maps.newHashMap();
 
 Review comment:
   `final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129549)
Time Spent: 10h 10m  (was: 10h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129547=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129547
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:09
Start Date: 01/Aug/18 00:09
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6114: 
[BEAM-4453] Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#discussion_r206720953
 
 

 ##
 File path: 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaRegistry.java
 ##
 @@ -67,8 +68,40 @@
   private final Map entries = Maps.newHashMap();
   private final ArrayDeque providers;
 
+  class PerTypeRegisteredProvider implements SchemaProvider {
+private Map providers = Maps.newHashMap();
+
+void registerProvider(TypeDescriptor typeDescriptor, SchemaProvider 
schemaProvider) {
+  providers.put(typeDescriptor, schemaProvider);
+}
+
+@Nullable
+@Override
+public  Schema schemaFor(TypeDescriptor typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.schemaFor(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction toRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.toRowFunction(typeDescriptor) : null;
+}
+
+@Nullable
+@Override
+public  SerializableFunction fromRowFunction(TypeDescriptor 
typeDescriptor) {
+  SchemaProvider schemaProvider = providers.get(typeDescriptor);
+  return (schemaProvider != null) ? 
schemaProvider.fromRowFunction(typeDescriptor) : null;
+}
+  }
+
+  private PerTypeRegisteredProvider perTypeRegisteredProviders = new 
PerTypeRegisteredProvider();
 
 Review comment:
   `final`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129547)
Time Spent: 10h  (was: 9h 50m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129546
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:07
Start Date: 01/Aug/18 00:07
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#discussion_r206720984
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
 ##
 @@ -154,13 +160,49 @@ public TopicPath topicPath() {
 return eventsTopicPath;
   }
 
+  public List listSubscriptions(ProjectPath projectPath, 
TopicPath topicPath)
 
 Review comment:
   nit: no need to expose this now


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129546)
Time Spent: 17h 10m  (was: 17h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 17h 10m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129545
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 01/Aug/18 00:07
Start Date: 01/Aug/18 00:07
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#discussion_r206721116
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/TestPubsub.java
 ##
 @@ -154,13 +160,49 @@ public TopicPath topicPath() {
 return eventsTopicPath;
   }
 
+  public List listSubscriptions(ProjectPath projectPath, 
TopicPath topicPath)
+  throws IOException {
+return pubsub.listSubscriptions(projectPath, topicPath);
+  }
+
   /** Publish messages to {@link #topicPath()}. */
   public void publish(List messages) throws IOException {
 List outgoingMessages =
 messages.stream().map(this::toOutgoingMessage).collect(toList());
 pubsub.publish(eventsTopicPath, outgoingMessages);
   }
 
+  /**
+   * Check if topics exist.
+   *
+   * @param project GCP project identifier.
+   * @param timeoutDuration Joda duration that sets a period of time before 
checking times out.
+   */
+  public void checkIfTopicExists(String project, Duration timeoutDuration)
 
 Review comment:
   nit: I would call it `checkIfAnySubscriptionExists()`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129545)
Time Spent: 17h  (was: 16h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 17h
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129544
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:59
Start Date: 31/Jul/18 23:59
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #6006: [BEAM-4808][SQL] add 
e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409405324
 
 
   SUCCESS 

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129544)
Time Spent: 16h 50m  (was: 16h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 16h 50m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129542
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:55
Start Date: 31/Jul/18 23:55
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409404715
 
 
   Right now this PR has removed tests that test `GROUP BY WINDOWING`. I 
created a separate JIRA to track tests for `GROUP BY WINDOWING`: 
https://issues.apache.org/jira/browse/BEAM-5054.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129542)
Time Spent: 16h 40m  (was: 16.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 16h 40m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129541
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:53
Start Date: 31/Jul/18 23:53
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#discussion_r206719032
 
 

 ##
 File path: 
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
 ##
 @@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.jdbc;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static 
org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLineTestingUtils.buildArgs;
+import static 
org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLineTestingUtils.toLines;
+import static org.hamcrest.CoreMatchers.everyItem;
+import static org.junit.Assert.assertThat;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.io.ByteArrayOutputStream;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.List;
+import java.util.TimeZone;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.extensions.gcp.options.GcpOptions;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.codehaus.jackson.map.ObjectMapper;
+import org.codehaus.jackson.node.ObjectNode;
+import org.hamcrest.collection.IsIn;
+import org.joda.time.Duration;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** BeamSqlLine integration tests. */
+public class BeamSqlLineIT implements Serializable {
+
+  @Rule public transient TestPubsub eventsTopic = TestPubsub.create();
+
+  private static String project;
+  private static String createPubsubTableStatement;
+  private static String setProject;
+  private static final SimpleDateFormat dateFormat = new 
SimpleDateFormat("-MM-dd HH:mm:ss");
+
+  private ExecutorService pool;
+
+  @BeforeClass
+  public static void setUpClass() {
+project = 
TestPipeline.testingPipelineOptions().as(GcpOptions.class).getProject();
+
+setProject = String.format("SET project = '%s';", project);
+
+createPubsubTableStatement =
+"CREATE TABLE taxi_rides (\n"
++ " event_timestamp TIMESTAMP,\n"
++ " attributes MAP,\n"
++ " payload ROW<\n"
++ "   ride_id VARCHAR,\n"
++ "   point_idx INT,\n"
++ "   latitude DOUBLE,\n"
++ "   longitude DOUBLE,\n"
++ "   meter_reading DOUBLE,\n"
++ "   meter_increment DOUBLE,\n"
++ "   ride_status VARCHAR,\n"
++ "   passenger_count TINYINT>)\n"
++ "   TYPE pubsub \n"
++ "   LOCATION '%s'\n"
++ "   TBLPROPERTIES '{\"timestampAttributeKey\": \"ts\"}';";
+
+dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
+  }
+
+  @Before
+  public void setUp() {
+pool = Executors.newFixedThreadPool(1);
+  }
+
+  @After
+  public void tearDown() {
+pool.shutdown();
+  }
+
+  @Test
+  public void testSelectFromPubsub() throws Exception {
+String[] args =
+buildArgs(
+String.format(createPubsubTableStatement, eventsTopic.topicPath()),
+setProject,
+"SELECT event_timestamp, taxi_rides.payload.ride_status, 
taxi_rides.payload.latitude, "
++ "taxi_rides.payload.longitude

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129540
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:52
Start Date: 31/Jul/18 23:52
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409404183
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129540)
Time Spent: 16h 20m  (was: 16h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 16h 20m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129539
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:51
Start Date: 31/Jul/18 23:51
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409404183
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129539)
Time Spent: 16h 10m  (was: 16h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 16h 10m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Go_GradleBuild #554

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129538
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:48
Start Date: 31/Jul/18 23:48
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409403567
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129538)
Time Spent: 16h  (was: 15h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 16h
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=129537=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129537
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:48
Start Date: 31/Jul/18 23:48
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-409403567
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129537)
Time Spent: 15h 50m  (was: 15h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 15h 50m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

svn commit: r28478 - /dev/beam/2.6.0/python/

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:44:02 2018
New Revision: 28478

Log:
Adding signatures and hashes for Python wheel files.

Added:

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.asc

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.sha512

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl.asc

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl.sha512

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl.asc

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl.sha512

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl.asc

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl.sha512

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc
==
--- 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc
 (added)
+++ 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.asc
 Tue Jul 31 23:44:02 2018
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEELx/tzfbdeZBCL0gvZSJOApLdilEFAltg8kUACgkQZSJOApLd
+ilE8GQ//dY+R71hBSuCMXkaL5W8TA5BCG2dnvQkDFmFQwfFJ0NhqcwF885rf2aSf
+mnvH/B8rhFN3u6+h9yJ6JtU00fByVrvZSvZ0zsdaAkx+jWi0X66qOq3FT9cJkyYd
+QUmBQFztw4cKh9tSYcpQbfAawMyXsUqOvz9a7csk17t3m+Jq6b525iLT9kCRViWm
+BskDNQehzswW4cCFAY6LUG0bsP3iMWQpHn03m+300JMcLtc78FOUgxovkbUbCH51
+QMW60JqkT0XuUpYVgTY/Pex5YPmmOUPWpyng6XouEsrlfvzJXuQorus+9ODUCEKK
+P/rPj8rnB8NZ1OwcnpBdYSiZ0yl4ndqInDPxPbsUYPZ9V4fyNFuHQEpMYun2RYIB
++ycXNB2FDw/RkAydCjfXhCo582U2L7oqNhtOsKTPxyW8M3BJRUGrToY7+UH2xKHT
+3spx8d5TrrxCbm/nv/X+DW+tU3wvgpHj558LbqEm5xtYxnXATMkWHWAZz0RbNk//
+pGLn0lBZDvU49jbZCYv+y9/yZ7anvdY6t2NvzWgZ9ORLGyFzA4dTn/5VNUViT8wR
+IjJN6VINTnMHZWuz9lPcKLyA1Rm9C7JsVBdbvfr3SjDNL2doE3nRbI1USJE3tw1p
+JGkeL891aIFbJi4+ItQZl7YqZq6hGvJtZ33ETDAC7EpJs87tGrk=
+=av8x
+-END PGP SIGNATURE-

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512
==
--- 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512
 (added)
+++ 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl.sha512
 Tue Jul 31 23:44:02 2018
@@ -0,0 +1 @@
+9b056399b67afc422ff8a54dffd9b3edbdaba202b420e3c3a4fd93645d326784754c14dec3066fed2e52b99ca8de8a404cd364a8678ae0ef22fc3fe70c7a7923
  
./apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.asc
==
--- 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.asc 
(added)
+++ 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.asc 
Tue Jul 31 23:44:02 2018
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEELx/tzfbdeZBCL0gvZSJOApLdilEFAltg8kUACgkQZSJOApLd
+ilEMRg/8DU4zMndVDFQPmoyyAzsRPI7GQjJP5uUOC3YLYJGINflHdkMR9SbDpO4W
+HuTA3ootdwNQD3nrTdYfUR88jEl0JHGnrftvJ6lmrsDjKuOmFb5hPQJguzTg/36q
+yuZBhKMjQmCTNWNQ+2IJ46HZaOieJFAaoThDLk3/c1hWhlSzRFVLCJNXs/roh/i7
+JHPSKpIrzNbcmJyOCo0q1GNspb3GcYXvipE08o8ks1fVsYwS3q9Oqyzqqb65CuVH
+Ez8fnw4zwymqcOvD962WLJnRTMzrw9B/xsDyXJiWPMKafEJIPZBiqrr1t53Xb8Py
+ZYDWlWLefe+FV7quhaQJvxGXVId48LagIuQFBA8YXUw5uCjyfiIYpT1x+zVBGGmJ
+JkjZq7i+ko30tGhcBm7FzOBUKsvptFIFzx3fHwPu+g3T37sPiBmWtbpsaI3u1U44
+DWqX/l2dXWao2ma47jkdtHjaGVlcyQbcdEBfCqy+rEb0sKRI7lE0xZbMjgSc8Pm5
+PWokyl9Lyjw+0k1PWGqP+U5CepzrUfKZ5fEYtqZz1Qkt8Qn7EgVDoJ7vuQ4WQ5Bw
+IrwlWivA/3cjHe3Tr85EQnBSRt/23bTVvQ+2aN4hEswgH8tkn8X82jpoa4dJJhbe
+xqHLNW3VvtXdheB3CrhHyOnYFCQH9WbyPZCM9cVXWq09xcucZPY=
+=End1
+-END PGP SIGNATURE-

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl.sha512
==
---

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=129536=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129536
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:40
Start Date: 31/Jul/18 23:40
Worklog Time Spent: 10m 
  Work Description: herohde closed pull request #6052: [BEAM-4852] Only 
read symbol table when required.
URL: https://github.com/apache/beam/pull/6052
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/symbols.go 
b/sdks/go/pkg/beam/core/runtime/symbols.go
index 238823696a8..c4f68163393 100644
--- a/sdks/go/pkg/beam/core/runtime/symbols.go
+++ b/sdks/go/pkg/beam/core/runtime/symbols.go
@@ -26,23 +26,41 @@ import (
 )
 
 var (
+   // Resolver is the accessible symbol resolver the runtime uses to find 
functions.
Resolver SymbolResolver
cache= make(map[string]interface{})
mu   sync.Mutex
 )
 
 func init() {
+   // defer initialization of the default resolver. This way
+   // the symbol table isn't read in unless strictly necessary.
+   Resolver = {initFn: initResolver}
+}
+
+type deferedResolver struct {
+   initFn func() SymbolResolver
+   r  SymbolResolver
+   init   sync.Once
+}
+
+func (d *deferedResolver) Sym2Addr(name string) (uintptr, error) {
+   d.init.Do(func() {
+   d.r = d.initFn()
+   })
+   return d.r.Sym2Addr(name)
+}
+
+func initResolver() SymbolResolver {
// First try the Linux location, since it's the most reliable.
if r, err := symtab.New("/proc/self/exe"); err == nil {
-   Resolver = r
-   return
+   return r
}
// For other OS's this works in most cases we need.
if r, err := symtab.New(os.Args[0]); err == nil {
-   Resolver = r
-   return
+   return r
}
-   Resolver = failResolver(false)
+   return failResolver(false)
 }
 
 // SymbolResolver resolves a symbol to an unsafe address.


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129536)
Time Spent: 1h 20m  (was: 1h 10m)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (4813879 -> fa41bac)

2018-07-31 Thread herohde

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4813879  Merge pull request #6104 from tvalentyn/patch-19
 add f5e517e  [BEAM-4852] Only read symbol table when required.
 add 697f66e  Update symbols.go
 add 1716717  Avoid overwritting user changes to Resolver
 add 4461966  Clean up deferedResolver
 new fa41bac  [BEAM-4852] Only read symbol table when required.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/symbols.go | 28 +++-
 1 file changed, 23 insertions(+), 5 deletions(-)

[beam] 01/01: [BEAM-4852] Only read symbol table when required.

2018-07-31 Thread herohde

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fa41bac7db88bda271fa52b26a87bb65c2a6e622
Merge: 4813879 4461966
Author: Henning Rohde 
AuthorDate: Tue Jul 31 16:40:42 2018 -0700

[BEAM-4852] Only read symbol table when required.

 sdks/go/pkg/beam/core/runtime/symbols.go | 28 +++-
 1 file changed, 23 insertions(+), 5 deletions(-)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129534
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:38
Start Date: 31/Jul/18 23:38
Worklog Time Spent: 10m 
  Work Description: reuvenlax opened a new pull request #6114: [BEAM-4453] 
Add convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129534)
Time Spent: 9h 40m  (was: 9.5h)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4453) Provide automatic schema registration for POJOs

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4453?focusedWorklogId=129535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129535
 ]

ASF GitHub Bot logged work on BEAM-4453:


Author: ASF GitHub Bot
Created on: 31/Jul/18 23:38
Start Date: 31/Jul/18 23:38
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6114: [BEAM-4453] Add 
convenience methods for pojo and javabean schema registration.
URL: https://github.com/apache/beam/pull/6114#issuecomment-409402043
 
 
   R:@apilloud 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129535)
Time Spent: 9h 50m  (was: 9h 40m)

> Provide automatic schema registration for POJOs
> ---
>
> Key: BEAM-4453
> URL: https://issues.apache.org/jira/browse/BEAM-4453
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4794) Move Nexmark and SQL to use the new Schema framework

2018-07-31 Thread Reuven Lax (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reuven Lax resolved BEAM-4794.
--
   Resolution: Fixed
Fix Version/s: 2.7.0

> Move Nexmark and SQL to use the new Schema framework
> 
>
> Key: BEAM-4794
> URL: https://issues.apache.org/jira/browse/BEAM-4794
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> This will allow SQL to accept user types. It will also allow the deletion of 
> a lot of code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5054) e2e tests for BeamSQL line to test group by window

2018-07-31 Thread Rui Wang (JIRA)

Rui Wang created BEAM-5054:
--

 Summary: e2e tests for BeamSQL line to test group by window
 Key: BEAM-5054
 URL: https://issues.apache.org/jira/browse/BEAM-5054
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Rui Wang
Assignee: Rui Wang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5053) CreateRequestsPerMinutePerUser quota limit exceeded

2018-07-31 Thread Mikhail Gryzykhin (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5053:
---

Assignee: Mikhail Gryzykhin  (was: Chamikara Jayalath)

> CreateRequestsPerMinutePerUser quota limit exceeded
> ---
>
> Key: BEAM-5053
> URL: https://issues.apache.org/jira/browse/BEAM-5053
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures, testing
>Reporter: Mikhail Gryzykhin
>Assignee: Mikhail Gryzykhin
>Priority: Major
>
> [https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull]
>  
> Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per 
> minute.
> Our Java Validate Runner tests spin up more than 250 jobs in a run.
> We should look into best way to fix the issue. Currently, I see following 
> options:
>  # Add retry logic. Although this limits us to 1 dataflow job start per 
> second for whole Jenkins. In long scale this can also block one test job if 
> other jobs take all the slots.
>  # Utilize different users to spin Dataflow jobs.
>  # Find way to rise quota limit on Dataflow. By default the field limits 
> value to 60 requests per minute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

svn commit: r28477 - /dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:26:51 2018
New Revision: 28477

Log:
Upload python wheels

Added:
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl  
 (with props)

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_i686.whl
--
svn:mime-type = application/octet-stream

svn commit: r28476 - /dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:23:54 2018
New Revision: 28476

Log:
Upload python wheels

Added:
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl 
  (with props)

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_i686.whl
--
svn:mime-type = application/octet-stream

svn commit: r28475 - /dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:22:38 2018
New Revision: 28475

Log:
Upload python wheels

Added:

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
   (with props)

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
--
svn:mime-type = application/octet-stream

[jira] [Created] (BEAM-5053) CreateRequestsPerMinutePerUser quota limit exceeded

2018-07-31 Thread Mikhail Gryzykhin (JIRA)

Mikhail Gryzykhin created BEAM-5053:
---

 Summary: CreateRequestsPerMinutePerUser quota limit exceeded
 Key: BEAM-5053
 URL: https://issues.apache.org/jira/browse/BEAM-5053
 Project: Beam
  Issue Type: Bug
  Components: test-failures, testing
Reporter: Mikhail Gryzykhin
Assignee: Chamikara Jayalath


[https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/553/consoleFull]

 

Dataflow has a CreateRequestsPerMinutePerUser quota that is 60 requests per 
minute.

Our Java Validate Runner tests spin up more than 250 jobs in a run.

We should look into best way to fix the issue. Currently, I see following 
options:
 # Add retry logic. Although this limits us to 1 dataflow job start per second 
for whole Jenkins. In long scale this can also block one test job if other jobs 
take all the slots.
 # Utilize different users to spin Dataflow jobs.
 # Find way to rise quota limit on Dataflow. By default the field limits value 
to 60 requests per minute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

svn commit: r28474 - /dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:22:29 2018
New Revision: 28474

Log:
Upload python wheels

Added:

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl   
(with props)

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27m-manylinux1_x86_64.whl
--
svn:mime-type = application/octet-stream

svn commit: r28473 - /dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 23:22:07 2018
New Revision: 28473

Log:
Upload python wheels

Added:

dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl   
(with props)

Added: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl
==
Binary file - no diff available.

Propchange: 
dev/beam/2.6.0/python/apache_beam-2.6.0.rc1-cp27-cp27mu-manylinux1_x86_64.whl
--
svn:mime-type = application/octet-stream

[jira] [Updated] (BEAM-4808) Add an integration test for BeamSqlLine

2018-07-31 Thread Rui Wang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang updated BEAM-4808:
---
Description: Test non group by window queries in Beam SQL Shell tutorial.

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 15h 40m
>  Remaining Estimate: 0h
>
> Test non group by window queries in Beam SQL Shell tutorial.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m

2018-07-31 Thread Udi Meiri (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564468#comment-16564468
 ] 

Udi Meiri commented on BEAM-5052:
-

[~swegner]

> beam_PreCommit_Website_Stage times out after 100m
> -
>
> Key: BEAM-5052
> URL: https://issues.apache.org/jira/browse/BEAM-5052
> Project: Beam
>  Issue Type: Bug
>  Components: testing, website
>Reporter: Udi Meiri
>Assignee: Jason Kuster
>Priority: Major
>
> This job stages the website on GCS for easier review.
>  I believe this only happens on the 2nd and further runs on a specific PR. A 
> lot of time is spent in deleting existing files (22k) and then copying over 
> newer ones.
> Example log: 
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText]
> A large chunk of the files are pydoc and javadoc files for version we don't 
> care about.
>  
> [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc]
>  
> [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc]
> Solution is to:
>  1. Delete unsupported versions' docs.
>  2. Only copy over to GCS the latest pydoc and javadoc, as the rest is 
> irrelevant for review (for almost all PRs). This can be done by deleting the 
> irrelevant directories before invoking "gsutil cp".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5030) Consolidate defer overhead per bundle

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5030?focusedWorklogId=129526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129526
 ]

ASF GitHub Bot logged work on BEAM-5030:


Author: ASF GitHub Bot
Created on: 31/Jul/18 22:58
Start Date: 31/Jul/18 22:58
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6102: [BEAM-5030]  
Consolidate defer overhead per bundle
URL: https://github.com/apache/beam/pull/6102#issuecomment-409394705
 
 
   ping @lostluck 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129526)
Time Spent: 20m  (was: 10m)

> Consolidate defer overhead per bundle
> -
>
> Key: BEAM-5030
> URL: https://issues.apache.org/jira/browse/BEAM-5030
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> At present, reflectx.CallNoPanic is invoked for every element, and adds  
> recovery defers for each element. Defers are cheap, but not "per element" 
> cheap. In a simple pipeline (a single ParDo), this represents around ~3% 
> overhead for the pipeline, which is aggravated for every pardo layer.
> Since the beam model's unit of re-triable computation is per bundle, the 
> recovery defers should be invoked per bundle only at the root. This is 
> already being done in 
> [exec/plan.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/plan.go#L112]
> This could be as simple as calling Call directly in 
> [exec/fn.go|https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/runtime/exec/fn.go#L99]
>  instead of with the reflectx wrapper as it is presently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4864) run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4864?focusedWorklogId=129525=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129525
 ]

ASF GitHub Bot logged work on BEAM-4864:


Author: ASF GitHub Bot
Created on: 31/Jul/18 22:57
Start Date: 31/Jul/18 22:57
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6061: [BEAM-4864] Make 
run_validatescontainer.sh not quite as hard coded.
URL: https://github.com/apache/beam/pull/6061#issuecomment-409394623
 
 
   Ping?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129525)
Time Spent: 40m  (was: 0.5h)

> run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness
> 
>
> Key: BEAM-4864
> URL: https://issues.apache.org/jira/browse/BEAM-4864
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The run_validatescontainer.sh script looks amazing! However I could not 
> validate my container, and this made me sad. We can make it configurable and 
> then people can validate their container changes more easily :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1133

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[github] Don't rely on order of elements in a PCollection after GBK in

--
[...truncated 19.58 MB...]
org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jul 31, 2018 10:53:29 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-07-31_15_53_28-11740841428829099983?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-07-31_15_53_28-11740841428829099983

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jul 31, 2018 10:53:29 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-07-31_15_53_28-11740841428829099983
Jul 31, 2018 10:53:29 PM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-07-31_15_53_28-11740841428829099983 with 0 
expected assertions.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:28.804Z: Autoscaling is enabled for job 
2018-07-31_15_53_28-11740841428829099983. The number of workers will be between 
1 and 1000.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:28.845Z: Autoscaling was automatically enabled for 
job 2018-07-31_15_53_28-11740841428829099983.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:31.792Z: Checking required Cloud APIs are enabled.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:31.988Z: Checking permissions granted to controller 
Service Account.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:35.360Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:35.842Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.051Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.091Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.317Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.351Z: Elided trivial flatten 
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.390Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.420Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.449Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 31, 2018 10:53:38 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:53:36.482Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema

[jira] [Commented] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m

2018-07-31 Thread Udi Meiri (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564463#comment-16564463
 ] 

Udi Meiri commented on BEAM-5052:
-

[~Ardagan] [~melap]

> beam_PreCommit_Website_Stage times out after 100m
> -
>
> Key: BEAM-5052
> URL: https://issues.apache.org/jira/browse/BEAM-5052
> Project: Beam
>  Issue Type: Bug
>  Components: testing, website
>Reporter: Udi Meiri
>Assignee: Jason Kuster
>Priority: Major
>
> This job stages the website on GCS for easier review.
>  I believe this only happens on the 2nd and further runs on a specific PR. A 
> lot of time is spent in deleting existing files (22k) and then copying over 
> newer ones.
> Example log: 
> [https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText]
> A large chunk of the files are pydoc and javadoc files for version we don't 
> care about.
>  
> [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc]
>  
> [https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc]
> Solution is to:
>  1. Delete unsupported versions' docs.
>  2. Only copy over to GCS the latest pydoc and javadoc, as the rest is 
> irrelevant for review (for almost all PRs). This can be done by deleting the 
> irrelevant directories before invoking "gsutil cp".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5052) beam_PreCommit_Website_Stage times out after 100m

2018-07-31 Thread Udi Meiri (JIRA)

Udi Meiri created BEAM-5052:
---

 Summary: beam_PreCommit_Website_Stage times out after 100m
 Key: BEAM-5052
 URL: https://issues.apache.org/jira/browse/BEAM-5052
 Project: Beam
  Issue Type: Bug
  Components: testing, website
Reporter: Udi Meiri
Assignee: Jason Kuster


This job stages the website on GCS for easier review.
 I believe this only happens on the 2nd and further runs on a specific PR. A 
lot of time is spent in deleting existing files (22k) and then copying over 
newer ones.

Example log: 
[https://builds.apache.org/job/beam_PreCommit_Website_Stage/1276/consoleText]

A large chunk of the files are pydoc and javadoc files for version we don't 
care about.
 
[https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/javadoc]
 
[https://github.com/apache/beam-site/tree/asf-site/content/documentation/sdks/pydoc]

Solution is to:
 1. Delete unsupported versions' docs.
 2. Only copy over to GCS the latest pydoc and javadoc, as the rest is 
irrelevant for review (for almost all PRs). This can be done by deleting the 
irrelevant directories before invoking "gsutil cp".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1132

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[pablo] Fix scheduling for jobs

--
[...truncated 20.73 MB...]
INFO: 2018-07-31T22:45:35.466Z: Autoscaling is enabled for job 
2018-07-31_15_45_35-1046827171708621319. The number of workers will be between 
1 and 1000.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:35.504Z: Autoscaling was automatically enabled for 
job 2018-07-31_15_45_35-1046827171708621319.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:38.074Z: Checking required Cloud APIs are enabled.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:38.396Z: Checking permissions granted to controller 
Service Account.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:41.808Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:42.304Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:42.550Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:42.601Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:42.878Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:42.925Z: Elided trivial flatten 
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.015Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.065Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.126Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.169Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.212Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.262Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jul 31, 2018 10:45:53 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T22:45:43.306Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema

svn commit: r28469 - in /dev/beam/2.6.0/python: ./ apache-beam-2.6.0.zip apache-beam-2.6.0.zip.asc apache-beam-2.6.0.zip.sha512

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 22:25:09 2018
New Revision: 28469

Log:
Staging python artifacts for 2.6.0 RC1.

Added:
dev/beam/2.6.0/python/
dev/beam/2.6.0/python/apache-beam-2.6.0.zip   (with props)
dev/beam/2.6.0/python/apache-beam-2.6.0.zip.asc
dev/beam/2.6.0/python/apache-beam-2.6.0.zip.sha512

Added: dev/beam/2.6.0/python/apache-beam-2.6.0.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.6.0/python/apache-beam-2.6.0.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.6.0/python/apache-beam-2.6.0.zip.asc
==
--- dev/beam/2.6.0/python/apache-beam-2.6.0.zip.asc (added)
+++ dev/beam/2.6.0/python/apache-beam-2.6.0.zip.asc Tue Jul 31 22:25:09 2018
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEELx/tzfbdeZBCL0gvZSJOApLdilEFAltg4TQACgkQZSJOApLd
+ilEXDQ//YZEYeSuAn36jXD3OB/NyF+2Vop4jQ9NfV0x1XOW/QlCKMaCpeUHiiZ6w
+tf+ZDZPgPMpA5xcXlUeL5W3IG9M0CKchCS+GnXzzksDSFRFHstvN/TwkR/Hx/kFa
+dU84PDAMVnHqtmmHhWT4+aDowF2yHqnC9VURbioLrofpah2RR3NkIhllq1Sdba+V
+EQweV4VrGS5/KEt1XnlEbh0+mM4NDKeo82BuNYT3d4nf9e9LVbytsBZho1eSrEBA
+4ubuTkSsK70V+vRgdY94ZMtFdtkx6etIuqz5nmHszXsB06tHw6HpByvHlBJzcUIP
+bkO0fMx0Ux+YZJHnEDPXhFSZ93mvoCufuRhrcY3RIctmvxfN8HoWd08l2hmcPSsz
+Hd8YghFWWe8W++hhza3Ysczq+qcoRk6XpRa2xNvCUxs5oNw6fILRHPtXQAHfciCV
+Hz267lTU9Io7SsmPUuy+KsKbtGIC3Tpuan7xZlxY4RfUKPrs6eSmCiVeDzKaU1Fx
+9HfUWEeADs1CfTcxiQcCPi46Z+P3TwqKDG5v2+T5jJ3ImYR2+S6axhtstlDUwZNt
+IkpP0JL4tAWkzAC795hMgHG52byjHQ7dkmuz6XAkJKUT03OmxXdz8qdcD60UDl+m
+mHyWnNnN6yV+Mrcm8L1Y+qddXRquOcei3gEjoNi/wEEfx9Q4nPc=
+=pinP
+-END PGP SIGNATURE-

Added: dev/beam/2.6.0/python/apache-beam-2.6.0.zip.sha512
==
--- dev/beam/2.6.0/python/apache-beam-2.6.0.zip.sha512 (added)
+++ dev/beam/2.6.0/python/apache-beam-2.6.0.zip.sha512 Tue Jul 31 22:25:09 2018
@@ -0,0 +1 @@
+237457e8894be9268b1b70bb72045faf5ab13d61df5015d6af092d442f7d0d803f18bed2f87e2df25290524a1ddaef8ae100fcdba70d47dc6cce89d40f3a7e50
  apache-beam-2.6.0.zip

svn commit: r28467 - in /dev/beam/2.6.0: ./ apache-beam-2.6.0-source-release.zip apache-beam-2.6.0-source-release.zip.asc apache-beam-2.6.0-source-release.zip.sha512

2018-07-31 Thread pabloem

Author: pabloem
Date: Tue Jul 31 22:14:22 2018
New Revision: 28467

Log:
Adding 2.6.0 source release candidate 1.

Added:
dev/beam/2.6.0/
dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip   (with props)
dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc
dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512

Added: dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip
==
Binary file - no diff available.

Propchange: dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip
--
svn:mime-type = application/octet-stream

Added: dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc
==
--- dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc (added)
+++ dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.asc Tue Jul 31 22:14:22 
2018
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEELx/tzfbdeZBCL0gvZSJOApLdilEFAltg3ewACgkQZSJOApLd
+ilF/Jw/8CrtwdlYDOd/j2Nv6U8kiZ2p0cBfjJJ+Dy48I1G0POtSnmcQevw5FPWIC
+5icai4zbvUpbArP++dO8osDZm+rUTgL1FC7YPST/F6kCnPLPEu3qvZ68uZQoVQ4t
+iA0//T73oV9pDSdTTVxUEgj7tl9S9DuPLmi102n56ikAQAfm97+D4qSwMv7or1Q2
+BCmZ3jIuqhEBtJ6NXwGkiMO7sP3FA2ECEO2CQQwYb8dsrzqz5IUCd/aSFTV+IpFw
++q/ofwMh3NuznNfzQDGRwo5Mg1QJshg1lh1NPJ0DxzbvB13OQZ1SiAYCwQ1NmeRp
+Jdd6wO72u0F/B5O2OhLmWwsn7mesa6BNDETYcfwBhdFEqGaW8kgTbXAK49ebumYT
+oU5+upXvH/j8XKmJ8k9fTlKr8bvDlXjovhtOwTw3O5azIzYmX2ZjLosCISOqt+FQ
+NxbzumTzZWrxTPZBKxw4Bghu/knsRn2+NlwzxxuzR+TzvJP12EGxa84tSKZJf+XI
+vlE7M6Kf4zh2Nv1yt0ifSkwWQ+TSlaARUrjjgg2DK/xwDQyeg4R+pXj/fwPwfYA8
+K7cWwQlL/XiFyk31QRQ6FkARBe3Rtv4rz/JdqeHCx0XXpugD1Sjt0WgbzLLVbQrR
+Hs0q9z7E2nf4jg8aR6n9Lr8hlssBMbwZf+5Hw+WgvXQnvci4Ppk=
+=RMVc
+-END PGP SIGNATURE-

Added: dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512
==
--- dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512 (added)
+++ dev/beam/2.6.0/apache-beam-2.6.0-source-release.zip.sha512 Tue Jul 31 
22:14:22 2018
@@ -0,0 +1 @@
+c69cadaf6b0d667f5fdabe24ef6f638d18d9c852ebf6b3aced8f8f6848c661b65badeda25d74bad349d100c324c7f4126fa8e20471f22fe2841602aee1479e0c
  apache-beam-2.6.0-source-release.zip

[jira] [Created] (BEAM-5051) Cleanup pubsub subscriptions

2018-07-31 Thread Andrew Pilloud (JIRA)

Andrew Pilloud created BEAM-5051:


 Summary: Cleanup pubsub subscriptions
 Key: BEAM-5051
 URL: https://issues.apache.org/jira/browse/BEAM-5051
 Project: Beam
  Issue Type: New Feature
  Components: io-java-gcp
Reporter: Andrew Pilloud
Assignee: Andrew Pilloud


[https://github.com/apache/beam/pull/346#discussion_r63624693]

We should cleanup pubsub subscriptions when the pipeline is canceled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4260) Document usage for hcatalog 1.1

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4260?focusedWorklogId=129518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129518
 ]

ASF GitHub Bot logged work on BEAM-4260:


Author: ASF GitHub Bot
Created on: 31/Jul/18 22:05
Start Date: 31/Jul/18 22:05
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #512: [BEAM-4260] 
Document HCatalogIO use with Hive 1.1
URL: https://github.com/apache/beam-site/pull/512#issuecomment-409383647
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129518)
Time Spent: 1h  (was: 50m)

> Document usage for hcatalog 1.1
> ---
>
> Key: BEAM-4260
> URL: https://issues.apache.org/jira/browse/BEAM-4260
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-hcatalog, website
>Affects Versions: 2.4.0
>Reporter: Tim Robertson
>Assignee: Tim Robertson
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The {{HCatalogIO}} does not work with environments providing Hive Server 1.x 
> which is in widespread use - as an example the latest Cloudera (5.14.2) 
> provides 1.1.x
>  
> The {{HCatalogIO}} marks it's Hive dependencies as provided, so I believe the 
> intention was to be open to multiple versions.
>  
> The issues come from the following:  
>  - use of {{HCatUtil.getHiveMetastoreClient(hiveConf)}} while previous 
> versions used the [now 
> deprecated|https://github.com/apache/hive/blob/master/hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java#L586]
>  {{getHiveClient(HiveConf hiveConf)}}  
>  - Changes to the signature of {{RetryingMetaStoreClient.getProxy(...)}}
>  
> Given this doesn't work in a major Hadoop distro, and will not until the next 
> CDH release later in 2018 (i.e. widespread adoption only expected in 2019) I 
> think it would be worthwhile providing a fix/workaround.
> I _think_ building for 2.3 and relocating in your own app might be a 
> workaround although I'm still testing it.  If that is successful I'd propose 
> adding it to the project README or in a separate markdown file linked from 
> the README.
> Does that sound like a reasonable approach please?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Go_GradleBuild #553

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[github] Don't rely on order of elements in a PCollection after GBK in

--
[...truncated 546.37 KB...]
"encoding": {
  "@type": "kind:windowed_value",
  "component_encodings": [
{
  "@type": "kind:bytes"
},
{
  "@type": "kind:global_window"
}
  ],
  "is_wrapper": true
}
  }
],
"parallel_input": {
  "@type": "OutputReference",
  "step_name": "e11",
  "output_name": "i0"
},
"serialized_fn": "e12"
  }
},
{
  "kind": "ParallelDo",
  "name": "e4",
  "properties": {
"user_name": "passert.Sum(emit1)/beam.addFixedKeyFn'1",
"output_info": [
  {
"user_name": "i0",
"output_name": "i0",
"encoding": {
  "@type": "kind:windowed_value",
  "component_encodings": [
{
  "@type": "kind:pair",
  "component_encodings": [
{
  "@type": "kind:length_prefix",
  "component_encodings": [
{
  "@type": 
"Cgd2YXJpbnR6EgIIAhqFAQpxZ2l0aHViLmNvbS9hcGFjaGUvYmVhbS9zZGtzL2dvL3Rlc3QvdmVuZG9yL2dpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4LmVuY1ZhckludFoSEAgWIgQIGUAPKgYIFBICCAgikQEKcWdpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby90ZXN0L3ZlbmRvci9naXRodWIuY29tL2FwYWNoZS9iZWFtL3Nka3MvZ28vcGtnL2JlYW0vY29yZS9ydW50aW1lL2NvZGVyeC5kZWNWYXJJbnRaEhwIFiIECBlAAyIGCBQSAggIKgQIGUAPKgQIGUAB"
}
  ]
},
{
  "@type": "kind:length_prefix",
  "component_encodings": [
{
  "@type": 
"Cgd2YXJpbnR6EgIIAhqFAQpxZ2l0aHViLmNvbS9hcGFjaGUvYmVhbS9zZGtzL2dvL3Rlc3QvdmVuZG9yL2dpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4LmVuY1ZhckludFoSEAgWIgQIGUAPKgYIFBICCAgikQEKcWdpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby90ZXN0L3ZlbmRvci9naXRodWIuY29tL2FwYWNoZS9iZWFtL3Nka3MvZ28vcGtnL2JlYW0vY29yZS9ydW50aW1lL2NvZGVyeC5kZWNWYXJJbnRaEhwIFiIECBlAAyIGCBQSAggIKgQIGUAPKgQIGUAB"
}
  ]
}
  ],
  "is_pair_like": true
},
{
  "@type": "kind:global_window"
}
  ],
  "is_wrapper": true
}
  }
],
"parallel_input": {
  "@type": "OutputReference",
  "step_name": "e3",
  "output_name": "i0"
},
"serialized_fn": "e4"
  }
},
{
  "kind": "GroupByKey",
  "name": "e5",
  "properties": {
"user_name": "passert.Sum(emit1)/CoGBK'1",
"disallow_combiner_lifting": true,
"output_info": [
  {
"user_name": "i0",
"output_name": "i0",
"encoding": {
  "@type": "kind:windowed_value",
  "component_encodings": [
{
  "@type": "kind:pair",
  "component_encodings": [
{
  "@type": "kind:length_prefix",
  "component_encodings": [
{
  "@type": 
"Cgd2YXJpbnR6EgIIAhqFAQpxZ2l0aHViLmNvbS9hcGFjaGUvYmVhbS9zZGtzL2dvL3Rlc3QvdmVuZG9yL2dpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4LmVuY1ZhckludFoSEAgWIgQIGUAPKgYIFBICCAgikQEKcWdpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby90ZXN0L3ZlbmRvci9naXRodWIuY29tL2FwYWNoZS9iZWFtL3Nka3MvZ28vcGtnL2JlYW0vY29yZS9ydW50aW1lL2NvZGVyeC5kZWNWYXJJbnRaEhwIFiIECBlAAyIGCBQSAggIKgQIGUAPKgQIGUAB"
}
  ]
},
{
  "@type": "kind:stream",
  "component_encodings": [
{
  "@type": "kind:length_prefix",
  "component_encodings": [
{
  "@type": 
"Cgd2YXJpbnR6EgIIAhqFAQpxZ2l0aHViLmNvbS9hcGFjaGUvYmVhbS9zZGtzL2dvL3Rlc3QvdmVuZG9yL2dpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvY29kZXJ4LmVuY1ZhckludFoSEAgWIgQIGUAPKgYIFBICCAgikQEKcWdpdGh1Yi5jb20vYXBhY2hlL2JlYW0vc2Rrcy9nby90ZXN0L3ZlbmRvci9naXRodWIuY29tL2FwYWNoZS9iZWFtL3Nka3MvZ28vcGtnL2JlYW0vY29yZS9ydW50aW1lL2NvZGVyeC5kZWNWYXJJbnRaEhwIFiIECBlAAyIGCBQSAggIKgQIGUAPKgQIGUAB"
}
  ]
}
  ],
  "is_stream_like":

[beam] branch master updated (fb9f5f4 -> 4813879)

2018-07-31 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fb9f5f4  Fix scheduling for jobs
 add 3bf1f98  Don't rely on order of elements in a PCollection after GBK in 
test_reshuffle_windows_unchanged.
 new 4813879  Merge pull request #6104 from tvalentyn/patch-19

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/transforms/util_test.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

[beam] 01/01: Merge pull request #6104 from tvalentyn/patch-19

2018-07-31 Thread altay

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4813879424f162a16b75e19acec16b8ede6a4952
Merge: fb9f5f4 3bf1f98
Author: Ahmet Altay 
AuthorDate: Tue Jul 31 14:58:11 2018 -0700

Merge pull request #6104 from tvalentyn/patch-19

Don't rely on order of elements in a PCollection after GBK in 
test_reshuffle_windows_unchanged

 sdks/python/apache_beam/transforms/util_test.py | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

[jira] [Resolved] (BEAM-2661) Add KuduIO

2018-07-31 Thread Tim Robertson (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tim Robertson resolved BEAM-2661.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).
> This work is in progress [on this 
> branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
> design aspects documented below.
> h2. The API
> The {{KuduIO}} API requires the user to provide a function to convert objects 
> into operations. This is similar to the {{JdbcIO}} but different to others, 
> such as {{HBaseIO}} which requires a pre-transform stage beforehand to 
> convert into the mutations to apply. It was originally intended to copy the 
> {{HBaseIO}} approach, but this was not possible:
>  # The Kudu 
> [Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
>  is a fat class, and is a subclass of {{KuduRpc}}. It 
> holds RPC logic, callbacks and a Kudu client. Because of this the 
> {{Operation}} does not serialize and furthermore, the logic for encoding the 
> operations (Insert, Upsert etc) in the Kudu Java API are one way only (no 
> decode) because the server is written in C++.
>  # An alternative could be to introduce a new object to beam (e.g. 
> {{o.a.b.sdk.io.kudu.KuduOperation}}) to enable 
> {{PCollection}}. This was considered but was discounted 
> because:
>  ## It is not a familiar API to those already knowing Kudu
>  ## It still requires serialization and deserialization of the operations. 
> Using the existing Kudu approach of serializing into compact byte arrays 
> would require a decoder along the lines of [this almost complete 
> example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
>  This is possible but has fragilities given the Kudu code itself continues to 
> evolve. 
>  ## It becomes a trivial codebase in Beam to maintain by defer the object to 
> mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
> precedent to do this.
> h2. Testing framework
> {{Kudu}} is written in C++. While a 
> [TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
>  does exist in Java, it requires binaries to be available for the target 
> environment which is not portable (edit: this is now a [work in 
> progress|https://issues.apache.org/jira/browse/KUDU-2411] in Kudu). Therefore 
> we opt for the following:
>  # Unit tests will use a mock Kudu client
>  # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
> Docker based Kudu instance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Fix scheduling for jobs

2018-07-31 Thread pabloem

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new fb9f5f4  Fix scheduling for jobs
fb9f5f4 is described below

commit fb9f5f44d53c01de18484f90fe4f22e80f65062c
Author: Mikhail Gryzykhin 
AuthorDate: Tue Jul 31 14:39:33 2018 -0700

Fix scheduling for jobs
---
 .test-infra/jenkins/CommonJobProperties.groovy  | 4 ++--
 .test-infra/jenkins/PostcommitJobBuilder.groovy | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/.test-infra/jenkins/CommonJobProperties.groovy 
b/.test-infra/jenkins/CommonJobProperties.groovy
index a804304..aa858ff 100644
--- a/.test-infra/jenkins/CommonJobProperties.groovy
+++ b/.test-infra/jenkins/CommonJobProperties.groovy
@@ -224,9 +224,9 @@ class CommonJobProperties {
 
   // Sets common config for jobs which run on a schedule; optionally on push
   static void setAutoJob(context,
- triggerOnCommit = false,
  String buildSchedule = '0 */6 * * *',
- notifyAddress = 'commits@beam.apache.org') {
+ notifyAddress = 'commits@beam.apache.org',
+ triggerOnCommit = false) {
 
 // Set build triggers
 context.triggers {
diff --git a/.test-infra/jenkins/PostcommitJobBuilder.groovy 
b/.test-infra/jenkins/PostcommitJobBuilder.groovy
index 3235582..3b53044 100644
--- a/.test-infra/jenkins/PostcommitJobBuilder.groovy
+++ b/.test-infra/jenkins/PostcommitJobBuilder.groovy
@@ -47,7 +47,7 @@ class PostcommitJobBuilder {
 
   void defineAutoPostCommitJob(name) {
 def autoBuilds = scope.job(name) {
-  commonJobProperties.setAutoJob delegate, true
+  commonJobProperties.setAutoJob delegate, '0 */6 * * *', 
'commits@beam.apache.org', true
 }
 
 autoBuilds.with(jobDefinition)

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129513=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129513
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:23
Start Date: 31/Jul/18 21:23
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #6111: 
[BEAM-4686] Add retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#discussion_r206688485
 
 

 ##
 File path: .test-infra/jenkins/job_PreCommit_Website_Merge.groovy
 ##
 @@ -17,6 +17,7 @@
  */
 
 import CommonJobProperties as commonJobProperties
 
 Review comment:
   That's github syntax highlight. It doesn't like underscores and sometimes 
spaces. I believe we figured this out with Scott at some point, but I can't 
remember specific reason for those by now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129513)
Time Spent: 1h 10m  (was: 1h)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129512
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:20
Start Date: 31/Jul/18 21:20
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on a change in pull request 
#6111: [BEAM-4686] Add retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#discussion_r206687604
 
 

 ##
 File path: .test-infra/jenkins/job_PreCommit_Website_Merge.groovy
 ##
 @@ -17,6 +17,7 @@
  */
 
 import CommonJobProperties as commonJobProperties
 
 Review comment:
   What's going on with the red boxes on this line?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129512)
Time Spent: 1h  (was: 50m)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1131

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[relax] Convert BeamSQL to use Schemas.

[relax] Deprecate getRowCoder.

[relax] Add setSchema to remaining Table objects.

[relax] Delete a bunch of code that is no longer used.

[relax] Move utilities into schemas.utils package.

[relax] Convert Nexmark to use schemas.

[relax] remove deprecated Schema.getRowCoder

[relax] Apply spotless.

[relax] Plumb schema through DoFn.

[relax] Add helper setRowSchema to eliminatre boilerplate.

--
[...truncated 19.81 MB...]
INFO: 2018-07-31T21:08:56.339Z: Autoscaling is enabled for job 
2018-07-31_14_08_56-7389818201553818395. The number of workers will be between 
1 and 1000.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:08:56.377Z: Autoscaling was automatically enabled for 
job 2018-07-31_14_08_56-7389818201553818395.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:08:59.259Z: Checking required Cloud APIs are enabled.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:08:59.474Z: Checking permissions granted to controller 
Service Account.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:03.229Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:03.675Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:03.932Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:03.980Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.233Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.280Z: Elided trivial flatten 
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.327Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.369Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.412Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.459Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.489Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 31, 2018 9:09:08 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T21:09:04.533Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=129509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129509
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:12
Start Date: 31/Jul/18 21:12
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r206685323
 
 

 ##
 File path: sdks/python/apache_beam/transforms/window.py
 ##
 @@ -246,10 +263,33 @@ def __init__(self, value, timestamp):
 self.value = value
 self.timestamp = Timestamp.of(timestamp)
 
-  def __cmp__(self, other):
-if type(self) is not type(other):
-  return cmp(type(self), type(other))
-return cmp((self.value, self.timestamp), (other.value, other.timestamp))
+  def __eq__(self, other):
 
 Review comment:
   Yes, I just retested with the full implementation which seems to work again. 
However, will be good to test the @total_ordering after your PR has been merged 
:-). 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129509)
Time Spent: 13h 40m  (was: 13.5h)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 13h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129510
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:12
Start Date: 31/Jul/18 21:12
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6111: [BEAM-4686] Add 
retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#issuecomment-409369632
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129510)
Time Spent: 50m  (was: 40m)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=129506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129506
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:06
Start Date: 31/Jul/18 21:06
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6110: [BEAM-4176] Tests 
for running Python on Flink.
URL: https://github.com/apache/beam/pull/6110#issuecomment-409368054
 
 
   R: @tweise @robertwb @bsidhom 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129506)
Time Spent: 8h 20m  (was: 8h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129504=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129504
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 21:04
Start Date: 31/Jul/18 21:04
Worklog Time Spent: 10m 
  Work Description: jasonkuster commented on issue #6111: [BEAM-4686] Add 
retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#issuecomment-409367551
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129504)
Time Spent: 40m  (was: 0.5h)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostRelease_NightlySnapshot #317

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[relax] Convert BeamSQL to use Schemas.

[relax] Deprecate getRowCoder.

[relax] Add setSchema to remaining Table objects.

[relax] Delete a bunch of code that is no longer used.

[relax] Move utilities into schemas.utils package.

[relax] Convert Nexmark to use schemas.

[relax] remove deprecated Schema.getRowCoder

[relax] Apply spotless.

[relax] Plumb schema through DoFn.

[relax] Add helper setRowSchema to eliminatre boilerplate.

--
[...truncated 1.61 MB...]
injector
LeaderBoard.java
StatefulTeamScore.java
UserScore.java
utils
Verified UserScore.java

**
* Test: Runs the WordCount Code with Dataflow runner
**

gsutil rm 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
 || echo 'No files'
CommandException: No URLs matched: 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count*
No files
mvn compile exec:java -q   
-Dexec.mainClass=org.apache.beam.examples.WordCount   
-Dexec.args="--runner=DataflowRunner
--project=apache-beam-testing
--gcpTempLocation=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp

--output=gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts
--inputFile=gs://apache-beam-samples/shakespeare/*" 
-Pdataflow-runner
Using maven /home/jenkins/tools/maven/apache-maven-3.5.2
Jul 31, 2018 8:56:29 PM 
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory
 create
INFO: No stagingLocation provided, falling back to gcpTempLocation
Jul 31, 2018 8:56:29 PM org.apache.beam.runners.dataflow.DataflowRunner 
fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from 
the classpath: will stage 114 files. Enable logging at DEBUG level to see which 
files will be staged.
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Executing pipeline on the Dataflow Service, which will have billing 
implications related to Google Compute Engine usage and other Google Cloud 
Services.
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Uploading 114 files from PipelineOptions.filesToStage to staging location 
to prepare for execution.
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-3027100417111690957-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-protobuf/2.7.0-SNAPSHOT/beam-sdks-java-extensions-protobuf-2.7.0-SNAPSHOT.jar
 to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-protobuf-2.7.0-SNAPSHOT-DhSo_R8cQVLPi_iTAtpZig.jar
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-3027100417111690957-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-extensions-google-cloud-platform-core/2.7.0-SNAPSHOT/beam-sdks-java-extensions-google-cloud-platform-core-2.7.0-SNAPSHOT.jar
 to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-extensions-google-cloud-platform-core-2.7.0-SNAPSHOT-DCjI2M-ZBLc0AIATAUjwaQ.jar
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-3027100417111690957-tmpdir/word-count-beam/target/classes 
to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/classes-AjNIrv5O9_ffvWejcw3bAQ.jar
Jul 31, 2018 8:56:30 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading 
/tmp/groovy-generated-3027100417111690957-tmpdir/.m2/repository/org/apache/beam/beam-sdks-java-core/2.7.0-SNAPSHOT/beam-sdks-java-core-2.7.0-SNAPSHOT.jar
 to 
gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/beam-sdks-java-core-2.7.0-SNAPSHOT-RQ__YLw1lDMHee53gMIbfA.jar
Jul 31, 2018 8:56:31 PM org.apache.beam.runners.dataflow.util.PackageUtil 
stageClasspathElements
INFO: Staging files complete: 110 files cached, 4 files newly uploaded
Jul 31, 2018 8:56:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding ReadLines/Read as step s1
Jul 31, 2018 8:56:32 PM org.apache.beam.sdk.io.FileBasedSource 
getEstimatedSizeBytes
INFO: Filepattern gs://apache-beam-samples/shakespeare/* matched 43 files with 
total size 5284696
Jul 31, 2018 8:56:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding WordCount.CountWords/ParDo(ExtractWords) as step s2
Jul 31, 2018 8:56:32 PM

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129499=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129499
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 20:52
Start Date: 31/Jul/18 20:52
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #6111: [BEAM-4686] Add 
retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#issuecomment-409363954
 
 
   R:@jasonkuster 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129499)
Time Spent: 0.5h  (was: 20m)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129497
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 20:51
Start Date: 31/Jul/18 20:51
Worklog Time Spent: 10m 
  Work Description: Ardagan opened a new pull request #6111: [BEAM-4686] 
Add retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111
 
 
   gnupg server has availability issues. 
   Adding retry logic to increase job stability.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129497)
Time Spent: 10m
Remaining Estimate: 0h

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>

[jira] [Work logged] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4686?focusedWorklogId=129498=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129498
 ]

ASF GitHub Bot logged work on BEAM-4686:


Author: ASF GitHub Bot
Created on: 31/Jul/18 20:51
Start Date: 31/Jul/18 20:51
Worklog Time Spent: 10m 
  Work Description: holdensmagicalunicorn commented on issue #6111: 
[BEAM-4686] Add retries for gpg key query logic.
URL: https://github.com/apache/beam/pull/6111#issuecomment-409363672
 
 
   @Ardagan, thanks! I am a bot who has found some folks who might be able to 
help with the review:@tgroh, @kennknowles and @lgajowy


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129498)
Time Spent: 20m  (was: 10m)

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #582

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[relax] Convert BeamSQL to use Schemas.

[relax] Deprecate getRowCoder.

[relax] Add setSchema to remaining Table objects.

[relax] Delete a bunch of code that is no longer used.

[relax] Move utilities into schemas.utils package.

[relax] Convert Nexmark to use schemas.

[relax] remove deprecated Schema.getRowCoder

[relax] Apply spotless.

[relax] Plumb schema through DoFn.

[relax] Add helper setRowSchema to eliminatre boilerplate.

--
[...truncated 253.20 KB...]
INFO: 2018-07-31T20:23:58.093Z: Fusing consumer Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow into 
Write xml files/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.140Z: Fusing consumer Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify into Write 
xml files/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.187Z: Fusing consumer Write xml 
files/WriteFiles/FinalizeTempFileBundles/Finalize into Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.227Z: Fusing consumer Write xml 
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with 
random key into Write xml files/WriteFiles/FinalizeTempFileBundles/Finalize
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.300Z: Fusing consumer Read xml 
files/ReadAllViaFileBasedSource/Read ranges into Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Values/Values/Map
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.371Z: Fusing consumer Get file names/Values/Map 
into Write xml 
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.421Z: Fusing consumer Write xml 
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
 into Write xml 
files/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.471Z: Fusing consumer Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write into Write 
xml files/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.520Z: Fusing consumer Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/GroupByKey/Write into Read 
xml files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.570Z: Fusing consumer Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/GroupByKey/Reify into Read 
xml 
files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.618Z: Fusing consumer Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Values/Values/Map into Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/ExpandIterable
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.655Z: Fusing consumer Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 into Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.702Z: Fusing consumer Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into Write xml 
files/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 31, 2018 8:24:02 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T20:23:58.752Z: Fusing consumer Read xml 
files/ReadAllViaFileBasedSource/Reshuffle/Reshuffle/Window.Into()/Window.Assign 
into Read

Jenkins build is back to normal : beam_PerformanceTests_JDBC #918

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=129493=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129493
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 31/Jul/18 20:34
Start Date: 31/Jul/18 20:34
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #6110: [BEAM-4176] 
Tests for running Python on Flink.
URL: https://github.com/apache/beam/pull/6110
 
 
   This provides a fairly comprehensive tests of the basic runner compatibility 
expectations for Python on Flink. Surprisingly, the suite only takes a minute 
or so to run, so totally feasible to run on presubmit.
   
   Copy of https://github.com/apache/beam/pull/5942
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129493)
Time Spent: 8h 10m  (was: 8h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type:

[jira] [Commented] (BEAM-4440) When filesToStage is empty, the DataflowRunner should fail.

2018-07-31 Thread Ahmet Altay (JIRA)



[ 
https://issues.apache.org/jira/browse/BEAM-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16564333#comment-16564333
 ] 

Ahmet Altay commented on BEAM-4440:
---

Skipping non-existent file to stage should also be an error instead of a 
warning.

https://github.com/apache/beam/blob/1974b920e4b3bbe8549e25fe789f9dada13c1769/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L357

> When filesToStage is empty, the DataflowRunner should fail.
> ---
>
> Key: BEAM-4440
> URL: https://issues.apache.org/jira/browse/BEAM-4440
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Thomas Groh
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>
> Staging no files will not permit any pipeline execution to succeed. This 
> should cause the pipeline to fail before job submission.
>  
> In general, if any file to stage is specified and expands to no files, this 
> should also be a failure. Specifying a glob that represents no files is 
> almost certainly user error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5050) [SQL] NULLs are aggregated incorrectly

2018-07-31 Thread Kai Jiang (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-5050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kai Jiang updated BEAM-5050:

Description: For example, COUNT(field) should not count records with NULL 
field. We also should handle and test on other aggregation functions (like AVG, 
SUM, MIN, MAX, VAR_POP, VAR_SAMP, etc.)  (was: For example, COUNT(field) should 
not count records with NULL field)

> [SQL] NULLs are aggregated incorrectly
> --
>
> Key: BEAM-5050
> URL: https://issues.apache.org/jira/browse/BEAM-5050
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Priority: Major
>
> For example, COUNT(field) should not count records with NULL field. We also 
> should handle and test on other aggregation functions (like AVG, SUM, MIN, 
> MAX, VAR_POP, VAR_SAMP, etc.)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4794) Move Nexmark and SQL to use the new Schema framework

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4794?focusedWorklogId=129489=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129489
 ]

ASF GitHub Bot logged work on BEAM-4794:


Author: ASF GitHub Bot
Created on: 31/Jul/18 20:15
Start Date: 31/Jul/18 20:15
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #5956: [BEAM-4794] Move 
SQL and SQL Nexmark to the schema framework
URL: https://github.com/apache/beam/pull/5956#issuecomment-409353062
 
 
   UnboundedEventSourceTest.resumeFromCheckpoint failed in Post Commit, but 
appears to simply be flaky. That same test ran successfully in PreCommit, and 
also passed locally.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129489)
Time Spent: 2h 20m  (was: 2h 10m)

> Move Nexmark and SQL to use the new Schema framework
> 
>
> Key: BEAM-4794
> URL: https://issues.apache.org/jira/browse/BEAM-4794
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> This will allow SQL to accept user types. It will also allow the deletion of 
> a lot of code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (eb0b611 -> 06128f2)

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from eb0b611  Merge pull request #6021: [BEAM-2661] Adds KuduIO
 add e3053bf  Convert BeamSQL to use Schemas.
 add ab7dd01  Deprecate getRowCoder.
 add 7478ef6  Add setSchema to remaining Table objects.
 add d5d6dc0  Delete a bunch of code that is no longer used.
 add 0956570  Move utilities into schemas.utils package.
 add e7bc066  Convert Nexmark to use schemas.
 add da97637  remove deprecated Schema.getRowCoder
 add ecc8e47  Apply spotless.
 add de7eea4  Plumb schema through DoFn.
 add 1c969b4  Add helper setRowSchema to eliminatre boilerplate.
 new 06128f2  Merge pull request #5956: [BEAM-4794] Move SQL and SQL 
Nexmark to the schema framework

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../reflect => schemas}/FieldValueGetter.java  |   4 +-
 .../FieldValueGetterFactory.java   |  10 +-
 .../reflect => schemas}/FieldValueSetter.java  |   4 +-
 .../FieldValueSetterFactory.java   |  10 +-
 .../sdk/schemas/GetterBasedSchemaProvider.java |  15 +-
 .../apache/beam/sdk/schemas/JavaBeanSchema.java|   6 +-
 .../apache/beam/sdk/schemas/JavaFieldSchema.java   |   6 +-
 .../java/org/apache/beam/sdk/schemas/Schema.java   |  42 ++--
 .../org/apache/beam/sdk/schemas/SchemaCoder.java   |   7 +
 .../beam/sdk/schemas/transforms/Convert.java   |   3 +-
 .../beam/sdk/schemas/utils/ByteBuddyUtils.java |   4 +-
 .../utils}/JavaBeanGetterFactory.java  |   5 +-
 .../utils}/JavaBeanSetterFactory.java  |   5 +-
 .../beam/sdk/schemas/utils/JavaBeanUtils.java  |   4 +-
 .../apache/beam/sdk/schemas/utils/POJOUtils.java   |   8 +-
 .../utils}/PojoValueGetterFactory.java |   5 +-
 .../utils}/PojoValueSetterFactory.java |   5 +-
 .../org/apache/beam/sdk/testing/TestStream.java|  11 ++
 .../org/apache/beam/sdk/transforms/JsonToRow.java  |   2 +-
 .../java/org/apache/beam/sdk/transforms/ParDo.java |  26 ++-
 .../org/apache/beam/sdk/values/PCollection.java|  14 ++
 .../main/java/org/apache/beam/sdk/values/Row.java  |   2 +-
 .../org/apache/beam/sdk/values/RowWithGetters.java |   4 +-
 .../beam/sdk/values/reflect/ByteBuddyUtils.java|  96 -
 .../sdk/values/reflect/DefaultSchemaFactory.java   |  92 -
 .../sdk/values/reflect/GeneratedGetterFactory.java | 106 --
 .../beam/sdk/values/reflect/GetterFactory.java |  31 ---
 .../beam/sdk/values/reflect/InferredRowCoder.java  | 121 
 .../beam/sdk/values/reflect/ReflectionGetter.java  |  59 --
 .../values/reflect/ReflectionGetterFactory.java|  57 --
 .../beam/sdk/values/reflect/ReflectionUtils.java   |  66 ---
 .../apache/beam/sdk/values/reflect/RowFactory.java | 140 -
 .../beam/sdk/values/reflect/SchemaFactory.java |  41 
 .../beam/sdk/values/reflect/SchemaGetters.java |  51 -
 .../beam/sdk/values/reflect/package-info.java  |  20 --
 .../beam/sdk/schemas/utils/JavaBeanUtilsTest.java  |   4 +-
 .../beam/sdk/schemas/utils/POJOUtilsTest.java  |   4 +-
 .../apache/beam/sdk/transforms/JsonToRowTest.java  |   2 +-
 .../beam/sdk/transforms/ParDoSchemaTest.java   |  15 ++
 .../values/reflect/DefaultSchemaFactoryTest.java   |  94 -
 .../values/reflect/GeneratedGetterFactoryTest.java |  97 -
 .../sdk/values/reflect/InferredRowCoderTest.java   |  90 -
 .../reflect/ReflectionGetterFactoryTest.java   |  97 -
 .../sdk/values/reflect/ReflectionGetterTest.java   |  90 -
 .../beam/sdk/values/reflect/RowFactoryTest.java| 135 -
 .../beam/sdk/values/reflect/SchemaGettersTest.java |  41 
 .../beam/sdk/extensions/sql/SchemaHelper.java  |  66 ---
 .../beam/sdk/extensions/sql/SqlTransform.java  |   9 +-
 .../sdk/extensions/sql/example/BeamSqlExample.java |   7 +-
 .../extensions/sql/example/BeamSqlPojoExample.java |  45 ++---
 .../sdk/extensions/sql/example/model/Customer.java |  17 ++
 .../sdk/extensions/sql/example/model/Order.java|  13 ++
 .../operator/BeamSqlCastExpression.java|   3 +-
 .../sql/impl/rel/BeamAggregationRel.java   |   8 +-
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |   9 +-
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  10 +-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |   8 +-
 .../extensions/sql/impl/rel/BeamUncollectRel.java  |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java |   2 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |   2 +-
 .../sql/impl/schema/BeamPCollectionTable.java  |  15 +-

[beam] 01/01: Merge pull request #5956: [BEAM-4794] Move SQL and SQL Nexmark to the schema framework

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 06128f27d1780f25c23ca65cc7ace693a78dac80
Merge: eb0b611 1c969b4
Author: reuvenlax 
AuthorDate: Tue Jul 31 13:15:33 2018 -0700

Merge pull request #5956: [BEAM-4794] Move SQL and SQL Nexmark to the 
schema framework

 .../reflect => schemas}/FieldValueGetter.java  |   4 +-
 .../FieldValueGetterFactory.java   |  10 +-
 .../reflect => schemas}/FieldValueSetter.java  |   4 +-
 .../FieldValueSetterFactory.java   |  10 +-
 .../sdk/schemas/GetterBasedSchemaProvider.java |  15 +-
 .../apache/beam/sdk/schemas/JavaBeanSchema.java|   6 +-
 .../apache/beam/sdk/schemas/JavaFieldSchema.java   |   6 +-
 .../java/org/apache/beam/sdk/schemas/Schema.java   |  42 ++--
 .../org/apache/beam/sdk/schemas/SchemaCoder.java   |   7 +
 .../beam/sdk/schemas/transforms/Convert.java   |   3 +-
 .../beam/sdk/schemas/utils/ByteBuddyUtils.java |   4 +-
 .../utils}/JavaBeanGetterFactory.java  |   5 +-
 .../utils}/JavaBeanSetterFactory.java  |   5 +-
 .../beam/sdk/schemas/utils/JavaBeanUtils.java  |   4 +-
 .../apache/beam/sdk/schemas/utils/POJOUtils.java   |   8 +-
 .../utils}/PojoValueGetterFactory.java |   5 +-
 .../utils}/PojoValueSetterFactory.java |   5 +-
 .../org/apache/beam/sdk/testing/TestStream.java|  11 ++
 .../org/apache/beam/sdk/transforms/JsonToRow.java  |   2 +-
 .../java/org/apache/beam/sdk/transforms/ParDo.java |  26 ++-
 .../org/apache/beam/sdk/values/PCollection.java|  14 ++
 .../main/java/org/apache/beam/sdk/values/Row.java  |   2 +-
 .../org/apache/beam/sdk/values/RowWithGetters.java |   4 +-
 .../beam/sdk/values/reflect/ByteBuddyUtils.java|  96 -
 .../sdk/values/reflect/DefaultSchemaFactory.java   |  92 -
 .../sdk/values/reflect/GeneratedGetterFactory.java | 106 --
 .../beam/sdk/values/reflect/GetterFactory.java |  31 ---
 .../beam/sdk/values/reflect/InferredRowCoder.java  | 121 
 .../beam/sdk/values/reflect/ReflectionGetter.java  |  59 --
 .../values/reflect/ReflectionGetterFactory.java|  57 --
 .../beam/sdk/values/reflect/ReflectionUtils.java   |  66 ---
 .../apache/beam/sdk/values/reflect/RowFactory.java | 140 -
 .../beam/sdk/values/reflect/SchemaFactory.java |  41 
 .../beam/sdk/values/reflect/SchemaGetters.java |  51 -
 .../beam/sdk/values/reflect/package-info.java  |  20 --
 .../beam/sdk/schemas/utils/JavaBeanUtilsTest.java  |   4 +-
 .../beam/sdk/schemas/utils/POJOUtilsTest.java  |   4 +-
 .../apache/beam/sdk/transforms/JsonToRowTest.java  |   2 +-
 .../beam/sdk/transforms/ParDoSchemaTest.java   |  15 ++
 .../values/reflect/DefaultSchemaFactoryTest.java   |  94 -
 .../values/reflect/GeneratedGetterFactoryTest.java |  97 -
 .../sdk/values/reflect/InferredRowCoderTest.java   |  90 -
 .../reflect/ReflectionGetterFactoryTest.java   |  97 -
 .../sdk/values/reflect/ReflectionGetterTest.java   |  90 -
 .../beam/sdk/values/reflect/RowFactoryTest.java| 135 -
 .../beam/sdk/values/reflect/SchemaGettersTest.java |  41 
 .../beam/sdk/extensions/sql/SchemaHelper.java  |  66 ---
 .../beam/sdk/extensions/sql/SqlTransform.java  |   9 +-
 .../sdk/extensions/sql/example/BeamSqlExample.java |   7 +-
 .../extensions/sql/example/BeamSqlPojoExample.java |  45 ++---
 .../sdk/extensions/sql/example/model/Customer.java |  17 ++
 .../sdk/extensions/sql/example/model/Order.java|  13 ++
 .../operator/BeamSqlCastExpression.java|   3 +-
 .../sql/impl/rel/BeamAggregationRel.java   |   8 +-
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |   9 +-
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  10 +-
 .../sdk/extensions/sql/impl/rel/BeamSortRel.java   |   8 +-
 .../extensions/sql/impl/rel/BeamUncollectRel.java  |   4 +-
 .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java |   2 +-
 .../sdk/extensions/sql/impl/rel/BeamValuesRel.java |   2 +-
 .../sql/impl/schema/BeamPCollectionTable.java  |  15 +-
 .../sql/impl/transform/BeamJoinTransforms.java |  64 +++---
 .../meta/provider/bigquery/BeamBigQueryTable.java  |   3 +-
 .../sql/meta/provider/kafka/BeamKafkaCSVTable.java |  26 +--
 .../sql/meta/provider/kafka/BeamKafkaTable.java|   3 +-
 .../meta/provider/pubsub/PubsubIOJsonTable.java|   3 +-
 .../sql/meta/provider/test/TestTableProvider.java  |   7 +-
 .../sql/meta/provider/text/TextTableProvider.java  |  20 +-
 .../sql/BeamSqlDslAggregationCovarianceTest.java   |   7 +-
 .../extensions/sql/BeamSqlDslAggregationTest.java  |  15 +-
 .../sql/BeamSqlDslAggregationVarianceTest.java |   7 +-
 .../sdk/extensions/sql/BeamSqlDslArrayTest.java| 177 +
 .../beam/sdk/extensions/sql/BeamSqlDslBase.java|  30 ++-

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1130

2018-07-31 Thread Apache Jenkins Server

See

[jira] [Work logged] (BEAM-2661) Add KuduIO

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?focusedWorklogId=129480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129480
 ]

ASF GitHub Bot logged work on BEAM-2661:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:42
Start Date: 31/Jul/18 19:42
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #6021: [BEAM-2661] 
Adds KuduIO
URL: https://github.com/apache/beam/pull/6021#issuecomment-409343767
 
 
   That was fast. Thanks @reuvenlax 
   
   FYI: I hope to be assigned owner of KuduIO, will file Jiras for all 
improvements, and will encourage others to contribute. I've also volunteered to 
write a guest blog on Beam/Kudu for the Kudu team who are trying to raise the 
profile of their project (CC @griscz for info)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129480)
Time Spent: 4h 50m  (was: 4h 40m)

> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).
> This work is in progress [on this 
> branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
> design aspects documented below.
> h2. The API
> The {{KuduIO}} API requires the user to provide a function to convert objects 
> into operations. This is similar to the {{JdbcIO}} but different to others, 
> such as {{HBaseIO}} which requires a pre-transform stage beforehand to 
> convert into the mutations to apply. It was originally intended to copy the 
> {{HBaseIO}} approach, but this was not possible:
>  # The Kudu 
> [Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
>  is a fat class, and is a subclass of {{KuduRpc}}. It 
> holds RPC logic, callbacks and a Kudu client. Because of this the 
> {{Operation}} does not serialize and furthermore, the logic for encoding the 
> operations (Insert, Upsert etc) in the Kudu Java API are one way only (no 
> decode) because the server is written in C++.
>  # An alternative could be to introduce a new object to beam (e.g. 
> {{o.a.b.sdk.io.kudu.KuduOperation}}) to enable 
> {{PCollection}}. This was considered but was discounted 
> because:
>  ## It is not a familiar API to those already knowing Kudu
>  ## It still requires serialization and deserialization of the operations. 
> Using the existing Kudu approach of serializing into compact byte arrays 
> would require a decoder along the lines of [this almost complete 
> example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
>  This is possible but has fragilities given the Kudu code itself continues to 
> evolve. 
>  ## It becomes a trivial codebase in Beam to maintain by defer the object to 
> mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
> precedent to do this.
> h2. Testing framework
> {{Kudu}} is written in C++. While a 
> [TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
>  does exist in Java, it requires binaries to be available for the target 
> environment which is not portable (edit: this is now a [work in 
> progress|https://issues.apache.org/jira/browse/KUDU-2411] in Kudu). Therefore 
> we opt for the following:
>  # Unit tests will use a mock Kudu client
>  # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
> Docker based Kudu instance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4842) Update Flink Runner to Flink 1.5.2

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4842?focusedWorklogId=129478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129478
 ]

ASF GitHub Bot logged work on BEAM-4842:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:40
Start Date: 31/Jul/18 19:40
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #6109: [BEAM-4842] 
Update Flink Runner to Flink 1.5.2
URL: https://github.com/apache/beam/pull/6109
 
 
   Update Flink Runner to Flink 1.5.2
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129478)
Time Spent: 2h  (was: 1h 50m)

> Update Flink Runner to Flink 1.5.2
> --
>
> Key: BEAM-4842
> URL: https://issues.apache.org/jira/browse/BEAM-4842
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4842) Update Flink Runner to Flink 1.5.2

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4842?focusedWorklogId=129479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129479
 ]

ASF GitHub Bot logged work on BEAM-4842:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:40
Start Date: 31/Jul/18 19:40
Worklog Time Spent: 10m 
  Work Description: holdensmagicalunicorn commented on issue #6109: 
[BEAM-4842] Update Flink Runner to Flink 1.5.2
URL: https://github.com/apache/beam/pull/6109#issuecomment-409343155
 
 
   @tweise, thanks! I am a bot who has found some folks who might be able to 
help with the review:@lukecwik, @jkff and @echauchot


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129479)
Time Spent: 2h 10m  (was: 2h)

> Update Flink Runner to Flink 1.5.2
> --
>
> Key: BEAM-4842
> URL: https://issues.apache.org/jira/browse/BEAM-4842
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=129476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129476
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:35
Start Date: 31/Jul/18 19:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6052: [BEAM-4852] Only 
read symbol table when required.
URL: https://github.com/apache/beam/pull/6052#issuecomment-409341836
 
 
   Thanks PTAL @herohde 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129476)
Time Spent: 1h  (was: 50m)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #123

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[relax] Fix BigQuery job retry mechanism.

[relax] Add exponential backoff on retries.

[relax] Apply code-review comment.

[relax] Properly handle running jobs.

--
[...truncated 20.51 MB...]
> Failed to publish publication 'mavenJava' to repository 'maven'
   > Could not write to resource 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-file-system/2.7.0-SNAPSHOT/beam-sdks-java-io-hadoop-file-system-2.7.0-20180731.193004-12.jar'.
  > Could not PUT 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-file-system/2.7.0-SNAPSHOT/beam-sdks-java-io-hadoop-file-system-2.7.0-20180731.193004-12.jar'.
 Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to 
get more log output. Run with --scan to get full insights.
==

43: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':beam-sdks-java-io-hadoop-input-format:publishMavenJavaPublicationToMavenRepository'.
> Failed to publish publication 'mavenJava' to repository 'maven'
   > Could not write to resource 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-input-format/2.7.0-SNAPSHOT/beam-sdks-java-io-hadoop-input-format-2.7.0-20180731.193026-12.jar'.
  > Could not PUT 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-input-format/2.7.0-SNAPSHOT/beam-sdks-java-io-hadoop-input-format-2.7.0-20180731.193026-12.jar'.
 Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to 
get more log output. Run with --scan to get full insights.
==

44: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':beam-sdks-java-io-hbase:publishMavenJavaPublicationToMavenRepository'.
> Failed to publish publication 'mavenJava' to repository 'maven'
   > Could not write to resource 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hbase/2.7.0-SNAPSHOT/beam-sdks-java-io-hbase-2.7.0-20180731.193040-12.jar'.
  > Could not PUT 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hbase/2.7.0-SNAPSHOT/beam-sdks-java-io-hbase-2.7.0-20180731.193040-12.jar'.
 Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to 
get more log output. Run with --scan to get full insights.
==

45: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':beam-sdks-java-io-hcatalog:publishMavenJavaPublicationToMavenRepository'.
> Failed to publish publication 'mavenJava' to repository 'maven'
   > Could not write to resource 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hcatalog/2.7.0-SNAPSHOT/beam-sdks-java-io-hcatalog-2.7.0-20180731.193056-12.jar'.
  > Could not PUT 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-hcatalog/2.7.0-SNAPSHOT/beam-sdks-java-io-hcatalog-2.7.0-20180731.193056-12.jar'.
 Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to 
get more log output. Run with --scan to get full insights.
==

46: Task failed with an exception.
---
* What went wrong:
Execution failed for task 
':beam-sdks-java-io-jdbc:publishMavenJavaPublicationToMavenRepository'.
> Failed to publish publication 'mavenJava' to repository 'maven'
   > Could not write to resource 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-jdbc/2.7.0-SNAPSHOT/beam-sdks-java-io-jdbc-2.7.0-20180731.193106-12.jar'.
  > Could not PUT 
'https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-sdks-java-io-jdbc/2.7.0-SNAPSHOT/beam-sdks-java-io-jdbc-2.7.0-20180731.193106-12.jar'.
 Received status code 401 from server: Unauthorized

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to 
get more log output. Run with --scan to get full insights.
==

47: Task failed with an exception.
---
* What went wrong:
Execution failed for task

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=129477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129477
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:35
Start Date: 31/Jul/18 19:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on issue #6052: [BEAM-4852] Only 
read symbol table when required.
URL: https://github.com/apache/beam/pull/6052#issuecomment-409341867
 
 
   Run Go PostCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129477)
Time Spent: 1h 10m  (was: 1h)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4852) [Go SDK] Beam should not retain the symbol table after function resolution

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-4852?focusedWorklogId=129475=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129475
 ]

ASF GitHub Bot logged work on BEAM-4852:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:35
Start Date: 31/Jul/18 19:35
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6052: 
[BEAM-4852] Only read symbol table when required.
URL: https://github.com/apache/beam/pull/6052#discussion_r206656687
 
 

 ##
 File path: sdks/go/pkg/beam/core/runtime/symbols.go
 ##
 @@ -26,12 +26,14 @@ import (
 )
 
 var (
-   Resolver SymbolResolver
-   cache= make(map[string]interface{})
-   mu   sync.Mutex
+   // Resolver is the accessible symbol resolver the runtime uses to find 
functions.
+   Resolver SymbolResolver
 
 Review comment:
   Good catch! Thanks for the suggestion!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129475)
Time Spent: 50m  (was: 40m)

> [Go SDK] Beam should not retain the symbol table after function resolution
> --
>
> Key: BEAM-4852
> URL: https://issues.apache.org/jira/browse/BEAM-4852
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Robert Burke
>Assignee: Robert Burke
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> In some instances, Beam Go requires introspecting the symbol table for the 
> binary to resolve functions. However it may be possible to cache these 
> results for all applicable functions, and then allow the table to be garbage 
> collected.
> The table represents a large heap cost that is retained for the lifetime of a 
> job.
> A secondary goal would be to avoid incurring the cost entirely when there's 
> nothing to look up for a job. Eg for unit tests, or ancillary uses of the 
> beam SDK (eg. migrating from some other system to beam shouldn't incur the 
> cost when the old system is being used, just because beam is linked in and 
> triggered by a runtime switch).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_JDBC #917

2018-07-31 Thread Apache Jenkins Server

See 


Changes:

[timrobertson100] [BEAM-2661] Adds KuduIO

[timrobertson100] [BEAM-2661] KuduIO: Add missing licenses

--
[...truncated 4.52 KB...]
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins3820119071328874264.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1317685798842428957.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4037835082187987960.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.7.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Collecting dill<=0.2.8.2,>=0.2.6 (from apache-beam==2.7.0.dev0)
Collecting grpcio<2,>=1.8 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/fa/53/685b9d68e5734cf8e2d13b2d4d29e6cede94714b665019a5627a623b3e21/grpcio-1.13.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.7.0.dev0)
Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.7.0.dev0)
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl
Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/27/e7/bf96130ebe633b08a3913da4bb25e50dac5779f1f68e51c99485423f7443/protobuf-3.6.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pydot<1.3,>=1.2.0 (from apache-beam==2.7.0.dev0)
Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (3.12)
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.7.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/usr/local/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.11.0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl
Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting future<1.0.0,>=0.16.0 (from apache-beam==2.7.0.dev0)
Collecting fastavro==0.19.7 (from apache-beam==2.7.0.dev0)
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/1d/0c/64f84f91643f775fdb64c6c10f4a4f0d827f8b0d98a2ba2b4bb9dc2f8646/google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.7.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.7.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/37/92/c74a643126d58505daec9addf872dfaffea3305981b90cc435f4b9213cdd/google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting proto-google-cloud-pubsub-v1==0.15.4 (from apache-beam==2.7.0.dev0)
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/76/67/6165c516ff6ceaa62eb61f11d8451e1b0acc4d3775e181630aba9652babb/google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting nose>=1.3.7 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
Collecting numpy<2,>=1.14.3 (from apache-beam==2.7.0.dev0)
  Using cached 
https://files.pythonhosted.org/packages/85/51/ba4564ded90e093dbb6adfc3e21f99ae953d9ad56477e1b0d4a93bacf7d3/numpy-1.15.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.7.0.dev0)
  Using cached

[jira] [Work logged] (BEAM-2661) Add KuduIO

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?focusedWorklogId=129457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129457
 ]

ASF GitHub Bot logged work on BEAM-2661:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:14
Start Date: 31/Jul/18 19:14
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6021: [BEAM-2661] Adds 
KuduIO
URL: https://github.com/apache/beam/pull/6021#issuecomment-409335583
 
 
   Thanks! PR is now merged. If you plan on following up on my comments, please 
file matching JIRAs


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 129457)
Time Spent: 4h 40m  (was: 4.5h)

> Add KuduIO
> --
>
> Key: BEAM-2661
> URL: https://issues.apache.org/jira/browse/BEAM-2661
> Project: Beam
>  Issue Type: New Feature
>  Components: io-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Tim Robertson
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> New IO for Apache Kudu ([https://kudu.apache.org/overview.html]).
> This work is in progress [on this 
> branch|https://github.com/timrobertson100/beam/tree/BEAM-2661-KuduIO] with 
> design aspects documented below.
> h2. The API
> The {{KuduIO}} API requires the user to provide a function to convert objects 
> into operations. This is similar to the {{JdbcIO}} but different to others, 
> such as {{HBaseIO}} which requires a pre-transform stage beforehand to 
> convert into the mutations to apply. It was originally intended to copy the 
> {{HBaseIO}} approach, but this was not possible:
>  # The Kudu 
> [Operation|https://kudu.apache.org/apidocs/org/apache/kudu/client/Operation.html]
>  is a fat class, and is a subclass of {{KuduRpc}}. It 
> holds RPC logic, callbacks and a Kudu client. Because of this the 
> {{Operation}} does not serialize and furthermore, the logic for encoding the 
> operations (Insert, Upsert etc) in the Kudu Java API are one way only (no 
> decode) because the server is written in C++.
>  # An alternative could be to introduce a new object to beam (e.g. 
> {{o.a.b.sdk.io.kudu.KuduOperation}}) to enable 
> {{PCollection}}. This was considered but was discounted 
> because:
>  ## It is not a familiar API to those already knowing Kudu
>  ## It still requires serialization and deserialization of the operations. 
> Using the existing Kudu approach of serializing into compact byte arrays 
> would require a decoder along the lines of [this almost complete 
> example|https://gist.github.com/timrobertson100/df77d1337ba8f5609319751ee7c6e01e].
>  This is possible but has fragilities given the Kudu code itself continues to 
> evolve. 
>  ## It becomes a trivial codebase in Beam to maintain by defer the object to 
> mutation mapping to within the KuduIO transform. {{JdbcIO}} gives us the 
> precedent to do this.
> h2. Testing framework
> {{Kudu}} is written in C++. While a 
> [TestMiniKuduCluster|https://github.com/cloudera/kudu/blob/master/java/kudu-client/src/test/java/org/apache/kudu/client/TestMiniKuduCluster.java]
>  does exist in Java, it requires binaries to be available for the target 
> environment which is not portable (edit: this is now a [work in 
> progress|https://issues.apache.org/jira/browse/KUDU-2411] in Kudu). Therefore 
> we opt for the following:
>  # Unit tests will use a mock Kudu client
>  # Integration tests will cover the full aspects of the {{KuduIO}} and use a 
> Docker based Kudu instance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2661) Add KuduIO

2018-07-31 Thread ASF GitHub Bot (JIRA)



 [ 
https://issues.apache.org/jira/browse/BEAM-2661?focusedWorklogId=129456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-129456
 ]

ASF GitHub Bot logged work on BEAM-2661:


Author: ASF GitHub Bot
Created on: 31/Jul/18 19:13
Start Date: 31/Jul/18 19:13
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #6021: [BEAM-2661] Adds 
KuduIO
URL: https://github.com/apache/beam/pull/6021
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/kudu/build.gradle b/sdks/java/io/kudu/build.gradle
new file mode 100644
index 000..5457ec7dfdd
--- /dev/null
+++ b/sdks/java/io/kudu/build.gradle
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyJavaNature()
+provideIntegrationTestingDependencies()
+enableJavaPerformanceTesting()
+
+description = "Apache Beam :: SDKs :: Java :: IO :: Kudu"
+ext.summary = "Library to read and write from/to Kudu"
+
+test {
+  systemProperty "log4j.configuration", "log4j-test.properties"
+  jvmArgs "-XX:-UseGCOverheadLimit"
+  if (System.getProperty("beamSurefireArgline")) {
+jvmArgs System.getProperty("beamSurefireArgline")
+  }
+}
+
+def kudu_version = "1.4.0"
+
+dependencies {
+  compile library.java.guava
+  shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
+  shadow "org.apache.kudu:kudu-client:$kudu_version"
+  shadow library.java.slf4j_api
+  testCompile project(path: ":beam-runners-direct-java", configuration: 
"shadow")
+  testCompile project(path: ":beam-sdks-java-core", configuration: 
"shadowTest")
+  testCompile project(path: ":beam-sdks-java-io-common", configuration: 
"shadow")
+  testCompile project(path: ":beam-sdks-java-io-common", configuration: 
"shadowTest")
+  testCompile library.java.hamcrest_core
+  testCompile library.java.hamcrest_library
+  testCompile library.java.junit
+}
+
diff --git 
a/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java 
b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
new file mode 100644
index 000..5694946d3dd
--- /dev/null
+++ b/sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kudu;
+
+import static com.google.common.base.Preconditions.checkArgument;
+import static com.google.common.base.Preconditions.checkState;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Splitter;
+import java.util.Collections;
+import java.util.List;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.Pipeline;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.CannotProvideCoderException;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.CoderRegistry;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import

[beam] branch master updated (57084e8 -> eb0b611)

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 57084e8  Merge pull request #6080: [BEAM-5040] Fix retry bug for 
BigQuery jobs.
 add 24b78f3  [BEAM-2661] Adds KuduIO
 add 13a8a90  [BEAM-2661] KuduIO: Add missing licenses
 new eb0b611  Merge pull request #6021: [BEAM-2661] Adds KuduIO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/{mongodb => kudu}/build.gradle|  28 +-
 .../java/org/apache/beam/sdk/io/kudu/KuduIO.java   | 477 +
 .../org/apache/beam/sdk/io/kudu/KuduService.java   |  59 +++
 .../apache/beam/sdk/io/kudu/KuduServiceImpl.java   | 250 +++
 .../apache/beam/sdk/io/kudu/TableAndRecord.java|  27 +-
 .../org/apache/beam/sdk/io/kudu}/package-info.java |   6 +-
 .../java/org/apache/beam/sdk/io/kudu/KuduIOIT.java | 230 ++
 .../org/apache/beam/sdk/io/kudu/KuduIOTest.java| 273 
 .../org/apache/beam/sdk/io/kudu/KuduTestUtils.java |  79 
 .../src/test/resources/log4j-test.properties   |   0
 settings.gradle|   2 +
 11 files changed, 1409 insertions(+), 22 deletions(-)
 copy sdks/java/io/{mongodb => kudu}/build.gradle (74%)
 create mode 100644 
sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduIO.java
 create mode 100644 
sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduService.java
 create mode 100644 
sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/KuduServiceImpl.java
 copy 
runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaExecutionContext.java
 => 
sdks/java/io/kudu/src/main/java/org/apache/beam/sdk/io/kudu/TableAndRecord.java 
(63%)
 copy sdks/java/io/{hbase/src/main/java/org/apache/beam/sdk/io/hbase => 
kudu/src/main/java/org/apache/beam/sdk/io/kudu}/package-info.java (85%)
 create mode 100644 
sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduIOIT.java
 create mode 100644 
sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduIOTest.java
 create mode 100644 
sdks/java/io/kudu/src/test/java/org/apache/beam/sdk/io/kudu/KuduTestUtils.java
 copy sdks/java/io/{hbase => kudu}/src/test/resources/log4j-test.properties 
(100%)

[beam] 01/01: Merge pull request #6021: [BEAM-2661] Adds KuduIO

2018-07-31 Thread reuvenlax

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit eb0b611e6342418a5964cf25b2d239ea3c7fb80d
Merge: 57084e8 13a8a90
Author: reuvenlax 
AuthorDate: Tue Jul 31 12:13:38 2018 -0700

Merge pull request #6021: [BEAM-2661] Adds KuduIO

 sdks/java/io/kudu/build.gradle |  50 +++
 .../java/org/apache/beam/sdk/io/kudu/KuduIO.java   | 477 +
 .../org/apache/beam/sdk/io/kudu/KuduService.java   |  59 +++
 .../apache/beam/sdk/io/kudu/KuduServiceImpl.java   | 250 +++
 .../apache/beam/sdk/io/kudu/TableAndRecord.java|  43 ++
 .../org/apache/beam/sdk/io/kudu/package-info.java  |  24 ++
 .../java/org/apache/beam/sdk/io/kudu/KuduIOIT.java | 230 ++
 .../org/apache/beam/sdk/io/kudu/KuduIOTest.java| 273 
 .../org/apache/beam/sdk/io/kudu/KuduTestUtils.java |  79 
 .../kudu/src/test/resources/log4j-test.properties  |  27 ++
 settings.gradle|   2 +
 11 files changed, 1514 insertions(+)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1129

2018-07-31 Thread Apache Jenkins Server

See 


--
[...truncated 19.59 MB...]
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.742Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.792Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.821Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForKeys/Read
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.861Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.903Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/ParDo(CollectWindows)
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.941Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:48.988Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Write mutations to Spanner into 
SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:49.030Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:49.084Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:49.128Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Group by partition/Reify into SpannerIO.Write/Write 
mutations to Cloud Spanner/Partition input
Jul 31, 2018 6:59:54 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-31T18:59:49.169Z: Fusing consumer SpannerIO.Write/To mutation 
group into ParDo(GenerateMutations)
Jul 31, 2018 6:59:54 PM

1 2 >

1 - 100 of 162 matches

Mail list logo